Oct 31 12:29:12.062122 ip-10-0-138-70 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 31 12:29:12.062135 ip-10-0-138-70 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 31 12:29:12.062144 ip-10-0-138-70 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 31 12:29:12.062459 ip-10-0-138-70 systemd[1]: Failed to start Kubernetes Kubelet. Oct 31 12:29:22.312914 ip-10-0-138-70 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 31 12:29:22.312931 ip-10-0-138-70 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot b97e6b3e70b3495a879f92f4cde9d215 -- Oct 31 12:31:22.657285 ip-10-0-138-70 systemd[1]: Starting Kubernetes Kubelet... Oct 31 12:31:23.042972 ip-10-0-138-70 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 12:31:23.042972 ip-10-0-138-70 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 12:31:23.042972 ip-10-0-138-70 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 12:31:23.042972 ip-10-0-138-70 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 12:31:23.048215 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048137 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:31:23.048215 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048214 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:31:23.048215 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048218 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:31:23.048316 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048221 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:31:23.048316 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048224 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:31:23.048316 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048227 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:31:23.048316 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048230 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:31:23.048316 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048232 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:31:23.048316 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048235 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:31:23.048316 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048240 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:31:23.048316 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048245 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:31:23.048316 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048249 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:31:23.048316 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048252 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:31:23.048316 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048255 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:31:23.048316 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048258 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:31:23.048316 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048260 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:31:23.048316 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048263 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:31:23.048316 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048265 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:31:23.048316 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048268 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:31:23.048316 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048271 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:31:23.048316 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048273 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:31:23.048316 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048276 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:31:23.048743 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048279 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:31:23.048743 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048281 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:31:23.048743 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048284 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:31:23.048743 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048286 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:31:23.048743 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048296 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:31:23.048743 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048299 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:31:23.048743 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048309 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:31:23.048743 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048312 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:31:23.048743 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048315 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:31:23.048743 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048319 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:31:23.048743 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048323 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:31:23.048743 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048326 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:31:23.048743 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048329 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:31:23.048743 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048332 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:31:23.048743 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048334 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:31:23.048743 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048337 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:31:23.048743 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048339 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:31:23.048743 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048342 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:31:23.048743 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048344 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:31:23.048743 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048347 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:31:23.049227 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048349 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:31:23.049227 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048352 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:31:23.049227 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048354 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:31:23.049227 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048358 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:31:23.049227 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048361 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:31:23.049227 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048363 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:31:23.049227 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048366 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:31:23.049227 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048368 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:31:23.049227 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048371 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:31:23.049227 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048373 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:31:23.049227 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048376 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:31:23.049227 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048379 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:31:23.049227 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048382 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:31:23.049227 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048385 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:31:23.049227 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048387 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:31:23.049227 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048390 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:31:23.049227 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048393 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:31:23.049227 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048395 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:31:23.049227 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048398 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:31:23.049227 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048400 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:31:23.049683 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048404 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:31:23.049683 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048409 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:31:23.049683 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048412 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:31:23.049683 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048415 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:31:23.049683 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048418 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:31:23.049683 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048421 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:31:23.049683 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048423 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:31:23.049683 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048426 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:31:23.049683 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048428 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:31:23.049683 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048431 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:31:23.049683 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048433 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:31:23.049683 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048436 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:31:23.049683 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048438 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:31:23.049683 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048441 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:31:23.049683 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048443 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:31:23.049683 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048446 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:31:23.049683 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048449 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:31:23.049683 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048453 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:31:23.049683 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048456 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:31:23.049683 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048458 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:31:23.050160 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048461 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:31:23.050160 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048464 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:31:23.050160 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048467 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:31:23.050160 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048470 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:31:23.050160 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048473 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:31:23.050160 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048475 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:31:23.050160 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048478 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:31:23.050160 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048481 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:31:23.050160 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048483 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:31:23.050160 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048486 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:31:23.050160 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048489 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:31:23.050160 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048491 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:31:23.050160 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048494 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:31:23.050160 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048497 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:31:23.050160 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048499 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:31:23.050160 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048502 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:31:23.050160 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048504 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:31:23.050160 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048507 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:31:23.050160 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048510 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:31:23.050610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048512 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:31:23.050610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048515 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:31:23.050610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048517 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:31:23.050610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048521 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:31:23.050610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.048524 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:31:23.050610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049312 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:31:23.050610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049319 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:31:23.050610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049322 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:31:23.050610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049325 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:31:23.050610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049328 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:31:23.050610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049332 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:31:23.050610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049334 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:31:23.050610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049337 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:31:23.050610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049340 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:31:23.050610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049343 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:31:23.050610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049346 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:31:23.050610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049349 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:31:23.050610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049352 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:31:23.050610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049356 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:31:23.051120 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049358 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:31:23.051120 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049362 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:31:23.051120 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049364 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:31:23.051120 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049367 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:31:23.051120 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049369 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:31:23.051120 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049372 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:31:23.051120 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049375 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:31:23.051120 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049378 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:31:23.051120 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049381 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:31:23.051120 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049383 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:31:23.051120 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049388 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:31:23.051120 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049391 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:31:23.051120 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049395 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:31:23.051120 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049398 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:31:23.051120 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049401 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:31:23.051120 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049403 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:31:23.051120 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049407 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:31:23.051120 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049409 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:31:23.051120 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049412 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:31:23.051566 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049415 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:31:23.051566 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049417 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:31:23.051566 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049420 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:31:23.051566 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049422 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:31:23.051566 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049425 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:31:23.051566 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049428 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:31:23.051566 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049431 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:31:23.051566 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049433 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:31:23.051566 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049436 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:31:23.051566 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049439 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:31:23.051566 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049442 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:31:23.051566 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049446 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:31:23.051566 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049448 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:31:23.051566 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049451 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:31:23.051566 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049453 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:31:23.051566 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049456 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:31:23.051566 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049458 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:31:23.051566 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049461 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:31:23.051566 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049464 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:31:23.051566 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049466 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:31:23.052059 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049469 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:31:23.052059 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049471 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:31:23.052059 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049474 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:31:23.052059 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049477 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:31:23.052059 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049479 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:31:23.052059 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049481 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:31:23.052059 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049484 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:31:23.052059 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049487 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:31:23.052059 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049489 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:31:23.052059 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049491 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:31:23.052059 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049495 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:31:23.052059 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049497 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:31:23.052059 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049500 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:31:23.052059 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049502 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:31:23.052059 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049505 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:31:23.052059 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049507 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:31:23.052059 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049509 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:31:23.052059 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049512 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:31:23.052059 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049515 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:31:23.052516 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049518 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:31:23.052516 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049521 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:31:23.052516 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049524 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:31:23.052516 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049526 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:31:23.052516 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049529 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:31:23.052516 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049531 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:31:23.052516 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049534 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:31:23.052516 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049537 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:31:23.052516 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049539 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:31:23.052516 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049542 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:31:23.052516 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049544 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:31:23.052516 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049547 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:31:23.052516 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049551 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:31:23.052516 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049554 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:31:23.052516 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049557 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:31:23.052516 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049560 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:31:23.052516 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049563 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:31:23.052516 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049565 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:31:23.052516 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049568 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:31:23.052516 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049571 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:31:23.053000 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049573 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:31:23.053000 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049576 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:31:23.053000 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049578 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:31:23.053000 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049581 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:31:23.053000 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049583 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:31:23.053000 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049587 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:31:23.053000 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049590 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:31:23.053000 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049593 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:31:23.053000 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049595 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:31:23.053000 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049597 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:31:23.053000 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049600 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:31:23.053000 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049603 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:31:23.053000 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049606 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:31:23.053000 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.049609 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:31:23.053000 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050083 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:31:23.053000 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050089 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:31:23.053000 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050092 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:31:23.053000 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050094 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:31:23.053000 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050097 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:31:23.053000 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050099 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:31:23.053471 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050102 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:31:23.053471 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050105 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:31:23.053471 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050108 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:31:23.053471 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050110 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:31:23.053471 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050113 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:31:23.053471 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050115 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:31:23.053471 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050118 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:31:23.053471 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050121 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:31:23.053471 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050124 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:31:23.053471 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050127 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:31:23.053471 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050130 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:31:23.053471 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050132 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:31:23.053471 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050135 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:31:23.053471 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050137 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:31:23.053471 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050140 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:31:23.053471 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050142 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:31:23.053471 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050144 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:31:23.053471 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050147 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:31:23.053471 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050149 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:31:23.053471 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050152 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:31:23.053935 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050154 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:31:23.053935 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050157 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:31:23.053935 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050160 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:31:23.053935 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050162 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:31:23.053935 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050166 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:31:23.053935 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050168 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:31:23.053935 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050171 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:31:23.053935 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050174 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:31:23.053935 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050176 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:31:23.053935 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050179 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:31:23.053935 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050181 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:31:23.053935 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050184 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:31:23.053935 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050186 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:31:23.053935 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050189 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:31:23.053935 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050191 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:31:23.053935 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050194 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:31:23.053935 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050196 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:31:23.053935 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050199 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:31:23.053935 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050202 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:31:23.053935 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050205 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:31:23.054414 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050207 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:31:23.054414 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050210 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:31:23.054414 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050212 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:31:23.054414 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050215 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:31:23.054414 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050217 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:31:23.054414 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050220 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:31:23.054414 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050222 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:31:23.054414 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050225 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:31:23.054414 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050227 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:31:23.054414 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050229 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:31:23.054414 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050234 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:31:23.054414 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050238 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:31:23.054414 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050242 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:31:23.054414 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050245 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:31:23.054414 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050248 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:31:23.054414 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050251 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:31:23.054414 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050253 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:31:23.054414 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050256 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:31:23.054414 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050259 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:31:23.054845 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050262 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:31:23.054845 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050264 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:31:23.054845 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050267 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:31:23.054845 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050269 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:31:23.054845 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050272 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:31:23.054845 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050275 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:31:23.054845 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050278 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:31:23.054845 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050280 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:31:23.054845 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050283 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:31:23.054845 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050286 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:31:23.054845 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050288 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:31:23.054845 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050292 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:31:23.054845 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050294 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:31:23.054845 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050297 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:31:23.054845 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050299 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:31:23.054845 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050302 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:31:23.054845 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050304 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:31:23.054845 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050307 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:31:23.054845 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050309 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:31:23.054845 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050312 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:31:23.055374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050314 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:31:23.055374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050317 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:31:23.055374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050319 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:31:23.055374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050322 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:31:23.055374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050324 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:31:23.055374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050327 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:31:23.055374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050330 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:31:23.055374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050333 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:31:23.055374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050335 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:31:23.055374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050338 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:31:23.055374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050340 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:31:23.055374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050344 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:31:23.055374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050348 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:31:23.055374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050351 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:31:23.055374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050354 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:31:23.055374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050356 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:31:23.055374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050359 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:31:23.055374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050362 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:31:23.055374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050364 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:31:23.055814 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050368 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:31:23.055814 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.050370 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:31:23.055814 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052269 2582 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 12:31:23.055814 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052279 2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 12:31:23.055814 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052286 2582 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 12:31:23.055814 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052291 2582 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 12:31:23.055814 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052296 2582 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 12:31:23.055814 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052299 2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 12:31:23.055814 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052315 2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 12:31:23.055814 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052322 2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 12:31:23.055814 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052325 2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 12:31:23.055814 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052328 2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 12:31:23.055814 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052332 2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 12:31:23.055814 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052335 2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 12:31:23.055814 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052338 2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 12:31:23.055814 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052342 2582 flags.go:64] FLAG: --cgroup-root="" Oct 31 12:31:23.055814 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052345 2582 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 12:31:23.055814 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052347 2582 flags.go:64] FLAG: --client-ca-file="" Oct 31 12:31:23.055814 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052350 2582 flags.go:64] FLAG: --cloud-provider="external" Oct 31 12:31:23.055814 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052353 2582 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 12:31:23.055814 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052357 2582 flags.go:64] FLAG: --cluster-domain="" Oct 31 12:31:23.055814 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052360 2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 12:31:23.055814 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052363 2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 12:31:23.055814 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052366 2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 12:31:23.056381 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052370 2582 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 12:31:23.056381 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052377 2582 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 12:31:23.056381 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052380 2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 12:31:23.056381 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052383 2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 12:31:23.056381 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052386 2582 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 12:31:23.056381 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052390 2582 flags.go:64] FLAG: --contention-profiling="false" Oct 31 12:31:23.056381 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052393 2582 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 12:31:23.056381 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052396 2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 12:31:23.056381 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052399 2582 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 12:31:23.056381 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052402 2582 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 12:31:23.056381 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052406 2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 12:31:23.056381 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052409 2582 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 12:31:23.056381 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052412 2582 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 12:31:23.056381 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052415 2582 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 12:31:23.056381 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052418 2582 flags.go:64] FLAG: --enable-server="true" Oct 31 12:31:23.056381 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052421 2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 12:31:23.056381 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052425 2582 flags.go:64] FLAG: --event-burst="100" Oct 31 12:31:23.056381 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052430 2582 flags.go:64] FLAG: --event-qps="50" Oct 31 12:31:23.056381 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052433 2582 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 12:31:23.056381 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052437 2582 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 12:31:23.056381 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052440 2582 flags.go:64] FLAG: --eviction-hard="" Oct 31 12:31:23.056381 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052444 2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 12:31:23.056381 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052447 2582 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 12:31:23.056381 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052450 2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 12:31:23.056381 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052453 2582 flags.go:64] FLAG: --eviction-soft="" Oct 31 12:31:23.057047 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052456 2582 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 12:31:23.057047 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052459 2582 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 12:31:23.057047 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052462 2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 12:31:23.057047 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052466 2582 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 12:31:23.057047 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052468 2582 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 12:31:23.057047 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052471 2582 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 12:31:23.057047 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052475 2582 flags.go:64] FLAG: --feature-gates="" Oct 31 12:31:23.057047 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052479 2582 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 12:31:23.057047 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052482 2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 12:31:23.057047 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052486 2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 12:31:23.057047 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052489 2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 12:31:23.057047 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052492 2582 flags.go:64] FLAG: --healthz-port="10248" Oct 31 12:31:23.057047 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052496 2582 flags.go:64] FLAG: --help="false" Oct 31 12:31:23.057047 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052499 2582 flags.go:64] FLAG: --hostname-override="ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:31:23.057047 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052502 2582 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 12:31:23.057047 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052506 2582 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 12:31:23.057047 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052509 2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 12:31:23.057047 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052513 2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 12:31:23.057047 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052516 2582 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 12:31:23.057047 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052519 2582 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 12:31:23.057047 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052522 2582 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 12:31:23.057047 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052525 2582 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 12:31:23.057047 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052529 2582 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 12:31:23.057591 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052532 2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 12:31:23.057591 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052536 2582 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 12:31:23.057591 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052538 2582 flags.go:64] FLAG: --kube-reserved="" Oct 31 12:31:23.057591 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052541 2582 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 12:31:23.057591 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052544 2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 12:31:23.057591 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052547 2582 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 12:31:23.057591 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052551 2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 12:31:23.057591 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052553 2582 flags.go:64] FLAG: --lock-file="" Oct 31 12:31:23.057591 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052556 2582 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 12:31:23.057591 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052560 2582 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 12:31:23.057591 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052563 2582 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 12:31:23.057591 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052568 2582 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 12:31:23.057591 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052571 2582 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 12:31:23.057591 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052574 2582 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 12:31:23.057591 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052577 2582 flags.go:64] FLAG: --logging-format="text" Oct 31 12:31:23.057591 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052580 2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 12:31:23.057591 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052583 2582 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 12:31:23.057591 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052586 2582 flags.go:64] FLAG: --manifest-url="" Oct 31 12:31:23.057591 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052589 2582 flags.go:64] FLAG: --manifest-url-header="" Oct 31 12:31:23.057591 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052594 2582 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 12:31:23.057591 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052598 2582 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 12:31:23.057591 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052602 2582 flags.go:64] FLAG: --max-pods="110" Oct 31 12:31:23.057591 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052605 2582 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 12:31:23.057591 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052607 2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 12:31:23.057591 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052610 2582 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 12:31:23.058266 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052613 2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 12:31:23.058266 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052616 2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 12:31:23.058266 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052619 2582 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 12:31:23.058266 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052622 2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 12:31:23.058266 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052630 2582 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 12:31:23.058266 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052633 2582 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 12:31:23.058266 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052636 2582 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 12:31:23.058266 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052639 2582 flags.go:64] FLAG: --pod-cidr="" Oct 31 12:31:23.058266 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052642 2582 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 12:31:23.058266 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052645 2582 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 12:31:23.058266 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052648 2582 flags.go:64] FLAG: --pods-per-core="0" Oct 31 12:31:23.058266 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052651 2582 flags.go:64] FLAG: --port="10250" Oct 31 12:31:23.058266 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052654 2582 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 12:31:23.058266 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052657 2582 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0214ebc941630a7b8" Oct 31 12:31:23.058266 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052661 2582 flags.go:64] FLAG: --qos-reserved="" Oct 31 12:31:23.058266 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052664 2582 flags.go:64] FLAG: --read-only-port="10255" Oct 31 12:31:23.058266 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052667 2582 flags.go:64] FLAG: --register-node="true" Oct 31 12:31:23.058266 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052669 2582 flags.go:64] FLAG: --register-with-taints="" Oct 31 12:31:23.058266 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052673 2582 flags.go:64] FLAG: --registry-burst="10" Oct 31 12:31:23.058266 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052676 2582 flags.go:64] FLAG: --registry-qps="5" Oct 31 12:31:23.058266 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052679 2582 flags.go:64] FLAG: --reserved-cpus="" Oct 31 12:31:23.058266 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052682 2582 flags.go:64] FLAG: --reserved-memory="" Oct 31 12:31:23.058266 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052686 2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 12:31:23.058266 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052689 2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 12:31:23.058266 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052692 2582 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 12:31:23.058266 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052695 2582 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 12:31:23.058904 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052698 2582 flags.go:64] FLAG: --runonce="false" Oct 31 12:31:23.058904 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052702 2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 12:31:23.058904 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052706 2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 12:31:23.058904 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052709 2582 flags.go:64] FLAG: --seccomp-default="false" Oct 31 12:31:23.058904 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052712 2582 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 12:31:23.058904 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052715 2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 12:31:23.058904 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052718 2582 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 12:31:23.058904 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052722 2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 12:31:23.058904 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052725 2582 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 12:31:23.058904 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052728 2582 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 12:31:23.058904 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052730 2582 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 12:31:23.058904 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052734 2582 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 12:31:23.058904 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052737 2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 12:31:23.058904 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052740 2582 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 12:31:23.058904 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052743 2582 flags.go:64] FLAG: --system-cgroups="" Oct 31 12:31:23.058904 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052746 2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 12:31:23.058904 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052751 2582 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 12:31:23.058904 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052753 2582 flags.go:64] FLAG: --tls-cert-file="" Oct 31 12:31:23.058904 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052756 2582 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 12:31:23.058904 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052760 2582 flags.go:64] FLAG: --tls-min-version="" Oct 31 12:31:23.058904 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052763 2582 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 12:31:23.058904 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052766 2582 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 12:31:23.058904 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052769 2582 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 12:31:23.058904 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052772 2582 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 12:31:23.058904 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052774 2582 flags.go:64] FLAG: --v="2" Oct 31 12:31:23.059495 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052778 2582 flags.go:64] FLAG: --version="false" Oct 31 12:31:23.059495 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052783 2582 flags.go:64] FLAG: --vmodule="" Oct 31 12:31:23.059495 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052787 2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 12:31:23.059495 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.052790 2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 12:31:23.059495 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.052920 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:31:23.059495 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.052923 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:31:23.059495 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.052926 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:31:23.059495 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.052928 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:31:23.059495 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.052931 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:31:23.059495 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.052940 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:31:23.059495 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.052943 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:31:23.059495 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.052946 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:31:23.059495 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.052949 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:31:23.059495 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.052951 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:31:23.059495 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.052954 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:31:23.059495 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.052956 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:31:23.059495 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.052959 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:31:23.059495 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.052962 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:31:23.059495 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.052965 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:31:23.059495 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.052967 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:31:23.059495 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.052970 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:31:23.060021 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.052973 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:31:23.060021 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.052975 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:31:23.060021 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.052994 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:31:23.060021 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.052997 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:31:23.060021 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053000 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:31:23.060021 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053002 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:31:23.060021 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053005 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:31:23.060021 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053008 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:31:23.060021 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053011 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:31:23.060021 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053013 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:31:23.060021 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053016 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:31:23.060021 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053018 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:31:23.060021 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053021 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:31:23.060021 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053024 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:31:23.060021 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053026 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:31:23.060021 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053029 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:31:23.060021 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053031 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:31:23.060021 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053034 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:31:23.060021 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053036 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:31:23.060487 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053039 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:31:23.060487 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053043 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:31:23.060487 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053046 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:31:23.060487 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053049 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:31:23.060487 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053052 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:31:23.060487 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053055 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:31:23.060487 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053057 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:31:23.060487 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053060 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:31:23.060487 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053062 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:31:23.060487 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053065 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:31:23.060487 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053068 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:31:23.060487 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053070 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:31:23.060487 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053074 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:31:23.060487 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053077 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:31:23.060487 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053079 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:31:23.060487 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053082 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:31:23.060487 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053085 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:31:23.060487 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053088 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:31:23.060487 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053090 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:31:23.060930 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053093 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:31:23.060930 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053095 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:31:23.060930 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053098 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:31:23.060930 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053101 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:31:23.060930 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053103 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:31:23.060930 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053106 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:31:23.060930 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053108 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:31:23.060930 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053111 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:31:23.060930 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053113 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:31:23.060930 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053116 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:31:23.060930 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053119 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:31:23.060930 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053121 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:31:23.060930 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053124 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:31:23.060930 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053127 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:31:23.060930 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053131 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:31:23.060930 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053134 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:31:23.060930 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053137 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:31:23.060930 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053139 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:31:23.060930 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053142 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:31:23.060930 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053144 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:31:23.061415 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053147 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:31:23.061415 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053149 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:31:23.061415 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053152 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:31:23.061415 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053156 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:31:23.061415 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053159 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:31:23.061415 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053164 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:31:23.061415 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053168 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:31:23.061415 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053170 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:31:23.061415 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053173 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:31:23.061415 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053176 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:31:23.061415 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053179 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:31:23.061415 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053181 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:31:23.061415 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053184 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:31:23.061415 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053187 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:31:23.061415 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053189 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:31:23.061415 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053192 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:31:23.061415 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053196 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:31:23.061415 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053199 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:31:23.061415 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053202 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:31:23.061862 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053205 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:31:23.061862 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053208 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:31:23.061862 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053211 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:31:23.061862 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053214 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:31:23.061862 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053216 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:31:23.061862 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053219 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:31:23.061862 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053222 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:31:23.061862 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053225 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:31:23.061862 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053228 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:31:23.061862 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053231 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:31:23.061862 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053234 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:31:23.061862 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053237 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:31:23.061862 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.053265 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 12:31:23.061862 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053304 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:31:23.061862 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053308 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:31:23.061862 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053312 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:31:23.062241 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053315 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:31:23.062241 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053317 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:31:23.062241 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053320 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:31:23.062241 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053325 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:31:23.062241 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053328 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:31:23.062241 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053331 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:31:23.062241 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053333 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:31:23.062241 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053336 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:31:23.062241 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053339 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:31:23.062241 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053342 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:31:23.062241 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053344 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:31:23.062241 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053347 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:31:23.062241 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053349 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:31:23.062241 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053352 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:31:23.062241 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053354 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:31:23.062241 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053357 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:31:23.062241 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053359 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:31:23.062241 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053362 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:31:23.062241 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053365 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:31:23.062690 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053367 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:31:23.062690 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053370 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:31:23.062690 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053373 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:31:23.062690 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053375 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:31:23.062690 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053379 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:31:23.062690 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053382 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:31:23.062690 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053384 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:31:23.062690 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053387 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:31:23.062690 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053390 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:31:23.062690 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053393 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:31:23.062690 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053395 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:31:23.062690 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053398 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:31:23.062690 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053400 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:31:23.062690 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053403 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:31:23.062690 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053406 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:31:23.062690 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053408 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:31:23.062690 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053412 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:31:23.062690 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053415 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:31:23.062690 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053417 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:31:23.062690 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053420 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:31:23.063276 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053422 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:31:23.063276 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053425 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:31:23.063276 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053427 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:31:23.063276 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053430 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:31:23.063276 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053432 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:31:23.063276 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053435 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:31:23.063276 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053437 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:31:23.063276 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053441 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:31:23.063276 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053445 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:31:23.063276 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053449 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:31:23.063276 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053451 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:31:23.063276 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053454 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:31:23.063276 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053457 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:31:23.063276 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053459 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:31:23.063276 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053462 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:31:23.063276 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053465 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:31:23.063276 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053468 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:31:23.063276 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053471 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:31:23.063276 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053474 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:31:23.063721 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053476 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:31:23.063721 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053480 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:31:23.063721 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053484 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:31:23.063721 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053487 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:31:23.063721 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053490 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:31:23.063721 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053492 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:31:23.063721 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053496 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:31:23.063721 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053499 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:31:23.063721 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053501 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:31:23.063721 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053505 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:31:23.063721 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053508 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:31:23.063721 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053510 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:31:23.063721 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053513 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:31:23.063721 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053516 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:31:23.063721 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053519 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:31:23.063721 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053521 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:31:23.063721 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053524 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:31:23.063721 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053526 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:31:23.063721 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053529 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:31:23.063721 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053532 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:31:23.064232 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053535 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:31:23.064232 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053537 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:31:23.064232 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053540 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:31:23.064232 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053542 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:31:23.064232 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053545 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:31:23.064232 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053547 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:31:23.064232 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053550 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:31:23.064232 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053552 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:31:23.064232 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053555 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:31:23.064232 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053559 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:31:23.064232 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053561 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:31:23.064232 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053564 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:31:23.064232 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053566 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:31:23.064232 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053569 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:31:23.064232 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053572 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:31:23.064232 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053575 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:31:23.064232 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053577 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:31:23.064232 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053580 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:31:23.064232 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053582 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:31:23.064232 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053586 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:31:23.064694 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053589 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:31:23.064694 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053592 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:31:23.064694 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053595 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:31:23.064694 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053598 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:31:23.064694 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.053600 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:31:23.064694 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.061993 2582 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 12:31:23.064694 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.062009 2582 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 12:31:23.064694 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.062746 2582 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 12:31:23.064694 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.062752 2582 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 12:31:23.064694 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062821 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:31:23.064694 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062826 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:31:23.064694 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062830 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:31:23.064694 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062834 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:31:23.064694 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062836 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:31:23.064694 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062839 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:31:23.064694 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062842 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:31:23.064694 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062844 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:31:23.064694 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062847 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:31:23.064694 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062850 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:31:23.065166 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062852 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:31:23.065166 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062855 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:31:23.065166 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062858 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:31:23.065166 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062862 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:31:23.065166 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062866 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:31:23.065166 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062870 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:31:23.065166 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062873 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:31:23.065166 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062876 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:31:23.065166 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062885 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:31:23.065166 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062888 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:31:23.065166 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062891 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:31:23.065166 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062894 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:31:23.065166 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062896 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:31:23.065166 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062899 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:31:23.065166 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062908 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:31:23.065166 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062911 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:31:23.065166 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062914 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:31:23.065166 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062916 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:31:23.065166 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062919 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:31:23.065610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062922 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:31:23.065610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062925 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:31:23.065610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062928 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:31:23.065610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062930 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:31:23.065610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062933 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:31:23.065610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062936 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:31:23.065610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062940 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:31:23.065610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062943 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:31:23.065610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062946 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:31:23.065610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062949 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:31:23.065610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062952 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:31:23.065610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062954 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:31:23.065610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062957 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:31:23.065610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062959 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:31:23.065610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062962 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:31:23.065610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062965 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:31:23.065610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062967 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:31:23.065610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062970 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:31:23.065610 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062972 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:31:23.066230 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062975 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:31:23.066230 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062977 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:31:23.066230 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.062993 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:31:23.066230 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063001 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:31:23.066230 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063004 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:31:23.066230 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063007 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:31:23.066230 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063010 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:31:23.066230 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063012 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:31:23.066230 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063015 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:31:23.066230 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063018 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:31:23.066230 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063021 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:31:23.066230 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063023 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:31:23.066230 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063026 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:31:23.066230 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063030 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:31:23.066230 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063033 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:31:23.066230 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063035 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:31:23.066230 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063038 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:31:23.066230 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063040 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:31:23.066230 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063043 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:31:23.066230 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063046 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:31:23.067083 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063048 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:31:23.067083 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063051 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:31:23.067083 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063054 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:31:23.067083 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063056 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:31:23.067083 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063059 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:31:23.067083 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063061 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:31:23.067083 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063064 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:31:23.067083 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063066 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:31:23.067083 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063068 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:31:23.067083 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063071 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:31:23.067083 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063073 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:31:23.067083 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063076 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:31:23.067083 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063079 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:31:23.067083 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063081 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:31:23.067083 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063083 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:31:23.067083 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063086 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:31:23.067083 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063094 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:31:23.067083 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063097 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:31:23.067083 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063100 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:31:23.067083 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063102 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:31:23.067749 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063105 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:31:23.067749 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063108 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:31:23.067749 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063111 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:31:23.067749 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063114 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:31:23.067749 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063116 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:31:23.067749 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063119 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:31:23.067749 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063122 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:31:23.067749 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063124 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:31:23.067749 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063127 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:31:23.067749 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063130 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:31:23.067749 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063132 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:31:23.067749 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063135 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:31:23.067749 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063137 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:31:23.067749 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063140 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:31:23.067749 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063142 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:31:23.067749 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063145 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:31:23.067749 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063147 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:31:23.067749 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063150 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:31:23.068227 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.063177 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 12:31:23.068227 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063363 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:31:23.068227 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063369 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:31:23.068227 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063372 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:31:23.068227 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063375 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:31:23.068227 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063378 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:31:23.068227 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063380 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:31:23.068227 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063383 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:31:23.068227 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063386 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:31:23.068227 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063388 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:31:23.068227 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063391 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:31:23.068227 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063399 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:31:23.068227 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063402 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:31:23.068227 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063405 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:31:23.068227 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063407 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:31:23.068227 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063410 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:31:23.068654 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063412 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:31:23.068654 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063415 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:31:23.068654 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063418 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:31:23.068654 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063420 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:31:23.068654 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063423 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:31:23.068654 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063425 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:31:23.068654 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063427 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:31:23.068654 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063430 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:31:23.068654 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063433 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:31:23.068654 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063435 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:31:23.068654 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063438 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:31:23.068654 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063440 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:31:23.068654 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063443 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:31:23.068654 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063445 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:31:23.068654 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063448 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:31:23.068654 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063450 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:31:23.068654 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063453 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:31:23.068654 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063456 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:31:23.068654 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063458 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:31:23.068654 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063461 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:31:23.069130 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063464 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:31:23.069130 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063466 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:31:23.069130 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063469 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:31:23.069130 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063471 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:31:23.069130 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063474 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:31:23.069130 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063476 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:31:23.069130 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063479 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:31:23.069130 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063481 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:31:23.069130 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063489 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:31:23.069130 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063492 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:31:23.069130 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063495 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:31:23.069130 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063497 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:31:23.069130 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063501 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:31:23.069130 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063505 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:31:23.069130 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063508 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:31:23.069130 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063511 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:31:23.069130 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063514 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:31:23.069130 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063516 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:31:23.069130 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063519 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:31:23.069574 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063521 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:31:23.069574 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063524 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:31:23.069574 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063526 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:31:23.069574 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063529 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:31:23.069574 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063531 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:31:23.069574 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063534 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:31:23.069574 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063537 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:31:23.069574 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063539 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:31:23.069574 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063542 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:31:23.069574 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063544 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:31:23.069574 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063547 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:31:23.069574 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063549 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:31:23.069574 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063552 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:31:23.069574 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063554 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:31:23.069574 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063557 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:31:23.069574 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063560 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:31:23.069574 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063562 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:31:23.069574 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063565 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:31:23.069574 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063567 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:31:23.070094 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063570 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:31:23.070094 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063572 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:31:23.070094 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063575 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:31:23.070094 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063583 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:31:23.070094 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063586 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:31:23.070094 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063589 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:31:23.070094 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063591 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:31:23.070094 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063594 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:31:23.070094 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063596 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:31:23.070094 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063599 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:31:23.070094 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063601 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:31:23.070094 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063604 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:31:23.070094 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063606 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:31:23.070094 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063609 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:31:23.070094 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063611 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:31:23.070094 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063614 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:31:23.070094 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063616 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:31:23.070094 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063619 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:31:23.070094 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063621 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:31:23.070548 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063625 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:31:23.070548 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063628 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:31:23.070548 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063631 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:31:23.070548 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063633 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:31:23.070548 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063636 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:31:23.070548 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063639 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:31:23.070548 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063642 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:31:23.070548 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063644 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:31:23.070548 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063647 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:31:23.070548 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063649 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:31:23.070548 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063651 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:31:23.070548 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063654 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:31:23.070548 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063657 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:31:23.070548 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063659 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:31:23.070548 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.063684 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 12:31:23.070548 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063733 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:31:23.070965 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063738 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:31:23.070965 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063747 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:31:23.070965 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063751 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:31:23.070965 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063753 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:31:23.070965 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063756 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:31:23.070965 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063759 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:31:23.070965 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063762 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:31:23.070965 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063765 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:31:23.070965 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063768 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:31:23.070965 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063770 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:31:23.070965 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063773 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:31:23.070965 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063776 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:31:23.070965 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063778 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:31:23.070965 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063781 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:31:23.070965 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063783 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:31:23.070965 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063788 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:31:23.070965 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063790 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:31:23.070965 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063793 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:31:23.070965 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063795 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:31:23.071416 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063798 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:31:23.071416 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063801 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:31:23.071416 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063804 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:31:23.071416 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063806 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:31:23.071416 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063809 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:31:23.071416 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063811 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:31:23.071416 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063813 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:31:23.071416 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063816 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:31:23.071416 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063819 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:31:23.071416 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063821 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:31:23.071416 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063824 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:31:23.071416 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063827 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:31:23.071416 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063829 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:31:23.071416 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063831 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:31:23.071416 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063840 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:31:23.071416 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063848 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:31:23.071416 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063851 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:31:23.071416 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063853 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:31:23.071416 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063856 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:31:23.071416 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063858 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:31:23.071918 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063861 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:31:23.071918 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063863 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:31:23.071918 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063866 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:31:23.071918 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063868 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:31:23.071918 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063873 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:31:23.071918 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063876 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:31:23.071918 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063879 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:31:23.071918 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063882 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:31:23.071918 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063886 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:31:23.071918 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063890 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:31:23.071918 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063892 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:31:23.071918 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063895 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:31:23.071918 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063898 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:31:23.071918 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063901 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:31:23.071918 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063904 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:31:23.071918 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063907 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:31:23.071918 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063910 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:31:23.071918 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063913 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:31:23.071918 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063916 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:31:23.072374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063918 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:31:23.072374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063921 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:31:23.072374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063923 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:31:23.072374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063926 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:31:23.072374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063928 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:31:23.072374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063931 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:31:23.072374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063933 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:31:23.072374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063936 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:31:23.072374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063939 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:31:23.072374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063947 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:31:23.072374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063950 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:31:23.072374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063952 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:31:23.072374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063955 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:31:23.072374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063957 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:31:23.072374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063960 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:31:23.072374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063962 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:31:23.072374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063965 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:31:23.072374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063968 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:31:23.072374 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063970 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:31:23.072813 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063972 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:31:23.072813 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063975 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:31:23.072813 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063989 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:31:23.072813 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063992 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:31:23.072813 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063995 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:31:23.072813 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.063997 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:31:23.072813 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.064000 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:31:23.072813 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.064003 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:31:23.072813 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.064005 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:31:23.072813 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.064008 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:31:23.072813 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.064011 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:31:23.072813 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.064015 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:31:23.072813 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.064019 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:31:23.072813 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.064022 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:31:23.072813 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.064025 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:31:23.072813 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.064028 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:31:23.072813 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.064030 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:31:23.072813 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.064033 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:31:23.072813 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.064035 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:31:23.072813 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.064038 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:31:23.073370 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.064040 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:31:23.073370 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.064043 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:31:23.073370 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.064045 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:31:23.073370 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.064054 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:31:23.073370 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.064057 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:31:23.073370 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.064059 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:31:23.073370 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.064062 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:31:23.073370 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:23.064064 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:31:23.073370 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.064131 2582 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 12:31:23.073370 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.067312 2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 12:31:23.073370 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.068364 2582 server.go:1014] "Starting client certificate rotation" Oct 31 12:31:23.073370 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.068453 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 12:31:23.073370 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.068486 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 12:31:23.073370 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.070926 2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 12:31:23.073370 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.072269 2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 12:31:23.085088 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.085072 2582 log.go:25] "Validated CRI v1 runtime API" Oct 31 12:31:23.089075 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.089060 2582 log.go:25] "Validated CRI v1 image API" Oct 31 12:31:23.089951 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.089926 2582 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 12:31:23.097573 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.097546 2582 fs.go:135] Filesystem UUIDs: map[01febce6-dac2-4843-81a5-19674d633042:/dev/nvme0n1p4 20bba8e4-2e3a-4312-9ae5-7078c5fbf135:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 31 12:31:23.097651 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.097573 2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 12:31:23.101131 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.101106 2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 12:31:23.103706 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.103588 2582 manager.go:217] Machine: {Timestamp:2026-10-31 12:31:23.102234307 +0000 UTC m=+0.356507825 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100823 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f22714cfba4a925c95ace1069530b SystemUUID:ec2f2271-4cfb-a4a9-25c9-5ace1069530b BootID:b97e6b3e-70b3-495a-879f-92f4cde9d215 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:cc:05:24:4d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:cc:05:24:4d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:d9:9b:96:33:27 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 12:31:23.104408 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.104395 2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 12:31:23.104536 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.104516 2582 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 12:31:23.108344 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.108319 2582 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 12:31:23.108501 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.108351 2582 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-70.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 12:31:23.108554 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.108506 2582 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 12:31:23.108554 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.108514 2582 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 12:31:23.108554 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.108525 2582 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 12:31:23.109501 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.109490 2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 12:31:23.110754 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.110745 2582 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 12:31:23.115498 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.115487 2582 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 12:31:23.115618 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.115608 2582 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 12:31:23.115651 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.115648 2582 kubelet.go:493] "Attempting to sync node with API server" Oct 31 12:31:23.115728 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.115659 2582 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 12:31:23.115728 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.115669 2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 12:31:23.115728 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.115682 2582 kubelet.go:405] "Adding apiserver pod source" Oct 31 12:31:23.115728 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.115689 2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 12:31:23.118354 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.118330 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 12:31:23.120749 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.120728 2582 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 12:31:23.124112 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.124095 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-k255p" Oct 31 12:31:23.127398 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.127386 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 12:31:23.128366 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.128354 2582 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 12:31:23.128400 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.128384 2582 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 12:31:23.128536 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.128527 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 12:31:23.128565 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.128542 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 12:31:23.128565 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.128549 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 12:31:23.128565 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.128554 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 12:31:23.128565 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.128560 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 12:31:23.128565 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.128565 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 12:31:23.129268 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.129260 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 12:31:23.129296 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.129269 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 12:31:23.129296 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.129276 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 12:31:23.129296 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.129283 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 12:31:23.129296 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.129291 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 12:31:23.129422 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.129300 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 12:31:23.130189 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.130178 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 12:31:23.130189 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.130188 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 12:31:23.133607 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.133572 2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-70.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 12:31:23.134218 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.134204 2582 server.go:1263] "Started kubelet" Oct 31 12:31:23.134314 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.134289 2582 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 12:31:23.134404 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.134348 2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 12:31:23.134507 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.134437 2582 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 12:31:23.135024 ip-10-0-138-70 systemd[1]: Started Kubernetes Kubelet. Oct 31 12:31:23.135972 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.135484 2582 server.go:317] "Adding debug handlers to kubelet server" Oct 31 12:31:23.135972 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.135640 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-k255p" Oct 31 12:31:23.136100 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.136057 2582 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 12:31:23.143012 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:23.142975 2582 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 12:31:23.144234 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.144216 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 12:31:23.144883 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.144863 2582 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 12:31:23.145036 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:23.145018 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-70.us-east-2.compute.internal\" not found" Oct 31 12:31:23.145800 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.145784 2582 factory.go:55] Registering systemd factory Oct 31 12:31:23.145800 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.145802 2582 factory.go:223] Registration of the systemd container factory successfully Oct 31 12:31:23.145894 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.145864 2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 12:31:23.145894 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.145865 2582 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 12:31:23.145967 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.145893 2582 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 12:31:23.146033 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.146021 2582 reconstruct.go:97] "Volume reconstruction finished" Oct 31 12:31:23.146033 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.146030 2582 reconciler.go:29] "Reconciler: start to sync state" Oct 31 12:31:23.146113 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.146060 2582 factory.go:153] Registering CRI-O factory Oct 31 12:31:23.146113 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.146070 2582 factory.go:223] Registration of the crio container factory successfully Oct 31 12:31:23.146113 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.146111 2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 12:31:23.146214 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.146125 2582 factory.go:103] Registering Raw factory Oct 31 12:31:23.146214 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.146135 2582 manager.go:1199] Started watching for new ooms in manager Oct 31 12:31:23.147230 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.147212 2582 manager.go:319] Starting recovery of all containers Oct 31 12:31:23.155008 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.154447 2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:31:23.156730 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:23.156713 2582 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-138-70.us-east-2.compute.internal\" not found" node="ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:31:23.157385 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.157365 2582 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:31:23.157812 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.157788 2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:31:23.162391 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.162329 2582 manager.go:324] Recovery completed Oct 31 12:31:23.166720 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.166705 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:31:23.168752 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.168737 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-70.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:31:23.168805 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.168765 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-70.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:31:23.168805 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.168775 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-70.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:31:23.169306 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.169291 2582 cpu_manager.go:226] "Starting" policy="none" Oct 31 12:31:23.169306 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.169303 2582 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 12:31:23.169424 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.169293 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:31:23.169424 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.169319 2582 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 12:31:23.169960 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.169946 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-70.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:31:23.170064 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.169971 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-70.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:31:23.170064 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.170035 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-70.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:31:23.171415 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.171404 2582 policy_none.go:50] "Start" Oct 31 12:31:23.171450 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.171420 2582 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 12:31:23.171450 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.171430 2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 12:31:23.173435 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.173426 2582 policy_none.go:44] "Start" Oct 31 12:31:23.197563 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.197539 2582 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 12:31:23.201081 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.201067 2582 manager.go:341] "Starting Device Plugin manager" Oct 31 12:31:23.201180 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:23.201103 2582 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 12:31:23.201180 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.201114 2582 server.go:87] "Starting device plugin registration server" Oct 31 12:31:23.201352 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.201334 2582 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 12:31:23.201438 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.201350 2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 12:31:23.201916 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.201533 2582 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 12:31:23.201916 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.201606 2582 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 12:31:23.201916 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.201612 2582 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 12:31:23.202291 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:23.202080 2582 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 12:31:23.202291 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:23.202151 2582 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-70.us-east-2.compute.internal\" not found" Oct 31 12:31:23.286455 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.286423 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 12:31:23.288441 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.287691 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 12:31:23.288441 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.287711 2582 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 12:31:23.288441 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.287730 2582 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 12:31:23.288441 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:23.287767 2582 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 31 12:31:23.291187 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.291172 2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:31:23.302411 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.302372 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:31:23.303578 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.303556 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-70.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:31:23.303670 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.303591 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-70.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:31:23.303670 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.303608 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-70.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:31:23.303670 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.303641 2582 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:31:23.310197 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.310182 2582 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:31:23.310280 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:23.310206 2582 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-70.us-east-2.compute.internal\": node \"ip-10-0-138-70.us-east-2.compute.internal\" not found" Oct 31 12:31:23.388927 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.388889 2582 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-70.us-east-2.compute.internal"] Oct 31 12:31:23.391400 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.391379 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:31:23.393255 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.393237 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:31:23.421554 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.421535 2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:31:23.424324 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.424310 2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:31:23.439633 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.439615 2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 12:31:23.439714 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.439620 2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 12:31:23.446958 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.446938 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/578d51cfd2278b7149c7a4e11f7c55d2-config\") pod \"kube-apiserver-proxy-ip-10-0-138-70.us-east-2.compute.internal\" (UID: \"578d51cfd2278b7149c7a4e11f7c55d2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:31:23.447053 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.446961 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/62138b49cad483f3e9534fc842796713-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal\" (UID: \"62138b49cad483f3e9534fc842796713\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:31:23.447053 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.447004 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62138b49cad483f3e9534fc842796713-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal\" (UID: \"62138b49cad483f3e9534fc842796713\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:31:23.547608 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.547582 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/62138b49cad483f3e9534fc842796713-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal\" (UID: \"62138b49cad483f3e9534fc842796713\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:31:23.547608 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.547609 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62138b49cad483f3e9534fc842796713-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal\" (UID: \"62138b49cad483f3e9534fc842796713\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:31:23.547786 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.547627 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/578d51cfd2278b7149c7a4e11f7c55d2-config\") pod \"kube-apiserver-proxy-ip-10-0-138-70.us-east-2.compute.internal\" (UID: \"578d51cfd2278b7149c7a4e11f7c55d2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:31:23.547786 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.547654 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/578d51cfd2278b7149c7a4e11f7c55d2-config\") pod \"kube-apiserver-proxy-ip-10-0-138-70.us-east-2.compute.internal\" (UID: \"578d51cfd2278b7149c7a4e11f7c55d2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:31:23.547786 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.547685 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/62138b49cad483f3e9534fc842796713-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal\" (UID: \"62138b49cad483f3e9534fc842796713\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:31:23.547786 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.547689 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62138b49cad483f3e9534fc842796713-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal\" (UID: \"62138b49cad483f3e9534fc842796713\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:31:23.743558 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.743533 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:31:23.744625 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:23.744604 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:31:24.068183 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.067979 2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 12:31:24.068697 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.068314 2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 12:31:24.068697 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.068320 2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 12:31:24.068697 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.068318 2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 12:31:24.117140 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.117108 2582 apiserver.go:52] "Watching apiserver" Oct 31 12:31:24.127206 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.127182 2582 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 12:31:24.127501 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.127481 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-g98lz","openshift-dns/node-resolver-7p252","kube-system/konnectivity-agent-2wtkj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal","openshift-network-diagnostics/network-check-target-7sf98","openshift-network-operator/iptables-alerter-27p2l","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4","openshift-image-registry/node-ca-chf5k","openshift-multus/multus-additional-cni-plugins-plzf9","openshift-multus/multus-rdcdb","openshift-multus/network-metrics-daemon-8bvmr","kube-system/kube-apiserver-proxy-ip-10-0-138-70.us-east-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-h9pd6","kube-system/global-pull-secret-syncer-blcxw"] Oct 31 12:31:24.130114 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.130095 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.132323 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.132303 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7p252" Oct 31 12:31:24.134456 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.134438 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2wtkj" Oct 31 12:31:24.134953 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.134935 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 12:31:24.135028 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.134954 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 12:31:24.135238 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.135210 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7phhl\"" Oct 31 12:31:24.136335 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.136322 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 12:31:24.136479 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.136460 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 12:31:24.136680 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.136655 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:31:24.136771 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:24.136724 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sf98" podUID="6ab66c79-2b0a-4671-9bf8-6aafea4e7454" Oct 31 12:31:24.136817 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.136766 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bkb2g\"" Oct 31 12:31:24.138165 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.138146 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 12:31:24.138165 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.138162 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 12:31:24.138304 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.138154 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hk7cx\"" Oct 31 12:31:24.138915 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.138898 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-27p2l" Oct 31 12:31:24.141326 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.141308 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:31:24.143652 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.143613 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 12:26:24 +0000 UTC" deadline="2028-07-26 10:03:30.615454662 +0000 UTC" Oct 31 12:31:24.143652 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.143650 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15213h32m6.471808281s" Oct 31 12:31:24.143790 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.143705 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-chf5k" Oct 31 12:31:24.144350 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.144334 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 12:31:24.146084 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.146068 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:31:24.148435 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.148419 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.150292 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.150274 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-host-run-netns\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.150368 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.150299 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-etc-sysctl-conf\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.150368 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.150316 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-sys\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.150368 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.150339 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-host-var-lib-cni-multus\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.150474 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.150387 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jd8l\" (UniqueName: \"kubernetes.io/projected/9fcef325-e046-4c6d-8599-6c67115f382c-kube-api-access-5jd8l\") pod \"node-ca-chf5k\" (UID: \"9fcef325-e046-4c6d-8599-6c67115f382c\") " pod="openshift-image-registry/node-ca-chf5k" Oct 31 12:31:24.150474 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.150412 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwz6m\" (UniqueName: \"kubernetes.io/projected/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-kube-api-access-fwz6m\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:31:24.150474 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.150454 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-etc-modprobe-d\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.150474 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.150472 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d89912fc-3948-4569-bfea-87c8dceedd36-socket-dir\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:31:24.150646 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.150487 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-etc-kubernetes\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.150646 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.150503 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-etc-systemd\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.150646 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.150523 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/29bc026b-84aa-4cbf-ba44-bb5c499e6bdc-agent-certs\") pod \"konnectivity-agent-2wtkj\" (UID: \"29bc026b-84aa-4cbf-ba44-bb5c499e6bdc\") " pod="kube-system/konnectivity-agent-2wtkj" Oct 31 12:31:24.150646 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.150552 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-host-var-lib-cni-bin\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.150646 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.150569 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-cni-binary-copy\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:31:24.150646 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.150586 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-run\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.150646 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.150602 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d89912fc-3948-4569-bfea-87c8dceedd36-etc-selinux\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:31:24.150646 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.150630 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8087a5a6-949e-491f-98b9-3360b93a6419-iptables-alerter-script\") pod \"iptables-alerter-27p2l\" (UID: \"8087a5a6-949e-491f-98b9-3360b93a6419\") " pod="openshift-network-operator/iptables-alerter-27p2l" Oct 31 12:31:24.151006 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.150706 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d89912fc-3948-4569-bfea-87c8dceedd36-device-dir\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:31:24.151006 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.150738 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4f51d774-d999-4957-a165-0346972d65bc-hosts-file\") pod \"node-resolver-7p252\" (UID: \"4f51d774-d999-4957-a165-0346972d65bc\") " pod="openshift-dns/node-resolver-7p252" Oct 31 12:31:24.151006 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.150758 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d89912fc-3948-4569-bfea-87c8dceedd36-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:31:24.151006 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.150774 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-host-run-multus-certs\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.151006 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.150824 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9fcef325-e046-4c6d-8599-6c67115f382c-serviceca\") pod \"node-ca-chf5k\" (UID: \"9fcef325-e046-4c6d-8599-6c67115f382c\") " pod="openshift-image-registry/node-ca-chf5k" Oct 31 12:31:24.151006 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.150836 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:31:24.151006 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.150857 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-host\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.151006 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:24.150890 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bvmr" podUID="856f4c46-f3c3-4afe-823e-1b77842932a4" Oct 31 12:31:24.151006 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.150952 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgk87\" (UniqueName: \"kubernetes.io/projected/2b6cb291-3443-496d-8598-3b253102832f-kube-api-access-xgk87\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.151006 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.150987 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhn4r\" (UniqueName: \"kubernetes.io/projected/4f51d774-d999-4957-a165-0346972d65bc-kube-api-access-fhn4r\") pod \"node-resolver-7p252\" (UID: \"4f51d774-d999-4957-a165-0346972d65bc\") " pod="openshift-dns/node-resolver-7p252" Oct 31 12:31:24.151389 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.151035 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-host-run-k8s-cni-cncf-io\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.151389 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.151062 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzs6g\" (UniqueName: \"kubernetes.io/projected/6ab66c79-2b0a-4671-9bf8-6aafea4e7454-kube-api-access-qzs6g\") pod \"network-check-target-7sf98\" (UID: \"6ab66c79-2b0a-4671-9bf8-6aafea4e7454\") " pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:31:24.151389 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.151098 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9fcef325-e046-4c6d-8599-6c67115f382c-host\") pod \"node-ca-chf5k\" (UID: \"9fcef325-e046-4c6d-8599-6c67115f382c\") " pod="openshift-image-registry/node-ca-chf5k" Oct 31 12:31:24.151389 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.151152 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-etc-sysconfig\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.151389 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.151171 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-lib-modules\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.151389 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.151236 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f51d774-d999-4957-a165-0346972d65bc-tmp-dir\") pod \"node-resolver-7p252\" (UID: \"4f51d774-d999-4957-a165-0346972d65bc\") " pod="openshift-dns/node-resolver-7p252" Oct 31 12:31:24.151389 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.151261 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b6cb291-3443-496d-8598-3b253102832f-tmp\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.151389 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.151285 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8wjg\" (UniqueName: \"kubernetes.io/projected/d89912fc-3948-4569-bfea-87c8dceedd36-kube-api-access-g8wjg\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:31:24.151389 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.151325 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-host-var-lib-kubelet\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.151389 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.151347 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggkt5\" (UniqueName: \"kubernetes.io/projected/8087a5a6-949e-491f-98b9-3360b93a6419-kube-api-access-ggkt5\") pod \"iptables-alerter-27p2l\" (UID: \"8087a5a6-949e-491f-98b9-3360b93a6419\") " pod="openshift-network-operator/iptables-alerter-27p2l" Oct 31 12:31:24.151857 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.151394 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2b6cb291-3443-496d-8598-3b253102832f-etc-tuned\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.151857 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.151704 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d89912fc-3948-4569-bfea-87c8dceedd36-sys-fs\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:31:24.151857 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.151845 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-hostroot\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.152007 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.151870 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-cnibin\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:31:24.152007 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.151890 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d89912fc-3948-4569-bfea-87c8dceedd36-registration-dir\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:31:24.152007 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.151969 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-cni-binary-copy\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.152204 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.152008 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-multus-socket-dir-parent\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.152204 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.152085 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-system-cni-dir\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:31:24.152204 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.152130 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-var-lib-kubelet\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.154260 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.154219 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-multus-cni-dir\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.154355 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.154307 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-system-cni-dir\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.154430 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.154347 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-cnibin\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.154489 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.154451 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.154489 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.154477 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8087a5a6-949e-491f-98b9-3360b93a6419-host-slash\") pod \"iptables-alerter-27p2l\" (UID: \"8087a5a6-949e-491f-98b9-3360b93a6419\") " pod="openshift-network-operator/iptables-alerter-27p2l" Oct 31 12:31:24.154594 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.154519 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-etc-sysctl-d\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.154995 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.154971 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/29bc026b-84aa-4cbf-ba44-bb5c499e6bdc-konnectivity-ca\") pod \"konnectivity-agent-2wtkj\" (UID: \"29bc026b-84aa-4cbf-ba44-bb5c499e6bdc\") " pod="kube-system/konnectivity-agent-2wtkj" Oct 31 12:31:24.155164 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.155138 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-multus-conf-dir\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.155249 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.155195 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-multus-daemon-config\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.155315 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.155260 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jrdh\" (UniqueName: \"kubernetes.io/projected/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-kube-api-access-6jrdh\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.155315 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.155295 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:31:24.155414 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.155321 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:31:24.155414 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.155366 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:31:24.155490 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.155409 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-etc-kubernetes\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.157429 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.157412 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:31:24.157506 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:24.157482 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-blcxw" podUID="ed7f6a91-3f6e-409f-bc57-b5d0f8e67770" Oct 31 12:31:24.166722 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.166702 2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 12:31:24.207968 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.207947 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9hrvm" Oct 31 12:31:24.220934 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.220903 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9hrvm" Oct 31 12:31:24.244376 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.244357 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 12:31:24.244747 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.244716 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 12:31:24.244747 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.244744 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 12:31:24.244891 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.244777 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 12:31:24.244891 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.244788 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 12:31:24.244891 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.244792 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 12:31:24.244891 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.244821 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 12:31:24.244891 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.244833 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 12:31:24.244891 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.244782 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 12:31:24.244891 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.244860 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 12:31:24.245435 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.245419 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 12:31:24.245746 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.245733 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 12:31:24.245796 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.245750 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 12:31:24.245796 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.245753 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 12:31:24.246249 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.246226 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 12:31:24.246331 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.246227 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 12:31:24.246331 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.246269 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8c8p7\"" Oct 31 12:31:24.246331 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.246290 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 12:31:24.246654 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.246637 2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 12:31:24.248100 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.247926 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 12:31:24.248100 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.247971 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 12:31:24.248100 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.247982 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 12:31:24.248100 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.248013 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w8vnc\"" Oct 31 12:31:24.248100 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.247991 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zzvpb\"" Oct 31 12:31:24.248100 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.248019 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6k457\"" Oct 31 12:31:24.248100 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.248035 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 12:31:24.248404 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.248117 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zfd9q\"" Oct 31 12:31:24.248676 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.248663 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fmbl2\"" Oct 31 12:31:24.256197 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.256178 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d89912fc-3948-4569-bfea-87c8dceedd36-registration-dir\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:31:24.256297 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.256205 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-cni-binary-copy\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.256297 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.256224 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-multus-socket-dir-parent\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.256297 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.256247 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-system-cni-dir\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:31:24.256406 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.256337 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d89912fc-3948-4569-bfea-87c8dceedd36-registration-dir\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:31:24.256406 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.256347 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-multus-socket-dir-parent\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.256406 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.256359 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-var-lib-kubelet\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.256406 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.256388 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-multus-cni-dir\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.256605 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.256412 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-system-cni-dir\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:31:24.256605 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.256459 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-var-lib-kubelet\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.256605 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.256472 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-multus-cni-dir\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.256605 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.256482 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-system-cni-dir\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.256605 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.256519 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-cnibin\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.256605 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.256550 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8087a5a6-949e-491f-98b9-3360b93a6419-host-slash\") pod \"iptables-alerter-27p2l\" (UID: \"8087a5a6-949e-491f-98b9-3360b93a6419\") " pod="openshift-network-operator/iptables-alerter-27p2l" Oct 31 12:31:24.256605 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.256575 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-etc-sysctl-d\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.256605 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.256592 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-cnibin\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.256605 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.256601 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-system-cni-dir\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.256969 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.256605 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/29bc026b-84aa-4cbf-ba44-bb5c499e6bdc-konnectivity-ca\") pod \"konnectivity-agent-2wtkj\" (UID: \"29bc026b-84aa-4cbf-ba44-bb5c499e6bdc\") " pod="kube-system/konnectivity-agent-2wtkj" Oct 31 12:31:24.256969 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.256723 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8087a5a6-949e-491f-98b9-3360b93a6419-host-slash\") pod \"iptables-alerter-27p2l\" (UID: \"8087a5a6-949e-491f-98b9-3360b93a6419\") " pod="openshift-network-operator/iptables-alerter-27p2l" Oct 31 12:31:24.256969 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.256740 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-var-lib-openvswitch\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.256969 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.256761 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-multus-conf-dir\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.256969 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.256803 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-etc-sysctl-d\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.256969 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.256802 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-multus-daemon-config\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.256969 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.256872 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-cni-binary-copy\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.256969 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.256905 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-multus-conf-dir\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.256969 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.256876 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jrdh\" (UniqueName: \"kubernetes.io/projected/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-kube-api-access-6jrdh\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.257419 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.257065 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:31:24.257419 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.257177 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:31:24.257419 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.257329 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:31:24.257708 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.257214 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:31:24.257791 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.257736 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-etc-kubernetes\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.257791 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.257771 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-etc-openvswitch\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.257881 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.257772 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/29bc026b-84aa-4cbf-ba44-bb5c499e6bdc-konnectivity-ca\") pod \"konnectivity-agent-2wtkj\" (UID: \"29bc026b-84aa-4cbf-ba44-bb5c499e6bdc\") " pod="kube-system/konnectivity-agent-2wtkj" Oct 31 12:31:24.257881 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.257803 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-host-run-netns\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.257881 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.257862 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-etc-kubernetes\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.257881 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.257871 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-host-run-netns\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.258043 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.257889 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:31:24.258043 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.257897 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-etc-sysctl-conf\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.258043 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.257949 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-sys\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.258043 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.257972 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-run-openvswitch\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.258043 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.257975 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-etc-sysctl-conf\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.258043 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.257990 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-cni-netd\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.258043 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.258000 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:31:24.258043 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.258017 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-multus-daemon-config\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.258367 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.258055 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-host-var-lib-cni-multus\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.258367 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.258074 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jd8l\" (UniqueName: \"kubernetes.io/projected/9fcef325-e046-4c6d-8599-6c67115f382c-kube-api-access-5jd8l\") pod \"node-ca-chf5k\" (UID: \"9fcef325-e046-4c6d-8599-6c67115f382c\") " pod="openshift-image-registry/node-ca-chf5k" Oct 31 12:31:24.258367 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.258077 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-sys\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.258367 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.258090 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwz6m\" (UniqueName: \"kubernetes.io/projected/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-kube-api-access-fwz6m\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:31:24.258367 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.258106 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-etc-modprobe-d\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.258367 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.258106 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-host-var-lib-cni-multus\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.258367 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.258257 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-etc-modprobe-d\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.258628 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.258540 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.258628 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.258574 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d89912fc-3948-4569-bfea-87c8dceedd36-socket-dir\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:31:24.258628 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.258589 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-etc-kubernetes\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.258628 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.258604 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-etc-systemd\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.258899 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.258630 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/29bc026b-84aa-4cbf-ba44-bb5c499e6bdc-agent-certs\") pod \"konnectivity-agent-2wtkj\" (UID: \"29bc026b-84aa-4cbf-ba44-bb5c499e6bdc\") " pod="kube-system/konnectivity-agent-2wtkj" Oct 31 12:31:24.258899 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.258647 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-kubelet\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.258899 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.258662 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-systemd-units\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.258899 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.258701 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-etc-kubernetes\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.258899 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.258729 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-etc-systemd\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.258899 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.258779 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-ovn-node-metrics-cert\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.258899 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.258788 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d89912fc-3948-4569-bfea-87c8dceedd36-socket-dir\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:31:24.258899 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.258811 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-host-var-lib-cni-bin\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.258899 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.258853 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-host-var-lib-cni-bin\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.258899 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.258846 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-cni-binary-copy\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:31:24.258899 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.258891 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-run\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.259491 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.258926 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-run-systemd\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.259491 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.258941 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-run-ovn\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.259491 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.258988 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-node-log\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.259491 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.259017 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-dbus\") pod \"global-pull-secret-syncer-blcxw\" (UID: \"ed7f6a91-3f6e-409f-bc57-b5d0f8e67770\") " pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:31:24.259491 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.259055 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d89912fc-3948-4569-bfea-87c8dceedd36-etc-selinux\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:31:24.259491 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.259094 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8087a5a6-949e-491f-98b9-3360b93a6419-iptables-alerter-script\") pod \"iptables-alerter-27p2l\" (UID: \"8087a5a6-949e-491f-98b9-3360b93a6419\") " pod="openshift-network-operator/iptables-alerter-27p2l" Oct 31 12:31:24.259491 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.259110 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-run\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.259491 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.259123 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d89912fc-3948-4569-bfea-87c8dceedd36-etc-selinux\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:31:24.259491 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.259124 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4hhg\" (UniqueName: \"kubernetes.io/projected/856f4c46-f3c3-4afe-823e-1b77842932a4-kube-api-access-n4hhg\") pod \"network-metrics-daemon-8bvmr\" (UID: \"856f4c46-f3c3-4afe-823e-1b77842932a4\") " pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:31:24.259491 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.259206 2582 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 12:31:24.259491 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.259259 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d89912fc-3948-4569-bfea-87c8dceedd36-device-dir\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:31:24.259491 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.259288 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4f51d774-d999-4957-a165-0346972d65bc-hosts-file\") pod \"node-resolver-7p252\" (UID: \"4f51d774-d999-4957-a165-0346972d65bc\") " pod="openshift-dns/node-resolver-7p252" Oct 31 12:31:24.259491 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.259315 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw546\" (UniqueName: \"kubernetes.io/projected/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-kube-api-access-cw546\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.259491 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.259349 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d89912fc-3948-4569-bfea-87c8dceedd36-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:31:24.259491 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.259362 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-cni-binary-copy\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:31:24.259491 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.259376 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-host-run-multus-certs\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.259491 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.259369 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d89912fc-3948-4569-bfea-87c8dceedd36-device-dir\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:31:24.260440 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.259405 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9fcef325-e046-4c6d-8599-6c67115f382c-serviceca\") pod \"node-ca-chf5k\" (UID: \"9fcef325-e046-4c6d-8599-6c67115f382c\") " pod="openshift-image-registry/node-ca-chf5k" Oct 31 12:31:24.260440 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.259467 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4f51d774-d999-4957-a165-0346972d65bc-hosts-file\") pod \"node-resolver-7p252\" (UID: \"4f51d774-d999-4957-a165-0346972d65bc\") " pod="openshift-dns/node-resolver-7p252" Oct 31 12:31:24.260440 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.259492 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-host-run-multus-certs\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.260440 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.259496 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-host\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.260440 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.259557 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-host\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.260440 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.259596 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgk87\" (UniqueName: \"kubernetes.io/projected/2b6cb291-3443-496d-8598-3b253102832f-kube-api-access-xgk87\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.260440 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.259706 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhn4r\" (UniqueName: \"kubernetes.io/projected/4f51d774-d999-4957-a165-0346972d65bc-kube-api-access-fhn4r\") pod \"node-resolver-7p252\" (UID: \"4f51d774-d999-4957-a165-0346972d65bc\") " pod="openshift-dns/node-resolver-7p252" Oct 31 12:31:24.260440 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.259733 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8087a5a6-949e-491f-98b9-3360b93a6419-iptables-alerter-script\") pod \"iptables-alerter-27p2l\" (UID: \"8087a5a6-949e-491f-98b9-3360b93a6419\") " pod="openshift-network-operator/iptables-alerter-27p2l" Oct 31 12:31:24.260440 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.259736 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-run-ovn-kubernetes\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.260440 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.259797 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d89912fc-3948-4569-bfea-87c8dceedd36-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:31:24.260440 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.259798 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-ovnkube-script-lib\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.260440 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.259882 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9fcef325-e046-4c6d-8599-6c67115f382c-serviceca\") pod \"node-ca-chf5k\" (UID: \"9fcef325-e046-4c6d-8599-6c67115f382c\") " pod="openshift-image-registry/node-ca-chf5k" Oct 31 12:31:24.260440 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.259931 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-host-run-k8s-cni-cncf-io\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.260440 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.259966 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-host-run-k8s-cni-cncf-io\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.260440 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.259994 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzs6g\" (UniqueName: \"kubernetes.io/projected/6ab66c79-2b0a-4671-9bf8-6aafea4e7454-kube-api-access-qzs6g\") pod \"network-check-target-7sf98\" (UID: \"6ab66c79-2b0a-4671-9bf8-6aafea4e7454\") " pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:31:24.260440 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.260059 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9fcef325-e046-4c6d-8599-6c67115f382c-host\") pod \"node-ca-chf5k\" (UID: \"9fcef325-e046-4c6d-8599-6c67115f382c\") " pod="openshift-image-registry/node-ca-chf5k" Oct 31 12:31:24.260440 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.260171 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-etc-sysconfig\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.260440 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.260186 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9fcef325-e046-4c6d-8599-6c67115f382c-host\") pod \"node-ca-chf5k\" (UID: \"9fcef325-e046-4c6d-8599-6c67115f382c\") " pod="openshift-image-registry/node-ca-chf5k" Oct 31 12:31:24.261438 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.260213 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-lib-modules\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.261438 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.260253 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f51d774-d999-4957-a165-0346972d65bc-tmp-dir\") pod \"node-resolver-7p252\" (UID: \"4f51d774-d999-4957-a165-0346972d65bc\") " pod="openshift-dns/node-resolver-7p252" Oct 31 12:31:24.261438 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.260264 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-etc-sysconfig\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.261438 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.260312 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-log-socket\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.261438 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.260336 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-env-overrides\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.261438 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.260362 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b6cb291-3443-496d-8598-3b253102832f-tmp\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.261438 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.260375 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-lib-modules\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.261438 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.260397 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-run-netns\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.261438 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.260432 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/856f4c46-f3c3-4afe-823e-1b77842932a4-metrics-certs\") pod \"network-metrics-daemon-8bvmr\" (UID: \"856f4c46-f3c3-4afe-823e-1b77842932a4\") " pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:31:24.261438 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.260451 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8wjg\" (UniqueName: \"kubernetes.io/projected/d89912fc-3948-4569-bfea-87c8dceedd36-kube-api-access-g8wjg\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:31:24.261438 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.260551 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-host-var-lib-kubelet\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.261438 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.260577 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggkt5\" (UniqueName: \"kubernetes.io/projected/8087a5a6-949e-491f-98b9-3360b93a6419-kube-api-access-ggkt5\") pod \"iptables-alerter-27p2l\" (UID: \"8087a5a6-949e-491f-98b9-3360b93a6419\") " pod="openshift-network-operator/iptables-alerter-27p2l" Oct 31 12:31:24.261438 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.260609 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2b6cb291-3443-496d-8598-3b253102832f-etc-tuned\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.261438 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.260621 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-host-var-lib-kubelet\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.261438 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.260576 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f51d774-d999-4957-a165-0346972d65bc-tmp-dir\") pod \"node-resolver-7p252\" (UID: \"4f51d774-d999-4957-a165-0346972d65bc\") " pod="openshift-dns/node-resolver-7p252" Oct 31 12:31:24.261438 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.260637 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-slash\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.261438 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.260717 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-cni-bin\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.262024 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.260748 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-kubelet-config\") pod \"global-pull-secret-syncer-blcxw\" (UID: \"ed7f6a91-3f6e-409f-bc57-b5d0f8e67770\") " pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:31:24.262024 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.260774 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-original-pull-secret\") pod \"global-pull-secret-syncer-blcxw\" (UID: \"ed7f6a91-3f6e-409f-bc57-b5d0f8e67770\") " pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:31:24.262024 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.260810 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d89912fc-3948-4569-bfea-87c8dceedd36-sys-fs\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:31:24.262024 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.260853 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-hostroot\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.262024 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.260879 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-cnibin\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:31:24.262024 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.260899 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d89912fc-3948-4569-bfea-87c8dceedd36-sys-fs\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:31:24.262024 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.260934 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-ovnkube-config\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.262024 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.260949 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-hostroot\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.262024 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.261015 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-cnibin\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:31:24.266812 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.263941 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b6cb291-3443-496d-8598-3b253102832f-tmp\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.266812 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.264138 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2b6cb291-3443-496d-8598-3b253102832f-etc-tuned\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.266812 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.264439 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/29bc026b-84aa-4cbf-ba44-bb5c499e6bdc-agent-certs\") pod \"konnectivity-agent-2wtkj\" (UID: \"29bc026b-84aa-4cbf-ba44-bb5c499e6bdc\") " pod="kube-system/konnectivity-agent-2wtkj" Oct 31 12:31:24.271857 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.271831 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jd8l\" (UniqueName: \"kubernetes.io/projected/9fcef325-e046-4c6d-8599-6c67115f382c-kube-api-access-5jd8l\") pod \"node-ca-chf5k\" (UID: \"9fcef325-e046-4c6d-8599-6c67115f382c\") " pod="openshift-image-registry/node-ca-chf5k" Oct 31 12:31:24.273402 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.273376 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jrdh\" (UniqueName: \"kubernetes.io/projected/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-kube-api-access-6jrdh\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.273486 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.273383 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwz6m\" (UniqueName: \"kubernetes.io/projected/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-kube-api-access-fwz6m\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:31:24.275416 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:24.275389 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 12:31:24.275416 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:24.275418 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 12:31:24.275578 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:24.275434 2582 projected.go:196] Error preparing data for projected volume kube-api-access-qzs6g for pod openshift-network-diagnostics/network-check-target-7sf98: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:24.275578 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:24.275508 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ab66c79-2b0a-4671-9bf8-6aafea4e7454-kube-api-access-qzs6g podName:6ab66c79-2b0a-4671-9bf8-6aafea4e7454 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:24.775486045 +0000 UTC m=+2.029759574 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qzs6g" (UniqueName: "kubernetes.io/projected/6ab66c79-2b0a-4671-9bf8-6aafea4e7454-kube-api-access-qzs6g") pod "network-check-target-7sf98" (UID: "6ab66c79-2b0a-4671-9bf8-6aafea4e7454") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:24.275578 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.275509 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggkt5\" (UniqueName: \"kubernetes.io/projected/8087a5a6-949e-491f-98b9-3360b93a6419-kube-api-access-ggkt5\") pod \"iptables-alerter-27p2l\" (UID: \"8087a5a6-949e-491f-98b9-3360b93a6419\") " pod="openshift-network-operator/iptables-alerter-27p2l" Oct 31 12:31:24.275929 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.275906 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgk87\" (UniqueName: \"kubernetes.io/projected/2b6cb291-3443-496d-8598-3b253102832f-kube-api-access-xgk87\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.276281 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.276259 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhn4r\" (UniqueName: \"kubernetes.io/projected/4f51d774-d999-4957-a165-0346972d65bc-kube-api-access-fhn4r\") pod \"node-resolver-7p252\" (UID: \"4f51d774-d999-4957-a165-0346972d65bc\") " pod="openshift-dns/node-resolver-7p252" Oct 31 12:31:24.279039 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.279017 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8wjg\" (UniqueName: \"kubernetes.io/projected/d89912fc-3948-4569-bfea-87c8dceedd36-kube-api-access-g8wjg\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:31:24.361565 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.361498 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-log-socket\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.361565 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.361535 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-env-overrides\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.361565 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.361563 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-run-netns\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.361820 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.361622 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-log-socket\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.361820 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.361665 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/856f4c46-f3c3-4afe-823e-1b77842932a4-metrics-certs\") pod \"network-metrics-daemon-8bvmr\" (UID: \"856f4c46-f3c3-4afe-823e-1b77842932a4\") " pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:31:24.361820 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.361699 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-slash\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.361820 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.361725 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-cni-bin\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.361820 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.361699 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-run-netns\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.361820 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.361751 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-kubelet-config\") pod \"global-pull-secret-syncer-blcxw\" (UID: \"ed7f6a91-3f6e-409f-bc57-b5d0f8e67770\") " pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:31:24.361820 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:24.361781 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:24.361820 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.361793 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-kubelet-config\") pod \"global-pull-secret-syncer-blcxw\" (UID: \"ed7f6a91-3f6e-409f-bc57-b5d0f8e67770\") " pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:31:24.361820 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.361769 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-slash\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.361820 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.361818 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-cni-bin\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.362281 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.361817 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-original-pull-secret\") pod \"global-pull-secret-syncer-blcxw\" (UID: \"ed7f6a91-3f6e-409f-bc57-b5d0f8e67770\") " pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:31:24.362281 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:24.361856 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/856f4c46-f3c3-4afe-823e-1b77842932a4-metrics-certs podName:856f4c46-f3c3-4afe-823e-1b77842932a4 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:24.861837265 +0000 UTC m=+2.116110798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/856f4c46-f3c3-4afe-823e-1b77842932a4-metrics-certs") pod "network-metrics-daemon-8bvmr" (UID: "856f4c46-f3c3-4afe-823e-1b77842932a4") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:24.362281 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:24.361898 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:24.362281 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.361947 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-ovnkube-config\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.362281 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:24.361958 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-original-pull-secret podName:ed7f6a91-3f6e-409f-bc57-b5d0f8e67770 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:24.861942656 +0000 UTC m=+2.116216186 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-original-pull-secret") pod "global-pull-secret-syncer-blcxw" (UID: "ed7f6a91-3f6e-409f-bc57-b5d0f8e67770") : object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:24.362281 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.362013 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-var-lib-openvswitch\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.362281 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.362076 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-etc-openvswitch\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.362281 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.362089 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-env-overrides\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.362281 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.362114 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-run-openvswitch\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.362281 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.362141 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-cni-netd\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.362281 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.362144 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-var-lib-openvswitch\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.362281 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.362175 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.362281 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.362193 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-run-openvswitch\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.362281 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.362203 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-cni-netd\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.362281 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.362209 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-kubelet\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.362281 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.362255 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-systemd-units\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.363099 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.362260 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-kubelet\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.363099 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.362258 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.363099 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.362282 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-ovn-node-metrics-cert\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.363099 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.362305 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-systemd-units\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.363099 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.362330 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-etc-openvswitch\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.363099 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.362331 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-run-systemd\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.363099 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.362372 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-run-ovn\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.363099 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.362390 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-run-systemd\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.363099 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.362402 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-node-log\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.363099 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.362428 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-dbus\") pod \"global-pull-secret-syncer-blcxw\" (UID: \"ed7f6a91-3f6e-409f-bc57-b5d0f8e67770\") " pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:31:24.363099 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.362433 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-run-ovn\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.363099 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.362442 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-ovnkube-config\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.363099 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.362459 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4hhg\" (UniqueName: \"kubernetes.io/projected/856f4c46-f3c3-4afe-823e-1b77842932a4-kube-api-access-n4hhg\") pod \"network-metrics-daemon-8bvmr\" (UID: \"856f4c46-f3c3-4afe-823e-1b77842932a4\") " pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:31:24.363099 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.362493 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw546\" (UniqueName: \"kubernetes.io/projected/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-kube-api-access-cw546\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.363099 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.362473 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-node-log\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.363099 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.362523 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-run-ovn-kubernetes\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.363099 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.362582 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-ovnkube-script-lib\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.363657 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.362586 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-run-ovn-kubernetes\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.363657 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.362586 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-dbus\") pod \"global-pull-secret-syncer-blcxw\" (UID: \"ed7f6a91-3f6e-409f-bc57-b5d0f8e67770\") " pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:31:24.363754 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.363721 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-ovnkube-script-lib\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.364508 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.364490 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-ovn-node-metrics-cert\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.382635 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.382612 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4hhg\" (UniqueName: \"kubernetes.io/projected/856f4c46-f3c3-4afe-823e-1b77842932a4-kube-api-access-n4hhg\") pod \"network-metrics-daemon-8bvmr\" (UID: \"856f4c46-f3c3-4afe-823e-1b77842932a4\") " pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:31:24.382763 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.382748 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw546\" (UniqueName: \"kubernetes.io/projected/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-kube-api-access-cw546\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.389008 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:24.388967 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62138b49cad483f3e9534fc842796713.slice/crio-9a45582cbe71480fd292ad3a12002be22ce33917475324ae2cc1b8d3156c7363 WatchSource:0}: Error finding container 9a45582cbe71480fd292ad3a12002be22ce33917475324ae2cc1b8d3156c7363: Status 404 returned error can't find the container with id 9a45582cbe71480fd292ad3a12002be22ce33917475324ae2cc1b8d3156c7363 Oct 31 12:31:24.397485 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.397469 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 12:31:24.402415 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:24.402391 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod578d51cfd2278b7149c7a4e11f7c55d2.slice/crio-2a5593f17ffaf057c3130cf9bb1de5849fbd82eddb2fbfe51ddf9144fd876f77 WatchSource:0}: Error finding container 2a5593f17ffaf057c3130cf9bb1de5849fbd82eddb2fbfe51ddf9144fd876f77: Status 404 returned error can't find the container with id 2a5593f17ffaf057c3130cf9bb1de5849fbd82eddb2fbfe51ddf9144fd876f77 Oct 31 12:31:24.463828 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.463796 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:31:24.473142 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.473125 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7p252" Oct 31 12:31:24.473664 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:24.473643 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b6cb291_3443_496d_8598_3b253102832f.slice/crio-70d6869e587b95b03790fa01ae08d0b214366d7e7d63ab14540bd30b30d32116 WatchSource:0}: Error finding container 70d6869e587b95b03790fa01ae08d0b214366d7e7d63ab14540bd30b30d32116: Status 404 returned error can't find the container with id 70d6869e587b95b03790fa01ae08d0b214366d7e7d63ab14540bd30b30d32116 Oct 31 12:31:24.481953 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:24.481930 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f51d774_d999_4957_a165_0346972d65bc.slice/crio-c5b7285bb83af01ab1e07e8abf8b7bcf1af2e34737a034a5d74b33e7a8b68f1b WatchSource:0}: Error finding container c5b7285bb83af01ab1e07e8abf8b7bcf1af2e34737a034a5d74b33e7a8b68f1b: Status 404 returned error can't find the container with id c5b7285bb83af01ab1e07e8abf8b7bcf1af2e34737a034a5d74b33e7a8b68f1b Oct 31 12:31:24.487636 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.487615 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2wtkj" Oct 31 12:31:24.496475 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:24.496456 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29bc026b_84aa_4cbf_ba44_bb5c499e6bdc.slice/crio-2aada7e02f979a289459c877f627fbcbe636ec005ad4901058d33e5b4fcc38a6 WatchSource:0}: Error finding container 2aada7e02f979a289459c877f627fbcbe636ec005ad4901058d33e5b4fcc38a6: Status 404 returned error can't find the container with id 2aada7e02f979a289459c877f627fbcbe636ec005ad4901058d33e5b4fcc38a6 Oct 31 12:31:24.501862 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.501846 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-27p2l" Oct 31 12:31:24.509577 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:24.509557 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8087a5a6_949e_491f_98b9_3360b93a6419.slice/crio-28a49097c3af65785e13df0b750453b22cda3a77c1091663e4b32d1a34931038 WatchSource:0}: Error finding container 28a49097c3af65785e13df0b750453b22cda3a77c1091663e4b32d1a34931038: Status 404 returned error can't find the container with id 28a49097c3af65785e13df0b750453b22cda3a77c1091663e4b32d1a34931038 Oct 31 12:31:24.522766 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.522747 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:31:24.531165 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:24.531140 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd89912fc_3948_4569_bfea_87c8dceedd36.slice/crio-2d16696a4281ee3226f6a8b5d61f1b670e057fa5bb3b46bcfc96f53442467faf WatchSource:0}: Error finding container 2d16696a4281ee3226f6a8b5d61f1b670e057fa5bb3b46bcfc96f53442467faf: Status 404 returned error can't find the container with id 2d16696a4281ee3226f6a8b5d61f1b670e057fa5bb3b46bcfc96f53442467faf Oct 31 12:31:24.533120 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.533097 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-chf5k" Oct 31 12:31:24.541701 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:24.541676 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fcef325_e046_4c6d_8599_6c67115f382c.slice/crio-91495a0e37b2ab21647b1e27e9cac114cf4fde7363c472c369a23b030bbb3fc0 WatchSource:0}: Error finding container 91495a0e37b2ab21647b1e27e9cac114cf4fde7363c472c369a23b030bbb3fc0: Status 404 returned error can't find the container with id 91495a0e37b2ab21647b1e27e9cac114cf4fde7363c472c369a23b030bbb3fc0 Oct 31 12:31:24.572497 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.572472 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:31:24.573642 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.573623 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rdcdb" Oct 31 12:31:24.583928 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:24.583894 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf23a1c72_8427_4b6f_ad2b_cc76a88709c9.slice/crio-29662ec5981eedb63451d94253c121403437cf209e58fdd35b582cc07cd6083e WatchSource:0}: Error finding container 29662ec5981eedb63451d94253c121403437cf209e58fdd35b582cc07cd6083e: Status 404 returned error can't find the container with id 29662ec5981eedb63451d94253c121403437cf209e58fdd35b582cc07cd6083e Oct 31 12:31:24.586133 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:24.586109 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b7d2569_9fcf_4e91_9e2f_42beb94593a8.slice/crio-80fd5c23535b507f55088f79647078b229734ff457ca4877f93320bfe66d162d WatchSource:0}: Error finding container 80fd5c23535b507f55088f79647078b229734ff457ca4877f93320bfe66d162d: Status 404 returned error can't find the container with id 80fd5c23535b507f55088f79647078b229734ff457ca4877f93320bfe66d162d Oct 31 12:31:24.600633 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.600616 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:24.610516 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:31:24.610494 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca9b156d_7499_4b53_b17f_01dc63ae0c5e.slice/crio-3aae7470827bb34e14998980bf978982122239e729acb001001ec90d56250f17 WatchSource:0}: Error finding container 3aae7470827bb34e14998980bf978982122239e729acb001001ec90d56250f17: Status 404 returned error can't find the container with id 3aae7470827bb34e14998980bf978982122239e729acb001001ec90d56250f17 Oct 31 12:31:24.867450 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.867356 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzs6g\" (UniqueName: \"kubernetes.io/projected/6ab66c79-2b0a-4671-9bf8-6aafea4e7454-kube-api-access-qzs6g\") pod \"network-check-target-7sf98\" (UID: \"6ab66c79-2b0a-4671-9bf8-6aafea4e7454\") " pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:31:24.867450 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.867423 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/856f4c46-f3c3-4afe-823e-1b77842932a4-metrics-certs\") pod \"network-metrics-daemon-8bvmr\" (UID: \"856f4c46-f3c3-4afe-823e-1b77842932a4\") " pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:31:24.867681 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:24.867456 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-original-pull-secret\") pod \"global-pull-secret-syncer-blcxw\" (UID: \"ed7f6a91-3f6e-409f-bc57-b5d0f8e67770\") " pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:31:24.867681 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:24.867519 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 12:31:24.867681 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:24.867542 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 12:31:24.867681 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:24.867551 2582 projected.go:196] Error preparing data for projected volume kube-api-access-qzs6g for pod openshift-network-diagnostics/network-check-target-7sf98: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:24.867681 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:24.867570 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:24.867681 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:24.867596 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ab66c79-2b0a-4671-9bf8-6aafea4e7454-kube-api-access-qzs6g podName:6ab66c79-2b0a-4671-9bf8-6aafea4e7454 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:25.867581835 +0000 UTC m=+3.121855372 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qzs6g" (UniqueName: "kubernetes.io/projected/6ab66c79-2b0a-4671-9bf8-6aafea4e7454-kube-api-access-qzs6g") pod "network-check-target-7sf98" (UID: "6ab66c79-2b0a-4671-9bf8-6aafea4e7454") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:24.867681 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:24.867623 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-original-pull-secret podName:ed7f6a91-3f6e-409f-bc57-b5d0f8e67770 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:25.86760596 +0000 UTC m=+3.121879482 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-original-pull-secret") pod "global-pull-secret-syncer-blcxw" (UID: "ed7f6a91-3f6e-409f-bc57-b5d0f8e67770") : object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:24.867681 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:24.867570 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:24.867681 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:24.867660 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/856f4c46-f3c3-4afe-823e-1b77842932a4-metrics-certs podName:856f4c46-f3c3-4afe-823e-1b77842932a4 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:25.867649238 +0000 UTC m=+3.121922749 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/856f4c46-f3c3-4afe-823e-1b77842932a4-metrics-certs") pod "network-metrics-daemon-8bvmr" (UID: "856f4c46-f3c3-4afe-823e-1b77842932a4") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:25.107481 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:25.107421 2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:31:25.224535 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:25.224468 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 12:26:25 +0000 UTC" deadline="2028-05-08 23:41:16.417366893 +0000 UTC" Oct 31 12:31:25.224535 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:25.224533 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13331h9m51.192838339s" Oct 31 12:31:25.291960 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:25.291324 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:31:25.291960 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:25.291460 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-blcxw" podUID="ed7f6a91-3f6e-409f-bc57-b5d0f8e67770" Oct 31 12:31:25.301299 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:25.301238 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" event={"ID":"ca9b156d-7499-4b53-b17f-01dc63ae0c5e","Type":"ContainerStarted","Data":"3aae7470827bb34e14998980bf978982122239e729acb001001ec90d56250f17"} Oct 31 12:31:25.307236 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:25.307203 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-plzf9" event={"ID":"f23a1c72-8427-4b6f-ad2b-cc76a88709c9","Type":"ContainerStarted","Data":"29662ec5981eedb63451d94253c121403437cf209e58fdd35b582cc07cd6083e"} Oct 31 12:31:25.309689 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:25.309619 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-chf5k" event={"ID":"9fcef325-e046-4c6d-8599-6c67115f382c","Type":"ContainerStarted","Data":"91495a0e37b2ab21647b1e27e9cac114cf4fde7363c472c369a23b030bbb3fc0"} Oct 31 12:31:25.311823 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:25.311740 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" event={"ID":"d89912fc-3948-4569-bfea-87c8dceedd36","Type":"ContainerStarted","Data":"2d16696a4281ee3226f6a8b5d61f1b670e057fa5bb3b46bcfc96f53442467faf"} Oct 31 12:31:25.314208 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:25.314143 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-27p2l" event={"ID":"8087a5a6-949e-491f-98b9-3360b93a6419","Type":"ContainerStarted","Data":"28a49097c3af65785e13df0b750453b22cda3a77c1091663e4b32d1a34931038"} Oct 31 12:31:25.317351 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:25.317250 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7p252" event={"ID":"4f51d774-d999-4957-a165-0346972d65bc","Type":"ContainerStarted","Data":"c5b7285bb83af01ab1e07e8abf8b7bcf1af2e34737a034a5d74b33e7a8b68f1b"} Oct 31 12:31:25.327101 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:25.327072 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rdcdb" event={"ID":"2b7d2569-9fcf-4e91-9e2f-42beb94593a8","Type":"ContainerStarted","Data":"80fd5c23535b507f55088f79647078b229734ff457ca4877f93320bfe66d162d"} Oct 31 12:31:25.336228 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:25.336199 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2wtkj" event={"ID":"29bc026b-84aa-4cbf-ba44-bb5c499e6bdc","Type":"ContainerStarted","Data":"2aada7e02f979a289459c877f627fbcbe636ec005ad4901058d33e5b4fcc38a6"} Oct 31 12:31:25.349158 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:25.348973 2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:31:25.351929 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:25.351858 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g98lz" event={"ID":"2b6cb291-3443-496d-8598-3b253102832f","Type":"ContainerStarted","Data":"70d6869e587b95b03790fa01ae08d0b214366d7e7d63ab14540bd30b30d32116"} Oct 31 12:31:25.353824 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:25.353763 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-70.us-east-2.compute.internal" event={"ID":"578d51cfd2278b7149c7a4e11f7c55d2","Type":"ContainerStarted","Data":"2a5593f17ffaf057c3130cf9bb1de5849fbd82eddb2fbfe51ddf9144fd876f77"} Oct 31 12:31:25.360423 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:25.360395 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal" event={"ID":"62138b49cad483f3e9534fc842796713","Type":"ContainerStarted","Data":"9a45582cbe71480fd292ad3a12002be22ce33917475324ae2cc1b8d3156c7363"} Oct 31 12:31:25.650657 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:25.650518 2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:31:25.875949 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:25.875910 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzs6g\" (UniqueName: \"kubernetes.io/projected/6ab66c79-2b0a-4671-9bf8-6aafea4e7454-kube-api-access-qzs6g\") pod \"network-check-target-7sf98\" (UID: \"6ab66c79-2b0a-4671-9bf8-6aafea4e7454\") " pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:31:25.876190 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:25.875968 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/856f4c46-f3c3-4afe-823e-1b77842932a4-metrics-certs\") pod \"network-metrics-daemon-8bvmr\" (UID: \"856f4c46-f3c3-4afe-823e-1b77842932a4\") " pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:31:25.876190 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:25.876021 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-original-pull-secret\") pod \"global-pull-secret-syncer-blcxw\" (UID: \"ed7f6a91-3f6e-409f-bc57-b5d0f8e67770\") " pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:31:25.876190 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:25.876186 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:25.876428 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:25.876251 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-original-pull-secret podName:ed7f6a91-3f6e-409f-bc57-b5d0f8e67770 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:27.876233029 +0000 UTC m=+5.130506542 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-original-pull-secret") pod "global-pull-secret-syncer-blcxw" (UID: "ed7f6a91-3f6e-409f-bc57-b5d0f8e67770") : object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:25.876717 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:25.876694 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 12:31:25.876805 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:25.876723 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 12:31:25.876805 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:25.876737 2582 projected.go:196] Error preparing data for projected volume kube-api-access-qzs6g for pod openshift-network-diagnostics/network-check-target-7sf98: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:25.876805 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:25.876786 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ab66c79-2b0a-4671-9bf8-6aafea4e7454-kube-api-access-qzs6g podName:6ab66c79-2b0a-4671-9bf8-6aafea4e7454 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:27.876770001 +0000 UTC m=+5.131043525 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qzs6g" (UniqueName: "kubernetes.io/projected/6ab66c79-2b0a-4671-9bf8-6aafea4e7454-kube-api-access-qzs6g") pod "network-check-target-7sf98" (UID: "6ab66c79-2b0a-4671-9bf8-6aafea4e7454") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:25.876940 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:25.876854 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:25.876940 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:25.876893 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/856f4c46-f3c3-4afe-823e-1b77842932a4-metrics-certs podName:856f4c46-f3c3-4afe-823e-1b77842932a4 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:27.87688176 +0000 UTC m=+5.131155272 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/856f4c46-f3c3-4afe-823e-1b77842932a4-metrics-certs") pod "network-metrics-daemon-8bvmr" (UID: "856f4c46-f3c3-4afe-823e-1b77842932a4") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:26.225385 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:26.225309 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 12:26:25 +0000 UTC" deadline="2028-06-21 15:57:33.276762606 +0000 UTC" Oct 31 12:31:26.225385 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:26.225349 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14379h26m7.051418647s" Oct 31 12:31:26.289094 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:26.288741 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:31:26.289094 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:26.288949 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sf98" podUID="6ab66c79-2b0a-4671-9bf8-6aafea4e7454" Oct 31 12:31:26.289878 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:26.289458 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:31:26.289878 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:26.289650 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bvmr" podUID="856f4c46-f3c3-4afe-823e-1b77842932a4" Oct 31 12:31:27.289812 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:27.289777 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:31:27.298103 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:27.292266 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-blcxw" podUID="ed7f6a91-3f6e-409f-bc57-b5d0f8e67770" Oct 31 12:31:27.898350 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:27.898168 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzs6g\" (UniqueName: \"kubernetes.io/projected/6ab66c79-2b0a-4671-9bf8-6aafea4e7454-kube-api-access-qzs6g\") pod \"network-check-target-7sf98\" (UID: \"6ab66c79-2b0a-4671-9bf8-6aafea4e7454\") " pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:31:27.898350 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:27.898230 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/856f4c46-f3c3-4afe-823e-1b77842932a4-metrics-certs\") pod \"network-metrics-daemon-8bvmr\" (UID: \"856f4c46-f3c3-4afe-823e-1b77842932a4\") " pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:31:27.898350 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:27.898263 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-original-pull-secret\") pod \"global-pull-secret-syncer-blcxw\" (UID: \"ed7f6a91-3f6e-409f-bc57-b5d0f8e67770\") " pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:31:27.898627 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:27.898406 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:27.898627 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:27.898473 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-original-pull-secret podName:ed7f6a91-3f6e-409f-bc57-b5d0f8e67770 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:31.89845434 +0000 UTC m=+9.152727864 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-original-pull-secret") pod "global-pull-secret-syncer-blcxw" (UID: "ed7f6a91-3f6e-409f-bc57-b5d0f8e67770") : object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:27.899027 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:27.899006 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 12:31:27.899131 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:27.899036 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 12:31:27.899131 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:27.899050 2582 projected.go:196] Error preparing data for projected volume kube-api-access-qzs6g for pod openshift-network-diagnostics/network-check-target-7sf98: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:27.899131 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:27.899109 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ab66c79-2b0a-4671-9bf8-6aafea4e7454-kube-api-access-qzs6g podName:6ab66c79-2b0a-4671-9bf8-6aafea4e7454 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:31.899096458 +0000 UTC m=+9.153369980 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qzs6g" (UniqueName: "kubernetes.io/projected/6ab66c79-2b0a-4671-9bf8-6aafea4e7454-kube-api-access-qzs6g") pod "network-check-target-7sf98" (UID: "6ab66c79-2b0a-4671-9bf8-6aafea4e7454") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:27.899301 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:27.899175 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:27.899301 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:27.899211 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/856f4c46-f3c3-4afe-823e-1b77842932a4-metrics-certs podName:856f4c46-f3c3-4afe-823e-1b77842932a4 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:31.899198851 +0000 UTC m=+9.153472360 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/856f4c46-f3c3-4afe-823e-1b77842932a4-metrics-certs") pod "network-metrics-daemon-8bvmr" (UID: "856f4c46-f3c3-4afe-823e-1b77842932a4") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:28.289027 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:28.288283 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:31:28.289027 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:28.288452 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bvmr" podUID="856f4c46-f3c3-4afe-823e-1b77842932a4" Oct 31 12:31:28.289027 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:28.288871 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:31:28.289027 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:28.288972 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sf98" podUID="6ab66c79-2b0a-4671-9bf8-6aafea4e7454" Oct 31 12:31:29.290283 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:29.289144 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:31:29.290283 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:29.289513 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-blcxw" podUID="ed7f6a91-3f6e-409f-bc57-b5d0f8e67770" Oct 31 12:31:29.397089 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:29.394611 2582 generic.go:356] "Generic (PLEG): container finished" podID="f23a1c72-8427-4b6f-ad2b-cc76a88709c9" containerID="780ab5a0a4d2e2eb28ff2c5025383863f3ad065385433f87859d20140f5aac9f" exitCode=0 Oct 31 12:31:29.397089 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:29.394708 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-plzf9" event={"ID":"f23a1c72-8427-4b6f-ad2b-cc76a88709c9","Type":"ContainerDied","Data":"780ab5a0a4d2e2eb28ff2c5025383863f3ad065385433f87859d20140f5aac9f"} Oct 31 12:31:29.403656 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:29.403563 2582 generic.go:356] "Generic (PLEG): container finished" podID="62138b49cad483f3e9534fc842796713" containerID="87ed63087d2abd6a68e84ca1a585e203c250221bc2fe933f4f7aaa8069e2801f" exitCode=0 Oct 31 12:31:29.403656 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:29.403607 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal" event={"ID":"62138b49cad483f3e9534fc842796713","Type":"ContainerDied","Data":"87ed63087d2abd6a68e84ca1a585e203c250221bc2fe933f4f7aaa8069e2801f"} Oct 31 12:31:30.289199 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:30.289109 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:31:30.289369 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:30.289281 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bvmr" podUID="856f4c46-f3c3-4afe-823e-1b77842932a4" Oct 31 12:31:30.289860 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:30.289688 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:31:30.289860 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:30.289786 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sf98" podUID="6ab66c79-2b0a-4671-9bf8-6aafea4e7454" Oct 31 12:31:30.418998 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:30.418962 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-chf5k" event={"ID":"9fcef325-e046-4c6d-8599-6c67115f382c","Type":"ContainerStarted","Data":"ccca76596678de14221c659f18d4febb64e19409baabb67926d64def88085eff"} Oct 31 12:31:30.431915 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:30.431881 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2wtkj" event={"ID":"29bc026b-84aa-4cbf-ba44-bb5c499e6bdc","Type":"ContainerStarted","Data":"979dfc9dd818b5e081adb5cfe14cda3418b1e6bc83ca28aba718e730c9557184"} Oct 31 12:31:30.439500 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:30.439451 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-70.us-east-2.compute.internal" event={"ID":"578d51cfd2278b7149c7a4e11f7c55d2","Type":"ContainerStarted","Data":"e0ffd45d57c973c009fe120efa680e684f44c0512316092e2d8bc4b2881c35c9"} Oct 31 12:31:30.445083 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:30.444808 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal" event={"ID":"62138b49cad483f3e9534fc842796713","Type":"ContainerStarted","Data":"fdcb1dbcaa299f2d31b8a91fa7f00b1f7a70dc61717d74c315491fadfd2022d2"} Oct 31 12:31:30.462381 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:30.460810 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-chf5k" podStartSLOduration=1.017067591 podStartE2EDuration="6.460793853s" totalImagesPullingTime="5.443726262s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:31:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:30.438980448 +0000 UTC m=+7.693253981" watchObservedRunningTime="2026-10-31 12:31:30.460793853 +0000 UTC m=+7.715067386" Oct 31 12:31:30.482211 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:30.481665 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2wtkj" podStartSLOduration=1.626952935 podStartE2EDuration="6.481646014s" totalImagesPullingTime="4.854693079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:31:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:30.462146354 +0000 UTC m=+7.716419884" watchObservedRunningTime="2026-10-31 12:31:30.481646014 +0000 UTC m=+7.735919550" Oct 31 12:31:30.482211 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:30.481785 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal" podStartSLOduration=6.481779054 podStartE2EDuration="6.481779054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:31:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:30.48068146 +0000 UTC m=+7.734954994" watchObservedRunningTime="2026-10-31 12:31:30.481779054 +0000 UTC m=+7.736052586" Oct 31 12:31:31.291284 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:31.291250 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:31:31.291811 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:31.291422 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-blcxw" podUID="ed7f6a91-3f6e-409f-bc57-b5d0f8e67770" Oct 31 12:31:31.454675 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:31.454635 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" event={"ID":"d89912fc-3948-4569-bfea-87c8dceedd36","Type":"ContainerStarted","Data":"c83a21f4804f8d4489ac536e18bed3be05e5db9086bf6e994c82cf4473d75d7d"} Oct 31 12:31:31.935429 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:31.935378 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-original-pull-secret\") pod \"global-pull-secret-syncer-blcxw\" (UID: \"ed7f6a91-3f6e-409f-bc57-b5d0f8e67770\") " pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:31:31.935622 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:31.935480 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzs6g\" (UniqueName: \"kubernetes.io/projected/6ab66c79-2b0a-4671-9bf8-6aafea4e7454-kube-api-access-qzs6g\") pod \"network-check-target-7sf98\" (UID: \"6ab66c79-2b0a-4671-9bf8-6aafea4e7454\") " pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:31:31.935622 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:31.935534 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/856f4c46-f3c3-4afe-823e-1b77842932a4-metrics-certs\") pod \"network-metrics-daemon-8bvmr\" (UID: \"856f4c46-f3c3-4afe-823e-1b77842932a4\") " pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:31:31.935622 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:31.935571 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:31.935802 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:31.935648 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-original-pull-secret podName:ed7f6a91-3f6e-409f-bc57-b5d0f8e67770 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:39.935627667 +0000 UTC m=+17.189901174 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-original-pull-secret") pod "global-pull-secret-syncer-blcxw" (UID: "ed7f6a91-3f6e-409f-bc57-b5d0f8e67770") : object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:31.935802 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:31.935644 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:31.935802 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:31.935662 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 12:31:31.935802 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:31.935686 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 12:31:31.935802 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:31.935700 2582 projected.go:196] Error preparing data for projected volume kube-api-access-qzs6g for pod openshift-network-diagnostics/network-check-target-7sf98: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:31.935802 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:31.935689 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/856f4c46-f3c3-4afe-823e-1b77842932a4-metrics-certs podName:856f4c46-f3c3-4afe-823e-1b77842932a4 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:39.935679102 +0000 UTC m=+17.189952617 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/856f4c46-f3c3-4afe-823e-1b77842932a4-metrics-certs") pod "network-metrics-daemon-8bvmr" (UID: "856f4c46-f3c3-4afe-823e-1b77842932a4") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:31.935802 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:31.935767 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ab66c79-2b0a-4671-9bf8-6aafea4e7454-kube-api-access-qzs6g podName:6ab66c79-2b0a-4671-9bf8-6aafea4e7454 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:39.935748785 +0000 UTC m=+17.190022311 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qzs6g" (UniqueName: "kubernetes.io/projected/6ab66c79-2b0a-4671-9bf8-6aafea4e7454-kube-api-access-qzs6g") pod "network-check-target-7sf98" (UID: "6ab66c79-2b0a-4671-9bf8-6aafea4e7454") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:32.289316 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:32.288548 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:31:32.289316 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:32.288701 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bvmr" podUID="856f4c46-f3c3-4afe-823e-1b77842932a4" Oct 31 12:31:32.289316 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:32.289142 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:31:32.289316 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:32.289259 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sf98" podUID="6ab66c79-2b0a-4671-9bf8-6aafea4e7454" Oct 31 12:31:33.289375 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:33.289340 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:31:33.289821 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:33.289464 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-blcxw" podUID="ed7f6a91-3f6e-409f-bc57-b5d0f8e67770" Oct 31 12:31:33.461582 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:33.461547 2582 generic.go:356] "Generic (PLEG): container finished" podID="f23a1c72-8427-4b6f-ad2b-cc76a88709c9" containerID="e8f134b1aa5d56622a42f23af37aa6503b139b7177e7620fdc45dd62710bbab5" exitCode=0 Oct 31 12:31:33.461700 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:33.461659 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-plzf9" event={"ID":"f23a1c72-8427-4b6f-ad2b-cc76a88709c9","Type":"ContainerDied","Data":"e8f134b1aa5d56622a42f23af37aa6503b139b7177e7620fdc45dd62710bbab5"} Oct 31 12:31:33.465480 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:33.465445 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7p252" event={"ID":"4f51d774-d999-4957-a165-0346972d65bc","Type":"ContainerStarted","Data":"32ca005e33aa517675026ad400d20ec27ae54da44ff9da07245eaaec5091001f"} Oct 31 12:31:33.468836 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:33.468804 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rdcdb" event={"ID":"2b7d2569-9fcf-4e91-9e2f-42beb94593a8","Type":"ContainerStarted","Data":"1399ec289c2d9239b805f45e3dbfd7cb56ffbdbc1fb8d82f2458c481296f6d20"} Oct 31 12:31:33.475208 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:33.475187 2582 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 12:31:33.488833 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:33.488784 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-70.us-east-2.compute.internal" podStartSLOduration=9.488768866000001 podStartE2EDuration="9.488768866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:31:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:30.498093524 +0000 UTC m=+7.752367069" watchObservedRunningTime="2026-10-31 12:31:33.488768866 +0000 UTC m=+10.743042402" Oct 31 12:31:33.510192 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:33.510151 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-7p252" podStartSLOduration=0.937150376 podStartE2EDuration="9.510138317s" totalImagesPullingTime="8.572987941s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:31:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:33.509895095 +0000 UTC m=+10.764168626" watchObservedRunningTime="2026-10-31 12:31:33.510138317 +0000 UTC m=+10.764411848" Oct 31 12:31:33.530765 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:33.530725 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-rdcdb" podStartSLOduration=1.096146552 podStartE2EDuration="9.530713281s" totalImagesPullingTime="8.434566729s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:31:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:33.530644862 +0000 UTC m=+10.784918393" watchObservedRunningTime="2026-10-31 12:31:33.530713281 +0000 UTC m=+10.784986812" Oct 31 12:31:33.819643 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:33.819603 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:34.104017 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:34.103922 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:34.207861 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:34.207591 2582 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T12:31:33.475202396Z","UUID":"8ad2164d-07b0-4020-886a-3aa0a3349310","Handler":null,"Name":"","Endpoint":""} Oct 31 12:31:34.209557 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:34.209533 2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 12:31:34.209707 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:34.209568 2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 12:31:34.288278 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:34.288245 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:31:34.288448 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:34.288302 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:31:34.288448 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:34.288409 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bvmr" podUID="856f4c46-f3c3-4afe-823e-1b77842932a4" Oct 31 12:31:34.288579 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:34.288545 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sf98" podUID="6ab66c79-2b0a-4671-9bf8-6aafea4e7454" Oct 31 12:31:34.473617 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:34.473585 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" event={"ID":"d89912fc-3948-4569-bfea-87c8dceedd36","Type":"ContainerStarted","Data":"37a6587e0f136ffa756e66e9350ae61a145ec48e9d01a291503fe7dafaf6384e"} Oct 31 12:31:34.476150 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:34.476105 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-27p2l" event={"ID":"8087a5a6-949e-491f-98b9-3360b93a6419","Type":"ContainerStarted","Data":"ed0524530890326c6f6f815e13c467f6861ac3f1dc67b3cd83301d0ca3948899"} Oct 31 12:31:34.503396 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:34.503346 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-27p2l" podStartSLOduration=2.448894046 podStartE2EDuration="10.503332433s" totalImagesPullingTime="8.054438387s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:31:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:34.502723871 +0000 UTC m=+11.756997404" watchObservedRunningTime="2026-10-31 12:31:34.503332433 +0000 UTC m=+11.757605964" Oct 31 12:31:34.792763 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:34.792717 2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2wtkj" Oct 31 12:31:34.793530 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:34.793495 2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2wtkj" Oct 31 12:31:35.112546 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:35.112230 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:35.288286 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:35.288253 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:31:35.288464 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:35.288397 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-blcxw" podUID="ed7f6a91-3f6e-409f-bc57-b5d0f8e67770" Oct 31 12:31:35.333420 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:35.333385 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:35.480149 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:35.480115 2582 generic.go:356] "Generic (PLEG): container finished" podID="f23a1c72-8427-4b6f-ad2b-cc76a88709c9" containerID="e82d1cf1b7a57a0964a18b86e1a23532f19f760215e3e9dd233db273b50b8302" exitCode=0 Oct 31 12:31:35.480670 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:35.480223 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-plzf9" event={"ID":"f23a1c72-8427-4b6f-ad2b-cc76a88709c9","Type":"ContainerDied","Data":"e82d1cf1b7a57a0964a18b86e1a23532f19f760215e3e9dd233db273b50b8302"} Oct 31 12:31:36.288711 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:36.288644 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:31:36.288855 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:36.288771 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sf98" podUID="6ab66c79-2b0a-4671-9bf8-6aafea4e7454" Oct 31 12:31:36.289117 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:36.289099 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:31:36.289200 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:36.289183 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bvmr" podUID="856f4c46-f3c3-4afe-823e-1b77842932a4" Oct 31 12:31:36.485927 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:36.485892 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" event={"ID":"d89912fc-3948-4569-bfea-87c8dceedd36","Type":"ContainerStarted","Data":"0c892237327f3a04cc01792ddfb665e3a85ae646d2114e33f71e655fc3ad3a13"} Oct 31 12:31:36.508385 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:36.508328 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" podStartSLOduration=1.335106003 podStartE2EDuration="12.508314521s" totalImagesPullingTime="11.173208518s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:31:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:36.508042372 +0000 UTC m=+13.762315901" watchObservedRunningTime="2026-10-31 12:31:36.508314521 +0000 UTC m=+13.762588051" Oct 31 12:31:37.186152 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:37.186118 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:37.288476 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:37.288406 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:31:37.288605 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:37.288533 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-blcxw" podUID="ed7f6a91-3f6e-409f-bc57-b5d0f8e67770" Oct 31 12:31:37.387877 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:37.387842 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:37.489560 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:37.489524 2582 generic.go:356] "Generic (PLEG): container finished" podID="f23a1c72-8427-4b6f-ad2b-cc76a88709c9" containerID="f31c0fb13238d5ddf59f35a74a3e0a0329b2343a58b850d64e58191a9b4735c3" exitCode=0 Oct 31 12:31:37.490330 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:37.489592 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-plzf9" event={"ID":"f23a1c72-8427-4b6f-ad2b-cc76a88709c9","Type":"ContainerDied","Data":"f31c0fb13238d5ddf59f35a74a3e0a0329b2343a58b850d64e58191a9b4735c3"} Oct 31 12:31:37.491400 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:37.491244 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g98lz" event={"ID":"2b6cb291-3443-496d-8598-3b253102832f","Type":"ContainerStarted","Data":"ea399028b0bccaf25564fe66ec1ae7391049451919ba12fc1bb4c4cdab3e7ff5"} Oct 31 12:31:37.538853 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:37.538770 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-g98lz" podStartSLOduration=0.982949643 podStartE2EDuration="13.538757256s" totalImagesPullingTime="12.555807613s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:31:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:37.53832692 +0000 UTC m=+14.792600450" watchObservedRunningTime="2026-10-31 12:31:37.538757256 +0000 UTC m=+14.793030784" Oct 31 12:31:38.288367 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:38.288337 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:31:38.288543 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:38.288337 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:31:38.288543 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:38.288470 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bvmr" podUID="856f4c46-f3c3-4afe-823e-1b77842932a4" Oct 31 12:31:38.288659 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:38.288589 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sf98" podUID="6ab66c79-2b0a-4671-9bf8-6aafea4e7454" Oct 31 12:31:39.288266 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:39.288231 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:31:39.288736 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:39.288360 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-blcxw" podUID="ed7f6a91-3f6e-409f-bc57-b5d0f8e67770" Oct 31 12:31:39.651548 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:39.651467 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:39.846052 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:39.846017 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:39.997414 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:39.997386 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/856f4c46-f3c3-4afe-823e-1b77842932a4-metrics-certs\") pod \"network-metrics-daemon-8bvmr\" (UID: \"856f4c46-f3c3-4afe-823e-1b77842932a4\") " pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:31:39.997633 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:39.997421 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-original-pull-secret\") pod \"global-pull-secret-syncer-blcxw\" (UID: \"ed7f6a91-3f6e-409f-bc57-b5d0f8e67770\") " pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:31:39.997633 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:39.997467 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzs6g\" (UniqueName: \"kubernetes.io/projected/6ab66c79-2b0a-4671-9bf8-6aafea4e7454-kube-api-access-qzs6g\") pod \"network-check-target-7sf98\" (UID: \"6ab66c79-2b0a-4671-9bf8-6aafea4e7454\") " pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:31:39.997633 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:39.997528 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:39.997633 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:39.997573 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 12:31:39.997633 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:39.997579 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:39.997633 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:39.997591 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/856f4c46-f3c3-4afe-823e-1b77842932a4-metrics-certs podName:856f4c46-f3c3-4afe-823e-1b77842932a4 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:55.997574361 +0000 UTC m=+33.251847881 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/856f4c46-f3c3-4afe-823e-1b77842932a4-metrics-certs") pod "network-metrics-daemon-8bvmr" (UID: "856f4c46-f3c3-4afe-823e-1b77842932a4") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:39.997633 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:39.997594 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 12:31:39.997633 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:39.997611 2582 projected.go:196] Error preparing data for projected volume kube-api-access-qzs6g for pod openshift-network-diagnostics/network-check-target-7sf98: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:39.997633 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:39.997629 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-original-pull-secret podName:ed7f6a91-3f6e-409f-bc57-b5d0f8e67770 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:55.997613772 +0000 UTC m=+33.251887303 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-original-pull-secret") pod "global-pull-secret-syncer-blcxw" (UID: "ed7f6a91-3f6e-409f-bc57-b5d0f8e67770") : object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:39.998089 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:39.997646 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ab66c79-2b0a-4671-9bf8-6aafea4e7454-kube-api-access-qzs6g podName:6ab66c79-2b0a-4671-9bf8-6aafea4e7454 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:55.997637209 +0000 UTC m=+33.251910721 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qzs6g" (UniqueName: "kubernetes.io/projected/6ab66c79-2b0a-4671-9bf8-6aafea4e7454-kube-api-access-qzs6g") pod "network-check-target-7sf98" (UID: "6ab66c79-2b0a-4671-9bf8-6aafea4e7454") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:40.288094 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:40.288023 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:31:40.288094 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:40.288052 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:31:40.288263 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:40.288132 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sf98" podUID="6ab66c79-2b0a-4671-9bf8-6aafea4e7454" Oct 31 12:31:40.288322 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:40.288271 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bvmr" podUID="856f4c46-f3c3-4afe-823e-1b77842932a4" Oct 31 12:31:40.501089 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:40.501057 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-plzf9" event={"ID":"f23a1c72-8427-4b6f-ad2b-cc76a88709c9","Type":"ContainerStarted","Data":"d85c9a834a2f716d2344f49a91500854101fd89a1168b80c86dbc48d5e5f09e6"} Oct 31 12:31:40.614297 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:40.614272 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2wtkj" Oct 31 12:31:40.614432 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:40.614410 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 12:31:40.614968 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:40.614944 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2wtkj" Oct 31 12:31:40.833644 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:40.833555 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:41.049056 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:41.049022 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:41.256274 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:41.256232 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:41.288821 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:41.288788 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:31:41.289287 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:41.289000 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-blcxw" podUID="ed7f6a91-3f6e-409f-bc57-b5d0f8e67770" Oct 31 12:31:41.507172 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:41.507090 2582 generic.go:356] "Generic (PLEG): container finished" podID="f23a1c72-8427-4b6f-ad2b-cc76a88709c9" containerID="d85c9a834a2f716d2344f49a91500854101fd89a1168b80c86dbc48d5e5f09e6" exitCode=0 Oct 31 12:31:41.507323 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:41.507215 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-plzf9" event={"ID":"f23a1c72-8427-4b6f-ad2b-cc76a88709c9","Type":"ContainerDied","Data":"d85c9a834a2f716d2344f49a91500854101fd89a1168b80c86dbc48d5e5f09e6"} Oct 31 12:31:41.574972 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:41.574901 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:41.784338 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:41.784284 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7p252_4f51d774-d999-4957-a165-0346972d65bc/dns-node-resolver/0.log" Oct 31 12:31:41.785796 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:41.785759 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:41.987946 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:41.987907 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:42.288239 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:42.288208 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:31:42.288384 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:42.288245 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:31:42.288384 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:42.288306 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sf98" podUID="6ab66c79-2b0a-4671-9bf8-6aafea4e7454" Oct 31 12:31:42.288476 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:42.288420 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bvmr" podUID="856f4c46-f3c3-4afe-823e-1b77842932a4" Oct 31 12:31:42.512952 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:42.512904 2582 generic.go:356] "Generic (PLEG): container finished" podID="f23a1c72-8427-4b6f-ad2b-cc76a88709c9" containerID="faf4539a6b9874b2afb3379486b68d8d3171b59b8f76a0951adb6c948e9454ec" exitCode=0 Oct 31 12:31:42.513391 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:42.512959 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-plzf9" event={"ID":"f23a1c72-8427-4b6f-ad2b-cc76a88709c9","Type":"ContainerDied","Data":"faf4539a6b9874b2afb3379486b68d8d3171b59b8f76a0951adb6c948e9454ec"} Oct 31 12:31:42.586184 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:42.586166 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-chf5k_9fcef325-e046-4c6d-8599-6c67115f382c/node-ca/0.log" Oct 31 12:31:43.153129 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:43.153097 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:43.289070 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:43.289038 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:31:43.289215 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:43.289156 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-blcxw" podUID="ed7f6a91-3f6e-409f-bc57-b5d0f8e67770" Oct 31 12:31:43.519245 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:43.519210 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-plzf9" event={"ID":"f23a1c72-8427-4b6f-ad2b-cc76a88709c9","Type":"ContainerStarted","Data":"7bc9606019058067504b8dcb7d4c04bc84ba0fd58b0587d392dceb6f99127b63"} Oct 31 12:31:43.551575 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:43.551530 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-plzf9" podStartSLOduration=8.206343857 podStartE2EDuration="19.551515812s" totalImagesPullingTime="11.345171955s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:31:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:43.549470603 +0000 UTC m=+20.803744131" watchObservedRunningTime="2026-10-31 12:31:43.551515812 +0000 UTC m=+20.805789340" Oct 31 12:31:43.602842 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:43.602811 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:43.839195 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:43.839111 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:44.046320 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:44.046283 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:44.253630 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:44.253596 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:44.288626 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:44.288597 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:31:44.288773 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:44.288712 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sf98" podUID="6ab66c79-2b0a-4671-9bf8-6aafea4e7454" Oct 31 12:31:44.288835 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:44.288767 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:31:44.288889 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:44.288872 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bvmr" podUID="856f4c46-f3c3-4afe-823e-1b77842932a4" Oct 31 12:31:44.965091 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:44.965047 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:45.162741 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:45.162708 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:45.291976 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:45.291907 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:31:45.292142 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:45.292048 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-blcxw" podUID="ed7f6a91-3f6e-409f-bc57-b5d0f8e67770" Oct 31 12:31:45.446056 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:45.446019 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:45.645833 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:45.645715 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:45.870838 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:45.870803 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:46.248263 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:46.248224 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:46.288183 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:46.288151 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:31:46.288348 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:46.288152 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:31:46.288348 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:46.288291 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bvmr" podUID="856f4c46-f3c3-4afe-823e-1b77842932a4" Oct 31 12:31:46.288450 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:46.288353 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sf98" podUID="6ab66c79-2b0a-4671-9bf8-6aafea4e7454" Oct 31 12:31:46.447389 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:46.447365 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:46.526837 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:46.526802 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" event={"ID":"ca9b156d-7499-4b53-b17f-01dc63ae0c5e","Type":"ContainerStarted","Data":"d549ba46253b9172b718622c0d1cb69e94dd833a1d09c3b0e91cf035cc2b8ac1"} Oct 31 12:31:46.661684 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:46.661519 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:47.066719 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:47.066686 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:47.288363 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:47.288333 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:31:47.288745 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:47.288460 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-blcxw" podUID="ed7f6a91-3f6e-409f-bc57-b5d0f8e67770" Oct 31 12:31:47.440756 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:47.440679 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:47.532288 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:47.532245 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" event={"ID":"ca9b156d-7499-4b53-b17f-01dc63ae0c5e","Type":"ContainerStarted","Data":"363b570c0e18151f431821b306d9c96de5bb33b50ce51d045c5b77767bfc2bc5"} Oct 31 12:31:47.532466 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:47.532421 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" event={"ID":"ca9b156d-7499-4b53-b17f-01dc63ae0c5e","Type":"ContainerStarted","Data":"f53b9d174075f69d8aa2b06707c0e9ca3ebbe21e2bd222e28bb9f7cbefa4e34f"} Oct 31 12:31:47.532526 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:47.532505 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" event={"ID":"ca9b156d-7499-4b53-b17f-01dc63ae0c5e","Type":"ContainerStarted","Data":"3aba773738dcf2c2460235167dff262a5207e50234bd740742d18a458475e0ba"} Oct 31 12:31:47.532581 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:47.532573 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" event={"ID":"ca9b156d-7499-4b53-b17f-01dc63ae0c5e","Type":"ContainerStarted","Data":"cfe3410c86c5a0d84abb42d9cfcd826df3eccf9876fe5d26bb9a1bd356acd1de"} Oct 31 12:31:47.532649 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:47.532634 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" event={"ID":"ca9b156d-7499-4b53-b17f-01dc63ae0c5e","Type":"ContainerStarted","Data":"86f49f604f5a62cb61e505524817aba349e7a218b7c69d535aded7f58010f8a3"} Oct 31 12:31:47.651046 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:47.651010 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:47.850919 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:47.850879 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:48.277114 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:48.277078 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:48.288362 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:48.288337 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:31:48.288495 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:48.288471 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bvmr" podUID="856f4c46-f3c3-4afe-823e-1b77842932a4" Oct 31 12:31:48.288824 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:48.288515 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:31:48.288824 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:48.288578 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sf98" podUID="6ab66c79-2b0a-4671-9bf8-6aafea4e7454" Oct 31 12:31:49.048628 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:49.048544 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:49.242392 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:49.242356 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:49.288138 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:49.288115 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:31:49.288267 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:49.288208 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-blcxw" podUID="ed7f6a91-3f6e-409f-bc57-b5d0f8e67770" Oct 31 12:31:49.540020 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:49.539972 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" event={"ID":"ca9b156d-7499-4b53-b17f-01dc63ae0c5e","Type":"ContainerStarted","Data":"4b4bd2ea82614ae256d5304d540bf81935cdeca9a83f2a344b5e02825db37474"} Oct 31 12:31:49.710044 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:49.710015 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:49.908371 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:49.908295 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:50.191297 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:50.191220 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:50.288259 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:50.288234 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:31:50.288406 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:50.288334 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bvmr" podUID="856f4c46-f3c3-4afe-823e-1b77842932a4" Oct 31 12:31:50.288406 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:50.288389 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:31:50.288482 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:50.288460 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sf98" podUID="6ab66c79-2b0a-4671-9bf8-6aafea4e7454" Oct 31 12:31:50.388545 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:50.388513 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:50.585097 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:50.585058 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:50.779616 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:50.779587 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:50.980683 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:50.980647 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:51.247798 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:51.247769 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:51.288938 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:51.288915 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:31:51.289082 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:51.289019 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-blcxw" podUID="ed7f6a91-3f6e-409f-bc57-b5d0f8e67770" Oct 31 12:31:51.450193 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:51.450154 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:51.551045 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:51.550953 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" event={"ID":"ca9b156d-7499-4b53-b17f-01dc63ae0c5e","Type":"ContainerStarted","Data":"05d6ff421e333674a67ef6608559ca3e263055d3d6a2315ed0bb8bc27273d5d0"} Oct 31 12:31:51.551300 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:51.551283 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:51.551351 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:51.551345 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:51.551401 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:51.551390 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:51.566225 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:51.566204 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:51.566354 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:51.566301 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:31:51.586662 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:51.586619 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" podStartSLOduration=5.756429492 podStartE2EDuration="27.586609672s" totalImagesPullingTime="21.83018018s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:31:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:51.585679738 +0000 UTC m=+28.839953265" watchObservedRunningTime="2026-10-31 12:31:51.586609672 +0000 UTC m=+28.840883200" Oct 31 12:31:52.044142 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:52.044113 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:52.288538 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:52.288504 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:31:52.288725 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:52.288544 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:31:52.288725 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:52.288610 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bvmr" podUID="856f4c46-f3c3-4afe-823e-1b77842932a4" Oct 31 12:31:52.288822 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:52.288740 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sf98" podUID="6ab66c79-2b0a-4671-9bf8-6aafea4e7454" Oct 31 12:31:52.347512 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:52.347427 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:52.646588 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:52.646510 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:52.843821 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:52.843783 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:53.040357 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:53.040327 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:53.248236 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:53.248196 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:53.289439 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:53.289416 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:31:53.289549 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:53.289498 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-blcxw" podUID="ed7f6a91-3f6e-409f-bc57-b5d0f8e67770" Oct 31 12:31:53.450903 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:53.450864 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:53.654743 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:53.654709 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:53.848334 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:53.848232 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:54.043035 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:54.043002 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:54.243757 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:54.243726 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:54.288728 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:54.288705 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:31:54.288855 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:54.288733 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:31:54.288855 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:54.288805 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bvmr" podUID="856f4c46-f3c3-4afe-823e-1b77842932a4" Oct 31 12:31:54.288936 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:54.288920 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sf98" podUID="6ab66c79-2b0a-4671-9bf8-6aafea4e7454" Oct 31 12:31:54.517352 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:54.517276 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:55.288667 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:55.288634 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:31:55.289195 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:55.288728 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-blcxw" podUID="ed7f6a91-3f6e-409f-bc57-b5d0f8e67770" Oct 31 12:31:55.352821 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:55.352783 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:56.013402 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:56.013375 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzs6g\" (UniqueName: \"kubernetes.io/projected/6ab66c79-2b0a-4671-9bf8-6aafea4e7454-kube-api-access-qzs6g\") pod \"network-check-target-7sf98\" (UID: \"6ab66c79-2b0a-4671-9bf8-6aafea4e7454\") " pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:31:56.013402 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:56.013407 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/856f4c46-f3c3-4afe-823e-1b77842932a4-metrics-certs\") pod \"network-metrics-daemon-8bvmr\" (UID: \"856f4c46-f3c3-4afe-823e-1b77842932a4\") " pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:31:56.013602 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:56.013430 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-original-pull-secret\") pod \"global-pull-secret-syncer-blcxw\" (UID: \"ed7f6a91-3f6e-409f-bc57-b5d0f8e67770\") " pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:31:56.013602 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:56.013518 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:56.013602 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:56.013563 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-original-pull-secret podName:ed7f6a91-3f6e-409f-bc57-b5d0f8e67770 nodeName:}" failed. No retries permitted until 2026-10-31 12:32:28.013551126 +0000 UTC m=+65.267824632 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-original-pull-secret") pod "global-pull-secret-syncer-blcxw" (UID: "ed7f6a91-3f6e-409f-bc57-b5d0f8e67770") : object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:56.013602 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:56.013519 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:56.013821 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:56.013632 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/856f4c46-f3c3-4afe-823e-1b77842932a4-metrics-certs podName:856f4c46-f3c3-4afe-823e-1b77842932a4 nodeName:}" failed. No retries permitted until 2026-10-31 12:32:28.013618918 +0000 UTC m=+65.267892423 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/856f4c46-f3c3-4afe-823e-1b77842932a4-metrics-certs") pod "network-metrics-daemon-8bvmr" (UID: "856f4c46-f3c3-4afe-823e-1b77842932a4") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:56.013821 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:56.013521 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 12:31:56.013821 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:56.013651 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 12:31:56.013821 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:56.013660 2582 projected.go:196] Error preparing data for projected volume kube-api-access-qzs6g for pod openshift-network-diagnostics/network-check-target-7sf98: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:56.013821 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:56.013709 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ab66c79-2b0a-4671-9bf8-6aafea4e7454-kube-api-access-qzs6g podName:6ab66c79-2b0a-4671-9bf8-6aafea4e7454 nodeName:}" failed. No retries permitted until 2026-10-31 12:32:28.013698848 +0000 UTC m=+65.267972367 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qzs6g" (UniqueName: "kubernetes.io/projected/6ab66c79-2b0a-4671-9bf8-6aafea4e7454-kube-api-access-qzs6g") pod "network-check-target-7sf98" (UID: "6ab66c79-2b0a-4671-9bf8-6aafea4e7454") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:56.288817 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:56.288744 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:31:56.289313 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:56.288857 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bvmr" podUID="856f4c46-f3c3-4afe-823e-1b77842932a4" Oct 31 12:31:56.289313 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:56.288918 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:31:56.289313 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:56.289046 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sf98" podUID="6ab66c79-2b0a-4671-9bf8-6aafea4e7454" Oct 31 12:31:57.288824 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:57.288796 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:31:57.289410 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:57.288918 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-blcxw" podUID="ed7f6a91-3f6e-409f-bc57-b5d0f8e67770" Oct 31 12:31:58.288679 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:58.288650 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:31:58.288853 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:58.288761 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bvmr" podUID="856f4c46-f3c3-4afe-823e-1b77842932a4" Oct 31 12:31:58.288853 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:58.288814 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:31:58.289196 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:58.288907 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sf98" podUID="6ab66c79-2b0a-4671-9bf8-6aafea4e7454" Oct 31 12:31:59.288822 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:59.288785 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:31:59.289021 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:31:59.288926 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-blcxw" podUID="ed7f6a91-3f6e-409f-bc57-b5d0f8e67770" Oct 31 12:31:59.678293 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:59.678212 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:31:59.869888 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:31:59.869853 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:00.104791 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:00.104763 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:00.288923 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:00.288890 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:32:00.288923 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:00.288926 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:32:00.289347 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:32:00.289025 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bvmr" podUID="856f4c46-f3c3-4afe-823e-1b77842932a4" Oct 31 12:32:00.289347 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:32:00.289142 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sf98" podUID="6ab66c79-2b0a-4671-9bf8-6aafea4e7454" Oct 31 12:32:00.313995 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:00.313958 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:00.510852 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:00.510822 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:01.288016 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:01.287968 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:32:01.288186 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:32:01.288100 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-blcxw" podUID="ed7f6a91-3f6e-409f-bc57-b5d0f8e67770" Oct 31 12:32:02.288159 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:02.288126 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:32:02.288541 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:32:02.288236 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bvmr" podUID="856f4c46-f3c3-4afe-823e-1b77842932a4" Oct 31 12:32:02.288541 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:02.288292 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:32:02.288541 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:32:02.288406 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sf98" podUID="6ab66c79-2b0a-4671-9bf8-6aafea4e7454" Oct 31 12:32:02.401447 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:02.401408 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:02.594318 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:02.594248 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:03.289499 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:03.289473 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:32:03.289858 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:32:03.289554 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-blcxw" podUID="ed7f6a91-3f6e-409f-bc57-b5d0f8e67770" Oct 31 12:32:04.288477 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:04.288452 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:32:04.288640 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:32:04.288550 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bvmr" podUID="856f4c46-f3c3-4afe-823e-1b77842932a4" Oct 31 12:32:04.288640 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:04.288601 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:32:04.288722 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:32:04.288677 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sf98" podUID="6ab66c79-2b0a-4671-9bf8-6aafea4e7454" Oct 31 12:32:05.186213 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:05.186176 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:05.288838 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:05.288803 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:32:05.289036 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:32:05.288936 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-blcxw" podUID="ed7f6a91-3f6e-409f-bc57-b5d0f8e67770" Oct 31 12:32:05.439467 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:05.439377 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:05.667266 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:05.667224 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:05.866553 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:05.866511 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:06.288170 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:06.288132 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:32:06.288589 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:32:06.288251 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bvmr" podUID="856f4c46-f3c3-4afe-823e-1b77842932a4" Oct 31 12:32:06.288589 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:06.288281 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:32:06.288589 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:32:06.288339 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sf98" podUID="6ab66c79-2b0a-4671-9bf8-6aafea4e7454" Oct 31 12:32:06.664395 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:06.664307 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-blcxw"] Oct 31 12:32:06.664395 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:06.664394 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7sf98"] Oct 31 12:32:06.664641 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:06.664473 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:32:06.664641 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:32:06.664577 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sf98" podUID="6ab66c79-2b0a-4671-9bf8-6aafea4e7454" Oct 31 12:32:06.664847 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:06.664808 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:32:06.664922 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:06.664855 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8bvmr"] Oct 31 12:32:06.664922 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:32:06.664906 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-blcxw" podUID="ed7f6a91-3f6e-409f-bc57-b5d0f8e67770" Oct 31 12:32:06.665013 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:06.664953 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:32:06.665079 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:32:06.665060 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bvmr" podUID="856f4c46-f3c3-4afe-823e-1b77842932a4" Oct 31 12:32:08.288609 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:08.288579 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:32:08.288609 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:08.288599 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:32:08.289052 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:08.288587 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:32:08.289052 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:32:08.288689 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-blcxw" podUID="ed7f6a91-3f6e-409f-bc57-b5d0f8e67770" Oct 31 12:32:08.289052 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:32:08.288743 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sf98" podUID="6ab66c79-2b0a-4671-9bf8-6aafea4e7454" Oct 31 12:32:08.289052 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:32:08.288844 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bvmr" podUID="856f4c46-f3c3-4afe-823e-1b77842932a4" Oct 31 12:32:10.097954 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.097916 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:10.287901 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.287872 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:32:10.288078 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:32:10.287975 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sf98" podUID="6ab66c79-2b0a-4671-9bf8-6aafea4e7454" Oct 31 12:32:10.288078 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.288063 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:32:10.288172 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:32:10.288156 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bvmr" podUID="856f4c46-f3c3-4afe-823e-1b77842932a4" Oct 31 12:32:10.288239 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.288226 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:32:10.288300 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:32:10.288288 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-blcxw" podUID="ed7f6a91-3f6e-409f-bc57-b5d0f8e67770" Oct 31 12:32:10.299703 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.299673 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:10.575053 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.575018 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:10.580552 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.580534 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-70.us-east-2.compute.internal" event="NodeReady" Oct 31 12:32:10.580666 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.580653 2582 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 12:32:10.626456 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.626432 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7p252_4f51d774-d999-4957-a165-0346972d65bc/dns-node-resolver/0.log" Oct 31 12:32:10.643466 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.643440 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tm5c4"] Oct 31 12:32:10.646386 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.646371 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tm5c4" Oct 31 12:32:10.651000 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.650961 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4kn7k"] Oct 31 12:32:10.653588 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.653570 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4kn7k" Oct 31 12:32:10.658769 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.658748 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 12:32:10.659825 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.659809 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 12:32:10.659939 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.659919 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 12:32:10.660050 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.659952 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 12:32:10.660050 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.659965 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mrn76\"" Oct 31 12:32:10.660050 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.660013 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nthx6\"" Oct 31 12:32:10.660167 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.660050 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 12:32:10.668360 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.668338 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tm5c4"] Oct 31 12:32:10.670172 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.670149 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4kn7k"] Oct 31 12:32:10.678582 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.678560 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9txq2"] Oct 31 12:32:10.681545 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.681529 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9txq2" Oct 31 12:32:10.692785 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.692763 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9txq2"] Oct 31 12:32:10.721390 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.721369 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e9e8e847-5b80-44de-9450-c5ae29a5b57b-crio-socket\") pod \"insights-runtime-extractor-9txq2\" (UID: \"e9e8e847-5b80-44de-9450-c5ae29a5b57b\") " pod="openshift-insights/insights-runtime-extractor-9txq2" Oct 31 12:32:10.721518 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.721494 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e9e8e847-5b80-44de-9450-c5ae29a5b57b-data-volume\") pod \"insights-runtime-extractor-9txq2\" (UID: \"e9e8e847-5b80-44de-9450-c5ae29a5b57b\") " pod="openshift-insights/insights-runtime-extractor-9txq2" Oct 31 12:32:10.721593 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.721527 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-865nz\" (UniqueName: \"kubernetes.io/projected/178dd4e3-f913-4638-b4c7-8c9e96c82053-kube-api-access-865nz\") pod \"ingress-canary-tm5c4\" (UID: \"178dd4e3-f913-4638-b4c7-8c9e96c82053\") " pod="openshift-ingress-canary/ingress-canary-tm5c4" Oct 31 12:32:10.721651 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.721603 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5dd720ed-3a46-454c-80c2-4850993e6b76-metrics-tls\") pod \"dns-default-4kn7k\" (UID: \"5dd720ed-3a46-454c-80c2-4850993e6b76\") " pod="openshift-dns/dns-default-4kn7k" Oct 31 12:32:10.721651 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.721643 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnpbz\" (UniqueName: \"kubernetes.io/projected/e9e8e847-5b80-44de-9450-c5ae29a5b57b-kube-api-access-bnpbz\") pod \"insights-runtime-extractor-9txq2\" (UID: \"e9e8e847-5b80-44de-9450-c5ae29a5b57b\") " pod="openshift-insights/insights-runtime-extractor-9txq2" Oct 31 12:32:10.721728 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.721670 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9e8e847-5b80-44de-9450-c5ae29a5b57b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9txq2\" (UID: \"e9e8e847-5b80-44de-9450-c5ae29a5b57b\") " pod="openshift-insights/insights-runtime-extractor-9txq2" Oct 31 12:32:10.721728 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.721713 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/178dd4e3-f913-4638-b4c7-8c9e96c82053-cert\") pod \"ingress-canary-tm5c4\" (UID: \"178dd4e3-f913-4638-b4c7-8c9e96c82053\") " pod="openshift-ingress-canary/ingress-canary-tm5c4" Oct 31 12:32:10.721788 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.721748 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5dd720ed-3a46-454c-80c2-4850993e6b76-config-volume\") pod \"dns-default-4kn7k\" (UID: \"5dd720ed-3a46-454c-80c2-4850993e6b76\") " pod="openshift-dns/dns-default-4kn7k" Oct 31 12:32:10.721861 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.721843 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9e8e847-5b80-44de-9450-c5ae29a5b57b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9txq2\" (UID: \"e9e8e847-5b80-44de-9450-c5ae29a5b57b\") " pod="openshift-insights/insights-runtime-extractor-9txq2" Oct 31 12:32:10.721897 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.721874 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dd720ed-3a46-454c-80c2-4850993e6b76-tmp-dir\") pod \"dns-default-4kn7k\" (UID: \"5dd720ed-3a46-454c-80c2-4850993e6b76\") " pod="openshift-dns/dns-default-4kn7k" Oct 31 12:32:10.721933 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.721917 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5sww\" (UniqueName: \"kubernetes.io/projected/5dd720ed-3a46-454c-80c2-4850993e6b76-kube-api-access-t5sww\") pod \"dns-default-4kn7k\" (UID: \"5dd720ed-3a46-454c-80c2-4850993e6b76\") " pod="openshift-dns/dns-default-4kn7k" Oct 31 12:32:10.756707 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.756687 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-52dg7\"" Oct 31 12:32:10.756818 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.756733 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 12:32:10.759203 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.759186 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 12:32:10.759203 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.759193 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 12:32:10.759396 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.759382 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 12:32:10.788104 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.788076 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:10.823134 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.823111 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e9e8e847-5b80-44de-9450-c5ae29a5b57b-crio-socket\") pod \"insights-runtime-extractor-9txq2\" (UID: \"e9e8e847-5b80-44de-9450-c5ae29a5b57b\") " pod="openshift-insights/insights-runtime-extractor-9txq2" Oct 31 12:32:10.823239 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.823224 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e9e8e847-5b80-44de-9450-c5ae29a5b57b-crio-socket\") pod \"insights-runtime-extractor-9txq2\" (UID: \"e9e8e847-5b80-44de-9450-c5ae29a5b57b\") " pod="openshift-insights/insights-runtime-extractor-9txq2" Oct 31 12:32:10.823275 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.823255 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e9e8e847-5b80-44de-9450-c5ae29a5b57b-data-volume\") pod \"insights-runtime-extractor-9txq2\" (UID: \"e9e8e847-5b80-44de-9450-c5ae29a5b57b\") " pod="openshift-insights/insights-runtime-extractor-9txq2" Oct 31 12:32:10.823306 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.823276 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-865nz\" (UniqueName: \"kubernetes.io/projected/178dd4e3-f913-4638-b4c7-8c9e96c82053-kube-api-access-865nz\") pod \"ingress-canary-tm5c4\" (UID: \"178dd4e3-f913-4638-b4c7-8c9e96c82053\") " pod="openshift-ingress-canary/ingress-canary-tm5c4" Oct 31 12:32:10.823306 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.823291 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5dd720ed-3a46-454c-80c2-4850993e6b76-metrics-tls\") pod \"dns-default-4kn7k\" (UID: \"5dd720ed-3a46-454c-80c2-4850993e6b76\") " pod="openshift-dns/dns-default-4kn7k" Oct 31 12:32:10.823371 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.823309 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnpbz\" (UniqueName: \"kubernetes.io/projected/e9e8e847-5b80-44de-9450-c5ae29a5b57b-kube-api-access-bnpbz\") pod \"insights-runtime-extractor-9txq2\" (UID: \"e9e8e847-5b80-44de-9450-c5ae29a5b57b\") " pod="openshift-insights/insights-runtime-extractor-9txq2" Oct 31 12:32:10.823470 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.823446 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9e8e847-5b80-44de-9450-c5ae29a5b57b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9txq2\" (UID: \"e9e8e847-5b80-44de-9450-c5ae29a5b57b\") " pod="openshift-insights/insights-runtime-extractor-9txq2" Oct 31 12:32:10.823525 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.823490 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/178dd4e3-f913-4638-b4c7-8c9e96c82053-cert\") pod \"ingress-canary-tm5c4\" (UID: \"178dd4e3-f913-4638-b4c7-8c9e96c82053\") " pod="openshift-ingress-canary/ingress-canary-tm5c4" Oct 31 12:32:10.823525 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.823514 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5dd720ed-3a46-454c-80c2-4850993e6b76-config-volume\") pod \"dns-default-4kn7k\" (UID: \"5dd720ed-3a46-454c-80c2-4850993e6b76\") " pod="openshift-dns/dns-default-4kn7k" Oct 31 12:32:10.823616 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.823546 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9e8e847-5b80-44de-9450-c5ae29a5b57b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9txq2\" (UID: \"e9e8e847-5b80-44de-9450-c5ae29a5b57b\") " pod="openshift-insights/insights-runtime-extractor-9txq2" Oct 31 12:32:10.823616 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.823575 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dd720ed-3a46-454c-80c2-4850993e6b76-tmp-dir\") pod \"dns-default-4kn7k\" (UID: \"5dd720ed-3a46-454c-80c2-4850993e6b76\") " pod="openshift-dns/dns-default-4kn7k" Oct 31 12:32:10.823616 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.823599 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5sww\" (UniqueName: \"kubernetes.io/projected/5dd720ed-3a46-454c-80c2-4850993e6b76-kube-api-access-t5sww\") pod \"dns-default-4kn7k\" (UID: \"5dd720ed-3a46-454c-80c2-4850993e6b76\") " pod="openshift-dns/dns-default-4kn7k" Oct 31 12:32:10.823758 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.823656 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e9e8e847-5b80-44de-9450-c5ae29a5b57b-data-volume\") pod \"insights-runtime-extractor-9txq2\" (UID: \"e9e8e847-5b80-44de-9450-c5ae29a5b57b\") " pod="openshift-insights/insights-runtime-extractor-9txq2" Oct 31 12:32:10.824001 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.823961 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dd720ed-3a46-454c-80c2-4850993e6b76-tmp-dir\") pod \"dns-default-4kn7k\" (UID: \"5dd720ed-3a46-454c-80c2-4850993e6b76\") " pod="openshift-dns/dns-default-4kn7k" Oct 31 12:32:10.824171 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.824149 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5dd720ed-3a46-454c-80c2-4850993e6b76-config-volume\") pod \"dns-default-4kn7k\" (UID: \"5dd720ed-3a46-454c-80c2-4850993e6b76\") " pod="openshift-dns/dns-default-4kn7k" Oct 31 12:32:10.824227 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.824156 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9e8e847-5b80-44de-9450-c5ae29a5b57b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9txq2\" (UID: \"e9e8e847-5b80-44de-9450-c5ae29a5b57b\") " pod="openshift-insights/insights-runtime-extractor-9txq2" Oct 31 12:32:10.827209 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.827188 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9e8e847-5b80-44de-9450-c5ae29a5b57b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9txq2\" (UID: \"e9e8e847-5b80-44de-9450-c5ae29a5b57b\") " pod="openshift-insights/insights-runtime-extractor-9txq2" Oct 31 12:32:10.827274 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.827188 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5dd720ed-3a46-454c-80c2-4850993e6b76-metrics-tls\") pod \"dns-default-4kn7k\" (UID: \"5dd720ed-3a46-454c-80c2-4850993e6b76\") " pod="openshift-dns/dns-default-4kn7k" Oct 31 12:32:10.827274 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.827229 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/178dd4e3-f913-4638-b4c7-8c9e96c82053-cert\") pod \"ingress-canary-tm5c4\" (UID: \"178dd4e3-f913-4638-b4c7-8c9e96c82053\") " pod="openshift-ingress-canary/ingress-canary-tm5c4" Oct 31 12:32:10.837367 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.837348 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5sww\" (UniqueName: \"kubernetes.io/projected/5dd720ed-3a46-454c-80c2-4850993e6b76-kube-api-access-t5sww\") pod \"dns-default-4kn7k\" (UID: \"5dd720ed-3a46-454c-80c2-4850993e6b76\") " pod="openshift-dns/dns-default-4kn7k" Oct 31 12:32:10.837501 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.837480 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-865nz\" (UniqueName: \"kubernetes.io/projected/178dd4e3-f913-4638-b4c7-8c9e96c82053-kube-api-access-865nz\") pod \"ingress-canary-tm5c4\" (UID: \"178dd4e3-f913-4638-b4c7-8c9e96c82053\") " pod="openshift-ingress-canary/ingress-canary-tm5c4" Oct 31 12:32:10.838973 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.838924 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnpbz\" (UniqueName: \"kubernetes.io/projected/e9e8e847-5b80-44de-9450-c5ae29a5b57b-kube-api-access-bnpbz\") pod \"insights-runtime-extractor-9txq2\" (UID: \"e9e8e847-5b80-44de-9450-c5ae29a5b57b\") " pod="openshift-insights/insights-runtime-extractor-9txq2" Oct 31 12:32:10.957563 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.957541 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tm5c4" Oct 31 12:32:10.966544 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.966515 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4kn7k" Oct 31 12:32:10.995389 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:10.995358 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9txq2" Oct 31 12:32:11.249283 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:11.249217 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tm5c4"] Oct 31 12:32:11.252190 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:11.252172 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4kn7k"] Oct 31 12:32:11.252392 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:32:11.252360 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod178dd4e3_f913_4638_b4c7_8c9e96c82053.slice/crio-4d9d1b5cbd5ebbbc65fc88798331a76530c991bd2afc75059b31e8c8f1e4fbdb WatchSource:0}: Error finding container 4d9d1b5cbd5ebbbc65fc88798331a76530c991bd2afc75059b31e8c8f1e4fbdb: Status 404 returned error can't find the container with id 4d9d1b5cbd5ebbbc65fc88798331a76530c991bd2afc75059b31e8c8f1e4fbdb Oct 31 12:32:11.253751 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:32:11.253732 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd720ed_3a46_454c_80c2_4850993e6b76.slice/crio-3e52ebafbf95c6d424b658edb726e95b68be80a1b2bb1f3d634696893f68eca4 WatchSource:0}: Error finding container 3e52ebafbf95c6d424b658edb726e95b68be80a1b2bb1f3d634696893f68eca4: Status 404 returned error can't find the container with id 3e52ebafbf95c6d424b658edb726e95b68be80a1b2bb1f3d634696893f68eca4 Oct 31 12:32:11.288095 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:11.288067 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9txq2"] Oct 31 12:32:11.291174 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:32:11.291151 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e8e847_5b80_44de_9450_c5ae29a5b57b.slice/crio-e9887e89089491f30a483df7856be6b371343b0a442b63fd84dba8e610efe12a WatchSource:0}: Error finding container e9887e89089491f30a483df7856be6b371343b0a442b63fd84dba8e610efe12a: Status 404 returned error can't find the container with id e9887e89089491f30a483df7856be6b371343b0a442b63fd84dba8e610efe12a Oct 31 12:32:11.593371 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:11.593287 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tm5c4" event={"ID":"178dd4e3-f913-4638-b4c7-8c9e96c82053","Type":"ContainerStarted","Data":"4d9d1b5cbd5ebbbc65fc88798331a76530c991bd2afc75059b31e8c8f1e4fbdb"} Oct 31 12:32:11.594606 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:11.594584 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9txq2" event={"ID":"e9e8e847-5b80-44de-9450-c5ae29a5b57b","Type":"ContainerStarted","Data":"4e5391ae6683a659395a2a52f0bb562b80c159a6498812cc53dc80888bffb413"} Oct 31 12:32:11.594728 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:11.594627 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9txq2" event={"ID":"e9e8e847-5b80-44de-9450-c5ae29a5b57b","Type":"ContainerStarted","Data":"e9887e89089491f30a483df7856be6b371343b0a442b63fd84dba8e610efe12a"} Oct 31 12:32:11.595549 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:11.595528 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4kn7k" event={"ID":"5dd720ed-3a46-454c-80c2-4850993e6b76","Type":"ContainerStarted","Data":"3e52ebafbf95c6d424b658edb726e95b68be80a1b2bb1f3d634696893f68eca4"} Oct 31 12:32:11.625144 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:11.625125 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-chf5k_9fcef325-e046-4c6d-8599-6c67115f382c/node-ca/0.log" Oct 31 12:32:12.288069 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:12.288032 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:32:12.288534 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:12.288076 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:32:12.288534 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:12.288306 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:32:12.333188 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:12.333121 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 12:32:12.333307 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:12.333291 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 12:32:12.334865 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:12.334844 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zbx6v\"" Oct 31 12:32:12.334976 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:12.334933 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l77g8\"" Oct 31 12:32:12.335051 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:12.334999 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-l5d6v\"" Oct 31 12:32:12.335128 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:12.335066 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 12:32:12.335184 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:12.335132 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 12:32:12.418536 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:12.418491 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:12.601812 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:12.601725 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9txq2" event={"ID":"e9e8e847-5b80-44de-9450-c5ae29a5b57b","Type":"ContainerStarted","Data":"8394f284ebc6b44487d6edf6e1b9f0247e0919f7bb76d5d4d0a26f82715be694"} Oct 31 12:32:12.954275 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:12.954238 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:13.156756 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:13.156713 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:13.355888 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:13.355855 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:13.606234 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:13.606154 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4kn7k" event={"ID":"5dd720ed-3a46-454c-80c2-4850993e6b76","Type":"ContainerStarted","Data":"ac6201ab996e344b89f539fe589d6b7d109b57ab0576537c23dd823729e8f63e"} Oct 31 12:32:13.606234 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:13.606220 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4kn7k" event={"ID":"5dd720ed-3a46-454c-80c2-4850993e6b76","Type":"ContainerStarted","Data":"2119a541bd2018b526a7ff408ec5abf96a0d44573ee907f7c3394b395e7fbfa0"} Oct 31 12:32:13.606404 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:13.606321 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4kn7k" Oct 31 12:32:13.626539 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:13.626481 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-4kn7k" podStartSLOduration=1.7694006 podStartE2EDuration="3.626464508s" totalImagesPullingTime="1.857063908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:32:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:13.625701822 +0000 UTC m=+50.879975372" watchObservedRunningTime="2026-10-31 12:32:13.626464508 +0000 UTC m=+50.880738038" Oct 31 12:32:14.284079 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.284039 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:14.423516 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.423487 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-v2x29"] Oct 31 12:32:14.426532 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.426511 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2x29" Oct 31 12:32:14.434864 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.434842 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6"] Oct 31 12:32:14.436223 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.436190 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 31 12:32:14.436321 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.436234 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 31 12:32:14.438252 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.437661 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 12:32:14.438252 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.437866 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-8bz5h\"" Oct 31 12:32:14.438252 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.438104 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 12:32:14.439191 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.439171 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" Oct 31 12:32:14.439271 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.439256 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 12:32:14.442199 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.442175 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-v2x29"] Oct 31 12:32:14.445054 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.445006 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-26lht"] Oct 31 12:32:14.448843 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.448825 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:14.451462 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.451430 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6"] Oct 31 12:32:14.458501 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.458478 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a12ea3cb-3a3e-4fab-a040-4b4d00926c07-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-lvmr6\" (UID: \"a12ea3cb-3a3e-4fab-a040-4b4d00926c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" Oct 31 12:32:14.458594 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.458514 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a12ea3cb-3a3e-4fab-a040-4b4d00926c07-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-lvmr6\" (UID: \"a12ea3cb-3a3e-4fab-a040-4b4d00926c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" Oct 31 12:32:14.458594 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.458542 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a12ea3cb-3a3e-4fab-a040-4b4d00926c07-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-lvmr6\" (UID: \"a12ea3cb-3a3e-4fab-a040-4b4d00926c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" Oct 31 12:32:14.458700 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.458669 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2da30edd-1262-4438-b719-da8055c3a763-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-v2x29\" (UID: \"2da30edd-1262-4438-b719-da8055c3a763\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2x29" Oct 31 12:32:14.458747 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.458698 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2da30edd-1262-4438-b719-da8055c3a763-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-v2x29\" (UID: \"2da30edd-1262-4438-b719-da8055c3a763\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2x29" Oct 31 12:32:14.458821 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.458804 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a12ea3cb-3a3e-4fab-a040-4b4d00926c07-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-lvmr6\" (UID: \"a12ea3cb-3a3e-4fab-a040-4b4d00926c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" Oct 31 12:32:14.458871 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.458833 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2da30edd-1262-4438-b719-da8055c3a763-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-v2x29\" (UID: \"2da30edd-1262-4438-b719-da8055c3a763\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2x29" Oct 31 12:32:14.458871 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.458860 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c69t\" (UniqueName: \"kubernetes.io/projected/2da30edd-1262-4438-b719-da8055c3a763-kube-api-access-9c69t\") pod \"openshift-state-metrics-7bd7489957-v2x29\" (UID: \"2da30edd-1262-4438-b719-da8055c3a763\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2x29" Oct 31 12:32:14.458967 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.458892 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj9cr\" (UniqueName: \"kubernetes.io/projected/a12ea3cb-3a3e-4fab-a040-4b4d00926c07-kube-api-access-fj9cr\") pod \"kube-state-metrics-5dc8d7d89b-lvmr6\" (UID: \"a12ea3cb-3a3e-4fab-a040-4b4d00926c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" Oct 31 12:32:14.459055 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.459021 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a12ea3cb-3a3e-4fab-a040-4b4d00926c07-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-lvmr6\" (UID: \"a12ea3cb-3a3e-4fab-a040-4b4d00926c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" Oct 31 12:32:14.487429 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.487405 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 12:32:14.487514 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.487462 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 31 12:32:14.487514 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.487510 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zhwtd\"" Oct 31 12:32:14.487653 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.487638 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 12:32:14.487695 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.487657 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4c2ww\"" Oct 31 12:32:14.487695 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.487681 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 31 12:32:14.489381 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.489356 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:14.538200 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.538179 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 12:32:14.538310 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.538262 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 31 12:32:14.559603 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.559577 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a12ea3cb-3a3e-4fab-a040-4b4d00926c07-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-lvmr6\" (UID: \"a12ea3cb-3a3e-4fab-a040-4b4d00926c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" Oct 31 12:32:14.559687 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.559616 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:14.559687 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.559640 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h58w\" (UniqueName: \"kubernetes.io/projected/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-kube-api-access-5h58w\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:14.559766 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.559722 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a12ea3cb-3a3e-4fab-a040-4b4d00926c07-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-lvmr6\" (UID: \"a12ea3cb-3a3e-4fab-a040-4b4d00926c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" Oct 31 12:32:14.559803 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.559780 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-sys\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:14.559836 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.559812 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a12ea3cb-3a3e-4fab-a040-4b4d00926c07-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-lvmr6\" (UID: \"a12ea3cb-3a3e-4fab-a040-4b4d00926c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" Oct 31 12:32:14.559880 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.559838 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a12ea3cb-3a3e-4fab-a040-4b4d00926c07-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-lvmr6\" (UID: \"a12ea3cb-3a3e-4fab-a040-4b4d00926c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" Oct 31 12:32:14.559880 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.559864 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-metrics-client-ca\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:14.560024 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.559999 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2da30edd-1262-4438-b719-da8055c3a763-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-v2x29\" (UID: \"2da30edd-1262-4438-b719-da8055c3a763\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2x29" Oct 31 12:32:14.560139 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.560063 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-root\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:14.560199 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.560136 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2da30edd-1262-4438-b719-da8055c3a763-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-v2x29\" (UID: \"2da30edd-1262-4438-b719-da8055c3a763\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2x29" Oct 31 12:32:14.560322 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.560286 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-node-exporter-tls\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:14.560407 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.560388 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2da30edd-1262-4438-b719-da8055c3a763-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-v2x29\" (UID: \"2da30edd-1262-4438-b719-da8055c3a763\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2x29" Oct 31 12:32:14.560478 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.560458 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c69t\" (UniqueName: \"kubernetes.io/projected/2da30edd-1262-4438-b719-da8055c3a763-kube-api-access-9c69t\") pod \"openshift-state-metrics-7bd7489957-v2x29\" (UID: \"2da30edd-1262-4438-b719-da8055c3a763\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2x29" Oct 31 12:32:14.560661 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.560620 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a12ea3cb-3a3e-4fab-a040-4b4d00926c07-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-lvmr6\" (UID: \"a12ea3cb-3a3e-4fab-a040-4b4d00926c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" Oct 31 12:32:14.560770 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.560672 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a12ea3cb-3a3e-4fab-a040-4b4d00926c07-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-lvmr6\" (UID: \"a12ea3cb-3a3e-4fab-a040-4b4d00926c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" Oct 31 12:32:14.560770 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.560678 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj9cr\" (UniqueName: \"kubernetes.io/projected/a12ea3cb-3a3e-4fab-a040-4b4d00926c07-kube-api-access-fj9cr\") pod \"kube-state-metrics-5dc8d7d89b-lvmr6\" (UID: \"a12ea3cb-3a3e-4fab-a040-4b4d00926c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" Oct 31 12:32:14.560770 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.560742 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-node-exporter-textfile\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:14.561009 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.560773 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-node-exporter-accelerators-collector-config\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:14.561009 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.560861 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-node-exporter-wtmp\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:14.561009 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.560886 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2da30edd-1262-4438-b719-da8055c3a763-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-v2x29\" (UID: \"2da30edd-1262-4438-b719-da8055c3a763\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2x29" Oct 31 12:32:14.561009 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.560962 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a12ea3cb-3a3e-4fab-a040-4b4d00926c07-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-lvmr6\" (UID: \"a12ea3cb-3a3e-4fab-a040-4b4d00926c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" Oct 31 12:32:14.561287 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.561268 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a12ea3cb-3a3e-4fab-a040-4b4d00926c07-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-lvmr6\" (UID: \"a12ea3cb-3a3e-4fab-a040-4b4d00926c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" Oct 31 12:32:14.562487 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.562461 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a12ea3cb-3a3e-4fab-a040-4b4d00926c07-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-lvmr6\" (UID: \"a12ea3cb-3a3e-4fab-a040-4b4d00926c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" Oct 31 12:32:14.562580 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.562562 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a12ea3cb-3a3e-4fab-a040-4b4d00926c07-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-lvmr6\" (UID: \"a12ea3cb-3a3e-4fab-a040-4b4d00926c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" Oct 31 12:32:14.562832 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.562815 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2da30edd-1262-4438-b719-da8055c3a763-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-v2x29\" (UID: \"2da30edd-1262-4438-b719-da8055c3a763\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2x29" Oct 31 12:32:14.562867 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.562825 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2da30edd-1262-4438-b719-da8055c3a763-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-v2x29\" (UID: \"2da30edd-1262-4438-b719-da8055c3a763\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2x29" Oct 31 12:32:14.572867 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.572846 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c69t\" (UniqueName: \"kubernetes.io/projected/2da30edd-1262-4438-b719-da8055c3a763-kube-api-access-9c69t\") pod \"openshift-state-metrics-7bd7489957-v2x29\" (UID: \"2da30edd-1262-4438-b719-da8055c3a763\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2x29" Oct 31 12:32:14.573264 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.573249 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj9cr\" (UniqueName: \"kubernetes.io/projected/a12ea3cb-3a3e-4fab-a040-4b4d00926c07-kube-api-access-fj9cr\") pod \"kube-state-metrics-5dc8d7d89b-lvmr6\" (UID: \"a12ea3cb-3a3e-4fab-a040-4b4d00926c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" Oct 31 12:32:14.610315 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.610294 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tm5c4" event={"ID":"178dd4e3-f913-4638-b4c7-8c9e96c82053","Type":"ContainerStarted","Data":"f0197c806d6610f680037aeb35c1a166b484408be5d7267e71d85c70901c4377"} Oct 31 12:32:14.612352 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.612329 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9txq2" event={"ID":"e9e8e847-5b80-44de-9450-c5ae29a5b57b","Type":"ContainerStarted","Data":"067487bbba9b3a9d00f8b87b0b8bf5d0c94e19ea9c05eb42832f0b8d5b3785e6"} Oct 31 12:32:14.628524 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.628484 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tm5c4" podStartSLOduration=1.687310036 podStartE2EDuration="4.628473077s" totalImagesPullingTime="2.941163041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:32:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:14.627740985 +0000 UTC m=+51.882014513" watchObservedRunningTime="2026-10-31 12:32:14.628473077 +0000 UTC m=+51.882746604" Oct 31 12:32:14.649261 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.649211 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9txq2" podStartSLOduration=2.414228825 podStartE2EDuration="4.649200019s" totalImagesPullingTime="2.234971194s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:32:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:14.648642156 +0000 UTC m=+51.902915702" watchObservedRunningTime="2026-10-31 12:32:14.649200019 +0000 UTC m=+51.903473548" Oct 31 12:32:14.662187 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.662163 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:14.662291 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.662199 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h58w\" (UniqueName: \"kubernetes.io/projected/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-kube-api-access-5h58w\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:14.662383 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.662359 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-sys\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:14.662503 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.662457 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-sys\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:14.662503 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.662462 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-metrics-client-ca\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:14.663002 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.662964 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-root\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:14.663164 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.663029 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-node-exporter-tls\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:14.663164 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.663081 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-node-exporter-textfile\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:14.663164 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.663113 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-node-exporter-accelerators-collector-config\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:14.663318 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.663175 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-node-exporter-wtmp\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:14.663318 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.663206 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-root\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:14.663318 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.663306 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-node-exporter-wtmp\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:14.663480 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.663317 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-metrics-client-ca\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:14.663527 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.663503 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-node-exporter-textfile\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:14.663564 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.663530 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-node-exporter-accelerators-collector-config\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:14.665057 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.665029 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:14.665277 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.665262 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-node-exporter-tls\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:14.674493 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.674469 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h58w\" (UniqueName: \"kubernetes.io/projected/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-kube-api-access-5h58w\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:14.685118 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.685094 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:14.745916 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.745889 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2x29" Oct 31 12:32:14.758088 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.757974 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" Oct 31 12:32:14.765070 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.765051 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:14.775689 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:32:14.775662 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ef4a37a_e3ae_4c25_abd8_8132d592be9e.slice/crio-293c6f8c3cdeab4b9c8a75f939fcb0aa32173b995bbb9e800a75e477f96cd418 WatchSource:0}: Error finding container 293c6f8c3cdeab4b9c8a75f939fcb0aa32173b995bbb9e800a75e477f96cd418: Status 404 returned error can't find the container with id 293c6f8c3cdeab4b9c8a75f939fcb0aa32173b995bbb9e800a75e477f96cd418 Oct 31 12:32:14.881744 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:14.881707 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:15.023316 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.023291 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-v2x29"] Oct 31 12:32:15.025117 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:32:15.025092 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da30edd_1262_4438_b719_da8055c3a763.slice/crio-0b2ae9cdc4c4abb0ffde53243a0fb5f7beaf231c85f9e7703d2b3ed9882eff3d WatchSource:0}: Error finding container 0b2ae9cdc4c4abb0ffde53243a0fb5f7beaf231c85f9e7703d2b3ed9882eff3d: Status 404 returned error can't find the container with id 0b2ae9cdc4c4abb0ffde53243a0fb5f7beaf231c85f9e7703d2b3ed9882eff3d Oct 31 12:32:15.042471 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.042420 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6"] Oct 31 12:32:15.049083 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:32:15.049053 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda12ea3cb_3a3e_4fab_a040_4b4d00926c07.slice/crio-8a34f6736e45de3c0c200bfcd0855b13be4f4622974d6499c8418c62d33cc800 WatchSource:0}: Error finding container 8a34f6736e45de3c0c200bfcd0855b13be4f4622974d6499c8418c62d33cc800: Status 404 returned error can't find the container with id 8a34f6736e45de3c0c200bfcd0855b13be4f4622974d6499c8418c62d33cc800 Oct 31 12:32:15.085272 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.085245 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:15.287235 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.287157 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:15.521654 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.521619 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 12:32:15.525519 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.525489 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.527306 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.527110 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:15.534998 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.534958 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 12:32:15.535088 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.535033 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 12:32:15.535088 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.535055 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 12:32:15.535176 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.535102 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 12:32:15.535176 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.535149 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 12:32:15.535244 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.535193 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jklxq\"" Oct 31 12:32:15.535244 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.535217 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 12:32:15.535244 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.534970 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 12:32:15.535377 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.535293 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 12:32:15.536454 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.536433 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 12:32:15.547163 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.547106 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 12:32:15.570558 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.570523 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-config-volume\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.570703 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.570584 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ea2d5c42-8213-489b-ba92-369c0435c6e5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.570703 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.570633 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.570703 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.570665 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea2d5c42-8213-489b-ba92-369c0435c6e5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.570851 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.570711 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.570851 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.570738 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ea2d5c42-8213-489b-ba92-369c0435c6e5-config-out\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.570851 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.570753 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ea2d5c42-8213-489b-ba92-369c0435c6e5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.570851 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.570774 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wkrd\" (UniqueName: \"kubernetes.io/projected/ea2d5c42-8213-489b-ba92-369c0435c6e5-kube-api-access-8wkrd\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.570851 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.570820 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-web-config\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.571104 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.570867 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.571104 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.570891 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.571104 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.570913 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea2d5c42-8213-489b-ba92-369c0435c6e5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.571104 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.570927 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.620659 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.620622 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2x29" event={"ID":"2da30edd-1262-4438-b719-da8055c3a763","Type":"ContainerStarted","Data":"488cb9e69535adb76f93ab926b409bbf8972d968c4687f1a6c3545e9a14f2d96"} Oct 31 12:32:15.620830 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.620690 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2x29" event={"ID":"2da30edd-1262-4438-b719-da8055c3a763","Type":"ContainerStarted","Data":"fe1e6d014fd880833302dbb1f99ca408819eaf3bcdbdf5f1c626a07abd34b026"} Oct 31 12:32:15.620830 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.620715 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2x29" event={"ID":"2da30edd-1262-4438-b719-da8055c3a763","Type":"ContainerStarted","Data":"0b2ae9cdc4c4abb0ffde53243a0fb5f7beaf231c85f9e7703d2b3ed9882eff3d"} Oct 31 12:32:15.622796 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.622767 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-26lht" event={"ID":"8ef4a37a-e3ae-4c25-abd8-8132d592be9e","Type":"ContainerStarted","Data":"293c6f8c3cdeab4b9c8a75f939fcb0aa32173b995bbb9e800a75e477f96cd418"} Oct 31 12:32:15.624335 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.624259 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" event={"ID":"a12ea3cb-3a3e-4fab-a040-4b4d00926c07","Type":"ContainerStarted","Data":"8a34f6736e45de3c0c200bfcd0855b13be4f4622974d6499c8418c62d33cc800"} Oct 31 12:32:15.671411 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.671367 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-web-config\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.671551 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.671478 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.671551 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.671520 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.671633 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.671555 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea2d5c42-8213-489b-ba92-369c0435c6e5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.671633 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.671583 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.671633 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.671612 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-config-volume\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.671744 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.671654 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ea2d5c42-8213-489b-ba92-369c0435c6e5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.671744 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.671680 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.671744 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.671704 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea2d5c42-8213-489b-ba92-369c0435c6e5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.671827 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.671796 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.671861 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.671822 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ea2d5c42-8213-489b-ba92-369c0435c6e5-config-out\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.671861 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.671845 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ea2d5c42-8213-489b-ba92-369c0435c6e5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.671921 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.671885 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wkrd\" (UniqueName: \"kubernetes.io/projected/ea2d5c42-8213-489b-ba92-369c0435c6e5-kube-api-access-8wkrd\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.674038 ip-10-0-138-70 kubenswrapper[2582]: E1031 12:32:15.673252 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ea2d5c42-8213-489b-ba92-369c0435c6e5-alertmanager-trusted-ca-bundle podName:ea2d5c42-8213-489b-ba92-369c0435c6e5 nodeName:}" failed. No retries permitted until 2026-10-31 12:32:16.17322844 +0000 UTC m=+53.427501960 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/ea2d5c42-8213-489b-ba92-369c0435c6e5-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "ea2d5c42-8213-489b-ba92-369c0435c6e5") : configmap references non-existent config key: ca-bundle.crt Oct 31 12:32:15.674834 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.674440 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ea2d5c42-8213-489b-ba92-369c0435c6e5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.675522 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.675205 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea2d5c42-8213-489b-ba92-369c0435c6e5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.675522 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.675486 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-web-config\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.675752 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.675728 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ea2d5c42-8213-489b-ba92-369c0435c6e5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.677293 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.677258 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.677948 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.677902 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ea2d5c42-8213-489b-ba92-369c0435c6e5-config-out\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.678058 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.677996 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.678058 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.678028 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.678058 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.678039 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.678603 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.678583 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-config-volume\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.679475 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.679414 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.714145 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.714096 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wkrd\" (UniqueName: \"kubernetes.io/projected/ea2d5c42-8213-489b-ba92-369c0435c6e5-kube-api-access-8wkrd\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:15.775420 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.775338 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:15.976407 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:15.976287 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:16.177409 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.177322 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea2d5c42-8213-489b-ba92-369c0435c6e5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:16.178346 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.178311 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea2d5c42-8213-489b-ba92-369c0435c6e5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:16.457551 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.457510 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:16.629725 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.629689 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" event={"ID":"a12ea3cb-3a3e-4fab-a040-4b4d00926c07","Type":"ContainerStarted","Data":"8951403a5e8e33612a1f7dbf1149b77a250bf0cb5ac85c8cf36e9a767ed07b64"} Oct 31 12:32:16.630138 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.629791 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" event={"ID":"a12ea3cb-3a3e-4fab-a040-4b4d00926c07","Type":"ContainerStarted","Data":"733395960fc82dd78795d7400356384a2684cfcde026df096221607ab6af7a58"} Oct 31 12:32:16.630138 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.629821 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" event={"ID":"a12ea3cb-3a3e-4fab-a040-4b4d00926c07","Type":"ContainerStarted","Data":"17c3dc8654d26fbf0b79969359bfb73bac48cc8b93226a8d7b592d938b33be9d"} Oct 31 12:32:16.631847 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.631823 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2x29" event={"ID":"2da30edd-1262-4438-b719-da8055c3a763","Type":"ContainerStarted","Data":"fca4ff8f16a3b2d1fb6a91a70065953a83c99db2bcfbdf29c724f37272b01bf5"} Oct 31 12:32:16.633452 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.633428 2582 generic.go:356] "Generic (PLEG): container finished" podID="8ef4a37a-e3ae-4c25-abd8-8132d592be9e" containerID="a3af927eaa09f018b068508d377b6fb747af174a3e060b91151cd573964c941b" exitCode=0 Oct 31 12:32:16.633530 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.633463 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-26lht" event={"ID":"8ef4a37a-e3ae-4c25-abd8-8132d592be9e","Type":"ContainerDied","Data":"a3af927eaa09f018b068508d377b6fb747af174a3e060b91151cd573964c941b"} Oct 31 12:32:16.655589 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.655564 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-56dc986dcb-tghc2"] Oct 31 12:32:16.660208 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.660192 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:16.666645 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.666595 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" podStartSLOduration=1.471375873 podStartE2EDuration="2.666583247s" totalImagesPullingTime="1.195207374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:16.665294404 +0000 UTC m=+53.919567934" watchObservedRunningTime="2026-10-31 12:32:16.666583247 +0000 UTC m=+53.920856775" Oct 31 12:32:16.678136 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.677225 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-56dc986dcb-tghc2"] Oct 31 12:32:16.681620 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.680425 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08d2ae45-d701-4555-bc6b-670feece190f-metrics-client-ca\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:16.681620 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.680486 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl8lr\" (UniqueName: \"kubernetes.io/projected/08d2ae45-d701-4555-bc6b-670feece190f-kube-api-access-pl8lr\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:16.681620 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.680522 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/08d2ae45-d701-4555-bc6b-670feece190f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:16.681620 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.680556 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/08d2ae45-d701-4555-bc6b-670feece190f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:16.681620 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.680825 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08d2ae45-d701-4555-bc6b-670feece190f-tmp\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:16.681620 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.680851 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/08d2ae45-d701-4555-bc6b-670feece190f-secret-grpc-tls\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:16.681620 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.680934 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/08d2ae45-d701-4555-bc6b-670feece190f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:16.681620 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.680959 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/08d2ae45-d701-4555-bc6b-670feece190f-secret-thanos-querier-tls\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:16.681620 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.681002 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/08d2ae45-d701-4555-bc6b-670feece190f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:16.682949 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.682932 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 31 12:32:16.683133 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.683118 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 31 12:32:16.683251 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.683238 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 31 12:32:16.683324 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.683310 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 31 12:32:16.684313 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.684295 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-c2m7gphuijb36\"" Oct 31 12:32:16.684407 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.684329 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 31 12:32:16.684477 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.684424 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-xcxdk\"" Oct 31 12:32:16.696205 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.696162 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2x29" podStartSLOduration=1.492477079 podStartE2EDuration="2.696150223s" totalImagesPullingTime="1.203673144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:16.694524267 +0000 UTC m=+53.948797815" watchObservedRunningTime="2026-10-31 12:32:16.696150223 +0000 UTC m=+53.950423750" Oct 31 12:32:16.764595 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.764576 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 12:32:16.767417 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:32:16.767394 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea2d5c42_8213_489b_ba92_369c0435c6e5.slice/crio-a63c715a7cc2b7b1d0a46462feb927341b0329e4ab774c81fee0450104ad97ac WatchSource:0}: Error finding container a63c715a7cc2b7b1d0a46462feb927341b0329e4ab774c81fee0450104ad97ac: Status 404 returned error can't find the container with id a63c715a7cc2b7b1d0a46462feb927341b0329e4ab774c81fee0450104ad97ac Oct 31 12:32:16.781938 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.781916 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/08d2ae45-d701-4555-bc6b-670feece190f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:16.782060 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.781951 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/08d2ae45-d701-4555-bc6b-670feece190f-secret-thanos-querier-tls\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:16.782060 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.781977 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/08d2ae45-d701-4555-bc6b-670feece190f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:16.782060 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.782029 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08d2ae45-d701-4555-bc6b-670feece190f-metrics-client-ca\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:16.782220 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.782076 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl8lr\" (UniqueName: \"kubernetes.io/projected/08d2ae45-d701-4555-bc6b-670feece190f-kube-api-access-pl8lr\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:16.782220 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.782102 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/08d2ae45-d701-4555-bc6b-670feece190f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:16.782220 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.782133 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/08d2ae45-d701-4555-bc6b-670feece190f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:16.782358 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.782242 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08d2ae45-d701-4555-bc6b-670feece190f-tmp\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:16.782358 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.782269 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/08d2ae45-d701-4555-bc6b-670feece190f-secret-grpc-tls\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:16.783337 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.783027 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08d2ae45-d701-4555-bc6b-670feece190f-tmp\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:16.783493 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.783448 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08d2ae45-d701-4555-bc6b-670feece190f-metrics-client-ca\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:16.784930 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.784905 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/08d2ae45-d701-4555-bc6b-670feece190f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:16.785537 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.785465 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/08d2ae45-d701-4555-bc6b-670feece190f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:16.785945 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.785922 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/08d2ae45-d701-4555-bc6b-670feece190f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:16.786522 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.786502 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/08d2ae45-d701-4555-bc6b-670feece190f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:16.786917 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.786900 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/08d2ae45-d701-4555-bc6b-670feece190f-secret-thanos-querier-tls\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:16.787258 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.787239 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/08d2ae45-d701-4555-bc6b-670feece190f-secret-grpc-tls\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:16.795952 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:16.795927 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl8lr\" (UniqueName: \"kubernetes.io/projected/08d2ae45-d701-4555-bc6b-670feece190f-kube-api-access-pl8lr\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:17.000747 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:17.000671 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:17.282419 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:17.282389 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-56dc986dcb-tghc2"] Oct 31 12:32:17.283942 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:32:17.283917 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08d2ae45_d701_4555_bc6b_670feece190f.slice/crio-08bf3028d74ba1be22b8eff52dfa80da4415491c5d2ace59cfe6855fe3f62de3 WatchSource:0}: Error finding container 08bf3028d74ba1be22b8eff52dfa80da4415491c5d2ace59cfe6855fe3f62de3: Status 404 returned error can't find the container with id 08bf3028d74ba1be22b8eff52dfa80da4415491c5d2ace59cfe6855fe3f62de3 Oct 31 12:32:17.564255 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:17.564178 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:17.637879 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:17.637845 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" event={"ID":"08d2ae45-d701-4555-bc6b-670feece190f","Type":"ContainerStarted","Data":"08bf3028d74ba1be22b8eff52dfa80da4415491c5d2ace59cfe6855fe3f62de3"} Oct 31 12:32:17.640252 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:17.640214 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea2d5c42-8213-489b-ba92-369c0435c6e5","Type":"ContainerStarted","Data":"a63c715a7cc2b7b1d0a46462feb927341b0329e4ab774c81fee0450104ad97ac"} Oct 31 12:32:17.643600 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:17.643567 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-26lht" event={"ID":"8ef4a37a-e3ae-4c25-abd8-8132d592be9e","Type":"ContainerStarted","Data":"5b293001290c5db35b8bc7b3c0023033a67afc377038c6c56f10d9a1cd7dd735"} Oct 31 12:32:17.643700 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:17.643627 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-26lht" event={"ID":"8ef4a37a-e3ae-4c25-abd8-8132d592be9e","Type":"ContainerStarted","Data":"5cac65c1dd34990061f98e8a6d955b92ac231dc5e06a8f155ae6c64a91057fce"} Oct 31 12:32:17.668060 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:17.668002 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-26lht" podStartSLOduration=2.5030152599999997 podStartE2EDuration="3.667972473s" totalImagesPullingTime="1.085096358s" totalInitContainerRuntime="79.860855ms" isStatefulPod=true podCreationTimestamp="2026-10-31 12:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:17.666515946 +0000 UTC m=+54.920789474" watchObservedRunningTime="2026-10-31 12:32:17.667972473 +0000 UTC m=+54.922246002" Oct 31 12:32:17.769623 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:17.769574 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:18.006019 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.005240 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:18.305554 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.305521 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-668c5b5566-4scvp"] Oct 31 12:32:18.308735 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.308710 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:18.314563 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.314538 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 31 12:32:18.314705 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.314680 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 31 12:32:18.314705 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.314680 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 31 12:32:18.315889 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.315864 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 31 12:32:18.316035 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.315930 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 31 12:32:18.316200 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.316176 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-m7qs2\"" Oct 31 12:32:18.316465 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.316308 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 31 12:32:18.316604 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.316521 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 31 12:32:18.319833 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.319812 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 31 12:32:18.321848 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.321823 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-668c5b5566-4scvp"] Oct 31 12:32:18.397129 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.397100 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f84bcfff-6360-4722-965f-8b3f152f2bdf-tmp\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:18.397307 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.397137 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8gsv\" (UniqueName: \"kubernetes.io/projected/f84bcfff-6360-4722-965f-8b3f152f2bdf-kube-api-access-b8gsv\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:18.397307 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.397174 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-config\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:18.397307 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.397247 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-serving-cert\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:18.397307 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.397285 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-oauth-serving-cert\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:18.397425 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.397340 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-service-ca\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:18.397425 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.397376 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-trusted-ca-bundle\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:18.397425 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.397402 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-oauth-config\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:18.498176 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.498096 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-config\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:18.498176 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.498157 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-serving-cert\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:18.498387 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.498181 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-oauth-serving-cert\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:18.498387 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.498234 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-service-ca\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:18.498387 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.498287 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-trusted-ca-bundle\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:18.498387 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.498333 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-oauth-config\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:18.498387 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.498356 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f84bcfff-6360-4722-965f-8b3f152f2bdf-tmp\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:18.498630 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.498397 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8gsv\" (UniqueName: \"kubernetes.io/projected/f84bcfff-6360-4722-965f-8b3f152f2bdf-kube-api-access-b8gsv\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:18.499084 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.499054 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-config\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:18.499198 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.499091 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-service-ca\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:18.499198 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.499116 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-oauth-serving-cert\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:18.499198 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.499172 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f84bcfff-6360-4722-965f-8b3f152f2bdf-tmp\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:18.499442 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.499426 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-trusted-ca-bundle\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:18.501099 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.501077 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-oauth-config\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:18.501200 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.501180 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-serving-cert\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:18.510746 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.510719 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8gsv\" (UniqueName: \"kubernetes.io/projected/f84bcfff-6360-4722-965f-8b3f152f2bdf-kube-api-access-b8gsv\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:18.622170 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.622138 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:18.650599 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.650566 2582 generic.go:356] "Generic (PLEG): container finished" podID="ea2d5c42-8213-489b-ba92-369c0435c6e5" containerID="475e94bf7c5e862bc04c010bf619e58932a88ded1c15f91acf581c4058959760" exitCode=0 Oct 31 12:32:18.651063 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.650710 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea2d5c42-8213-489b-ba92-369c0435c6e5","Type":"ContainerDied","Data":"475e94bf7c5e862bc04c010bf619e58932a88ded1c15f91acf581c4058959760"} Oct 31 12:32:18.906115 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:18.906034 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-668c5b5566-4scvp"] Oct 31 12:32:18.909175 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:32:18.909149 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf84bcfff_6360_4722_965f_8b3f152f2bdf.slice/crio-2fad3db0afe9655284a9a370727d43c125b97c26b3561f4a17f974fc4544524e WatchSource:0}: Error finding container 2fad3db0afe9655284a9a370727d43c125b97c26b3561f4a17f974fc4544524e: Status 404 returned error can't find the container with id 2fad3db0afe9655284a9a370727d43c125b97c26b3561f4a17f974fc4544524e Oct 31 12:32:19.344338 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:19.344257 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:19.561845 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:19.561806 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:19.656998 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:19.656959 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" event={"ID":"08d2ae45-d701-4555-bc6b-670feece190f","Type":"ContainerStarted","Data":"6acfb827311e306dfcdc1558073d9ef917b4644faa72d4fd75a5c4c56aebc483"} Oct 31 12:32:19.657388 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:19.657062 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" event={"ID":"08d2ae45-d701-4555-bc6b-670feece190f","Type":"ContainerStarted","Data":"08eea76648cb14ccf58546249c65591021cb96e84e4a3d3d4979c2a8b2f20df8"} Oct 31 12:32:19.657388 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:19.657107 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" event={"ID":"08d2ae45-d701-4555-bc6b-670feece190f","Type":"ContainerStarted","Data":"ae7fcf18d40c6d10f1c7594bb3743e433837e6ffcd8fbab7f5a6171c54debe65"} Oct 31 12:32:19.658289 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:19.658259 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-668c5b5566-4scvp" event={"ID":"f84bcfff-6360-4722-965f-8b3f152f2bdf","Type":"ContainerStarted","Data":"2fad3db0afe9655284a9a370727d43c125b97c26b3561f4a17f974fc4544524e"} Oct 31 12:32:20.203402 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.203308 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:20.416162 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.416132 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:20.666187 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.666151 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" event={"ID":"08d2ae45-d701-4555-bc6b-670feece190f","Type":"ContainerStarted","Data":"336a60114d96c720a12378666eac3ee311cf4dfc4e79ae9b1911012353788fb5"} Oct 31 12:32:20.666601 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.666221 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" event={"ID":"08d2ae45-d701-4555-bc6b-670feece190f","Type":"ContainerStarted","Data":"431f5c7dbf20983509ccb89e6d701d708086e74b6802183e49540d57a823abce"} Oct 31 12:32:20.666601 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.666248 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" event={"ID":"08d2ae45-d701-4555-bc6b-670feece190f","Type":"ContainerStarted","Data":"8f9129f649d8400eea6481e95f46c08e5100d7173a1837029036ccf3f3973e7d"} Oct 31 12:32:20.666601 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.666391 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:20.698237 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.698101 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" podStartSLOduration=1.936810715 podStartE2EDuration="4.698083098s" totalImagesPullingTime="2.761272383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:32:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:20.696713698 +0000 UTC m=+57.950987253" watchObservedRunningTime="2026-10-31 12:32:20.698083098 +0000 UTC m=+57.952356627" Oct 31 12:32:20.824189 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.824116 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 12:32:20.827843 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.827817 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:20.843691 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.843672 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 12:32:20.844019 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.843997 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 12:32:20.844123 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.844051 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 12:32:20.844194 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.844176 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 12:32:20.846511 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.846494 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 12:32:20.850501 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.850481 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 12:32:20.856141 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.856121 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 12:32:20.856141 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.856138 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 12:32:20.856378 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.856352 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 12:32:20.856453 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.856399 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 12:32:20.857025 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.857005 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 12:32:20.857025 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.857007 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 12:32:20.857158 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.857009 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 12:32:20.857158 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.857075 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-crggt40j4ekbj\"" Oct 31 12:32:20.857158 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.857084 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 12:32:20.857158 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.857081 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 12:32:20.857280 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.857174 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-bgz64\"" Oct 31 12:32:20.863817 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.863794 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 12:32:20.920415 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.920386 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:20.920537 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.920433 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:20.920537 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.920462 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-web-config\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:20.920637 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.920572 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:20.920637 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.920610 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:20.920733 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.920640 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:20.920733 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.920700 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:20.920793 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.920777 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:20.920830 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.920806 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:20.920890 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.920870 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r52xd\" (UniqueName: \"kubernetes.io/projected/db5feef6-b3fc-4263-bfc1-66b070df97b4-kube-api-access-r52xd\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:20.920970 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.920914 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-config\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:20.920970 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.920936 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/db5feef6-b3fc-4263-bfc1-66b070df97b4-config-out\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:20.921075 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.921016 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:20.921566 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.921416 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:20.921566 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.921474 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:20.921566 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.921523 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/db5feef6-b3fc-4263-bfc1-66b070df97b4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:20.921761 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.921616 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:20.921813 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.921780 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:20.921865 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.921855 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:20.921911 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.921885 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:20.943135 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:20.943110 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:21.022731 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.022655 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.022877 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.022856 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.022956 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.022913 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.023051 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.022945 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-web-config\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.023051 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.023016 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.023153 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.023066 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.023153 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.023092 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.023153 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.023136 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.023290 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.023170 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.023290 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.023231 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.023383 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.023295 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r52xd\" (UniqueName: \"kubernetes.io/projected/db5feef6-b3fc-4263-bfc1-66b070df97b4-kube-api-access-r52xd\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.023383 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.023344 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-config\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.023484 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.023387 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/db5feef6-b3fc-4263-bfc1-66b070df97b4-config-out\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.023484 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.023411 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.023484 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.023463 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.023624 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.023506 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.023624 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.023564 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/db5feef6-b3fc-4263-bfc1-66b070df97b4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.023624 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.023610 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.023765 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.023643 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.023765 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.023719 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.024554 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.024532 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.025047 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.024997 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.026416 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.025893 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.026416 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.026123 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.026932 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.026903 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.027290 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.027265 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.027437 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.027386 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.028339 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.028299 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-config\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.028456 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.028432 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/db5feef6-b3fc-4263-bfc1-66b070df97b4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.029000 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.028932 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.030083 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.029947 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.030660 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.030641 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.031930 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.031275 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.031930 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.031293 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.031930 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.031418 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.031930 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.031727 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.031930 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.031814 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-web-config\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.031930 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.031897 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/db5feef6-b3fc-4263-bfc1-66b070df97b4-config-out\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.033403 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.033354 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.037539 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.037520 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r52xd\" (UniqueName: \"kubernetes.io/projected/db5feef6-b3fc-4263-bfc1-66b070df97b4-kube-api-access-r52xd\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.155724 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.155563 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:21.488576 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.488541 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 12:32:21.492155 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:32:21.492113 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb5feef6_b3fc_4263_bfc1_66b070df97b4.slice/crio-8d00465d2d3bd7107ed192550f20c47c88dd6e2d52bff4bd6d44a90e4036a013 WatchSource:0}: Error finding container 8d00465d2d3bd7107ed192550f20c47c88dd6e2d52bff4bd6d44a90e4036a013: Status 404 returned error can't find the container with id 8d00465d2d3bd7107ed192550f20c47c88dd6e2d52bff4bd6d44a90e4036a013 Oct 31 12:32:21.534500 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.534469 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:21.674433 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.674394 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea2d5c42-8213-489b-ba92-369c0435c6e5","Type":"ContainerStarted","Data":"fd75328d99325ec80255557e92d33d54426512a43db674e68ed9b2cee254ffc0"} Oct 31 12:32:21.674870 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.674490 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea2d5c42-8213-489b-ba92-369c0435c6e5","Type":"ContainerStarted","Data":"f1713d3fec45df734eb6e5d1509fbd0a322e6c4ea54725424d8b66202b0b5663"} Oct 31 12:32:21.674870 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.674537 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea2d5c42-8213-489b-ba92-369c0435c6e5","Type":"ContainerStarted","Data":"0951cedc3fd3ba47485810d62d387718fdb8a63f28c8f9f37bc7009e60631e37"} Oct 31 12:32:21.674870 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.674572 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea2d5c42-8213-489b-ba92-369c0435c6e5","Type":"ContainerStarted","Data":"61980da9f603f489ea868d5fb29135ab2e58edd1cfa6653b1b7ceb3866063430"} Oct 31 12:32:21.674870 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.674612 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea2d5c42-8213-489b-ba92-369c0435c6e5","Type":"ContainerStarted","Data":"05fac39194a8d616b3f0aa4814b10e65876efe3f857383aa52021d53f57d42dd"} Oct 31 12:32:21.674870 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.674649 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea2d5c42-8213-489b-ba92-369c0435c6e5","Type":"ContainerStarted","Data":"2eb17c149b4df36e04b064b83e92f0b1039815a5e4beae36f18b21f05945cadc"} Oct 31 12:32:21.676145 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.676119 2582 generic.go:356] "Generic (PLEG): container finished" podID="db5feef6-b3fc-4263-bfc1-66b070df97b4" containerID="4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc" exitCode=0 Oct 31 12:32:21.676250 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.676150 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db5feef6-b3fc-4263-bfc1-66b070df97b4","Type":"ContainerDied","Data":"4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc"} Oct 31 12:32:21.676250 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.676226 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db5feef6-b3fc-4263-bfc1-66b070df97b4","Type":"ContainerStarted","Data":"8d00465d2d3bd7107ed192550f20c47c88dd6e2d52bff4bd6d44a90e4036a013"} Oct 31 12:32:21.716657 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.716599 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.306035714 podStartE2EDuration="6.716582933s" totalImagesPullingTime="3.392673616s" totalInitContainerRuntime="17.873603ms" isStatefulPod=false podCreationTimestamp="2026-10-31 12:32:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:21.715678509 +0000 UTC m=+58.969952036" watchObservedRunningTime="2026-10-31 12:32:21.716582933 +0000 UTC m=+58.970856461" Oct 31 12:32:21.809703 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:21.809630 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:22.093672 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:22.093586 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:22.297023 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:22.296959 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:22.681194 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:22.681158 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-668c5b5566-4scvp" event={"ID":"f84bcfff-6360-4722-965f-8b3f152f2bdf","Type":"ContainerStarted","Data":"a7619af8ff161fc7f9fbdabb4b43ead005820e717cb2100fd925ca8d6de5d2a2"} Oct 31 12:32:22.702419 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:22.702376 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-668c5b5566-4scvp" podStartSLOduration=1.7471644579999999 podStartE2EDuration="4.702363503s" totalImagesPullingTime="2.955199045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:32:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:22.701967004 +0000 UTC m=+59.956240546" watchObservedRunningTime="2026-10-31 12:32:22.702363503 +0000 UTC m=+59.956637031" Oct 31 12:32:23.568231 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:23.568200 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:23.615137 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:23.615102 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4kn7k" Oct 31 12:32:26.530292 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:26.530204 2582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-668c5b5566-4scvp"] Oct 31 12:32:26.682346 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:26.682320 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:26.702548 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:26.702502 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db5feef6-b3fc-4263-bfc1-66b070df97b4","Type":"ContainerStarted","Data":"241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef"} Oct 31 12:32:26.702738 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:26.702580 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db5feef6-b3fc-4263-bfc1-66b070df97b4","Type":"ContainerStarted","Data":"f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d"} Oct 31 12:32:26.702738 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:26.702630 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db5feef6-b3fc-4263-bfc1-66b070df97b4","Type":"ContainerStarted","Data":"a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a"} Oct 31 12:32:26.702738 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:26.702660 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db5feef6-b3fc-4263-bfc1-66b070df97b4","Type":"ContainerStarted","Data":"3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413"} Oct 31 12:32:26.702738 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:26.702684 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db5feef6-b3fc-4263-bfc1-66b070df97b4","Type":"ContainerStarted","Data":"ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906"} Oct 31 12:32:26.702738 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:26.702720 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db5feef6-b3fc-4263-bfc1-66b070df97b4","Type":"ContainerStarted","Data":"adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f"} Oct 31 12:32:26.749782 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:26.749720 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.459038701 podStartE2EDuration="6.749701532s" totalImagesPullingTime="4.267693662s" totalInitContainerRuntime="22.969169ms" isStatefulPod=false podCreationTimestamp="2026-10-31 12:32:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:26.745977525 +0000 UTC m=+64.000251073" watchObservedRunningTime="2026-10-31 12:32:26.749701532 +0000 UTC m=+64.003975061" Oct 31 12:32:27.350921 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:27.350879 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:27.553460 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:27.553420 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:28.104354 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:28.104322 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/856f4c46-f3c3-4afe-823e-1b77842932a4-metrics-certs\") pod \"network-metrics-daemon-8bvmr\" (UID: \"856f4c46-f3c3-4afe-823e-1b77842932a4\") " pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:32:28.104550 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:28.104410 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzs6g\" (UniqueName: \"kubernetes.io/projected/6ab66c79-2b0a-4671-9bf8-6aafea4e7454-kube-api-access-qzs6g\") pod \"network-check-target-7sf98\" (UID: \"6ab66c79-2b0a-4671-9bf8-6aafea4e7454\") " pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:32:28.104550 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:28.104445 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-original-pull-secret\") pod \"global-pull-secret-syncer-blcxw\" (UID: \"ed7f6a91-3f6e-409f-bc57-b5d0f8e67770\") " pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:32:28.107597 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:28.107572 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 12:32:28.107719 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:28.107697 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 12:32:28.107719 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:28.107697 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 12:32:28.116918 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:28.116890 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/856f4c46-f3c3-4afe-823e-1b77842932a4-metrics-certs\") pod \"network-metrics-daemon-8bvmr\" (UID: \"856f4c46-f3c3-4afe-823e-1b77842932a4\") " pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:32:28.117271 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:28.117004 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-original-pull-secret\") pod \"global-pull-secret-syncer-blcxw\" (UID: \"ed7f6a91-3f6e-409f-bc57-b5d0f8e67770\") " pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:32:28.117556 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:28.117539 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 12:32:28.128035 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:28.128013 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzs6g\" (UniqueName: \"kubernetes.io/projected/6ab66c79-2b0a-4671-9bf8-6aafea4e7454-kube-api-access-qzs6g\") pod \"network-check-target-7sf98\" (UID: \"6ab66c79-2b0a-4671-9bf8-6aafea4e7454\") " pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:32:28.214410 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:28.214381 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l77g8\"" Oct 31 12:32:28.219640 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:28.219614 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zbx6v\"" Oct 31 12:32:28.221725 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:28.221703 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:32:28.227456 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:28.227202 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:32:28.230346 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:28.227842 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-l5d6v\"" Oct 31 12:32:28.238059 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:28.235784 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:32:28.487004 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:28.483667 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:28.529808 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:28.529721 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-blcxw"] Oct 31 12:32:28.558363 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:28.558338 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7sf98"] Oct 31 12:32:28.562442 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:28.562422 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8bvmr"] Oct 31 12:32:28.562896 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:32:28.562866 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab66c79_2b0a_4671_9bf8_6aafea4e7454.slice/crio-25a2eb84a3073687d9fcce0cbad1b493db4f67bc08c85ebb41007424685dee7d WatchSource:0}: Error finding container 25a2eb84a3073687d9fcce0cbad1b493db4f67bc08c85ebb41007424685dee7d: Status 404 returned error can't find the container with id 25a2eb84a3073687d9fcce0cbad1b493db4f67bc08c85ebb41007424685dee7d Oct 31 12:32:28.564292 ip-10-0-138-70 kubenswrapper[2582]: W1031 12:32:28.564270 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod856f4c46_f3c3_4afe_823e_1b77842932a4.slice/crio-9688828c9124c834c36dd87c69743ea0641dd9b01c80effe16b52064f6f407bf WatchSource:0}: Error finding container 9688828c9124c834c36dd87c69743ea0641dd9b01c80effe16b52064f6f407bf: Status 404 returned error can't find the container with id 9688828c9124c834c36dd87c69743ea0641dd9b01c80effe16b52064f6f407bf Oct 31 12:32:28.622757 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:28.622734 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:28.683434 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:28.683406 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:28.717827 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:28.717795 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7sf98" event={"ID":"6ab66c79-2b0a-4671-9bf8-6aafea4e7454","Type":"ContainerStarted","Data":"25a2eb84a3073687d9fcce0cbad1b493db4f67bc08c85ebb41007424685dee7d"} Oct 31 12:32:28.718915 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:28.718889 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-blcxw" event={"ID":"ed7f6a91-3f6e-409f-bc57-b5d0f8e67770","Type":"ContainerStarted","Data":"c2d8995e2f9691d5a668296466755363b80f2a33df4df094e9fdd205125a1a1e"} Oct 31 12:32:28.719913 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:28.719892 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8bvmr" event={"ID":"856f4c46-f3c3-4afe-823e-1b77842932a4","Type":"ContainerStarted","Data":"9688828c9124c834c36dd87c69743ea0641dd9b01c80effe16b52064f6f407bf"} Oct 31 12:32:30.227319 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:30.227286 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:30.433350 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:30.433253 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:30.732157 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:30.732119 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8bvmr" event={"ID":"856f4c46-f3c3-4afe-823e-1b77842932a4","Type":"ContainerStarted","Data":"893108ecc425110615988b701fe645a6fb189b1932b99a62e13177dd7693b43b"} Oct 31 12:32:30.732353 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:30.732194 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8bvmr" event={"ID":"856f4c46-f3c3-4afe-823e-1b77842932a4","Type":"ContainerStarted","Data":"319c2b8dabb52793a14a2ff7a41ee301cf95bc3f83c67d08a28ed65b262f391b"} Oct 31 12:32:30.754847 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:30.754299 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8bvmr" podStartSLOduration=65.654170681 podStartE2EDuration="1m6.754285475s" totalImagesPullingTime="1.100114794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:31:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:30.751104255 +0000 UTC m=+68.005377810" watchObservedRunningTime="2026-10-31 12:32:30.754285475 +0000 UTC m=+68.008559044" Oct 31 12:32:30.964930 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:30.964785 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:31.156788 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:31.156718 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:31.205480 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:31.205441 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:32.313560 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:32.313500 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:32.517945 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:32.517909 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:32.742505 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:32.742461 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7sf98" event={"ID":"6ab66c79-2b0a-4671-9bf8-6aafea4e7454","Type":"ContainerStarted","Data":"15132ecb4840d3660a4b5120976317033fef09a4b42f4741f201b010a090a13e"} Oct 31 12:32:32.742676 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:32.742636 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:32:32.762430 ip-10-0-138-70 kubenswrapper[2582]: I1031 12:32:32.762381 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7sf98" podStartSLOduration=65.565764954 podStartE2EDuration="1m8.762368143s" totalImagesPullingTime="3.196603189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:31:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:32.761344177 +0000 UTC m=+70.015617704" watchObservedRunningTime="2026-10-31 12:32:32.762368143 +0000 UTC m=+70.016641671" Oct 31 12:32:33.461114 ip-10-0-138-70 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 12:32:33.476363 ip-10-0-138-70 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 12:32:33.476534 ip-10-0-138-70 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 12:32:33.476716 ip-10-0-138-70 systemd[1]: kubelet.service: Consumed 4.425s CPU time. Oct 31 12:32:33.485706 ip-10-0-138-70 systemd[1]: Starting Kubernetes Kubelet... Oct 31 12:32:33.547760 ip-10-0-138-70 kubenswrapper[6841]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 12:32:33.547760 ip-10-0-138-70 kubenswrapper[6841]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 12:32:33.547760 ip-10-0-138-70 kubenswrapper[6841]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 12:32:33.547760 ip-10-0-138-70 kubenswrapper[6841]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 12:32:33.549901 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.549818 6841 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:32:33.549901 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.549900 6841 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:32:33.549961 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.549905 6841 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:32:33.549961 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.549908 6841 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:32:33.549961 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.549911 6841 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:32:33.549961 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.549914 6841 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:32:33.549961 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.549918 6841 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:32:33.549961 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.549921 6841 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:32:33.549961 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.549924 6841 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:32:33.549961 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.549927 6841 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:32:33.549961 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.549936 6841 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:32:33.549961 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.549940 6841 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:32:33.549961 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.549942 6841 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:32:33.549961 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.549945 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:32:33.549961 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.549948 6841 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:32:33.549961 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.549951 6841 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:32:33.549961 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.549954 6841 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:32:33.549961 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.549956 6841 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:32:33.549961 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.549959 6841 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:32:33.549961 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.549962 6841 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:32:33.549961 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.549965 6841 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:32:33.549961 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.549968 6841 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:32:33.550440 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.549971 6841 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:32:33.550440 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.549974 6841 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:32:33.550440 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.549977 6841 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:32:33.550440 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.549997 6841 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:32:33.550440 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550000 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:32:33.550440 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550003 6841 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:32:33.550440 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550006 6841 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:32:33.550440 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550008 6841 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:32:33.550440 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550011 6841 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:32:33.550440 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550014 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:32:33.550440 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550016 6841 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:32:33.550440 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550019 6841 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:32:33.550440 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550022 6841 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:32:33.550440 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550025 6841 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:32:33.550440 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550028 6841 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:32:33.550440 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550033 6841 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:32:33.550440 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550035 6841 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:32:33.550440 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550038 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:32:33.550440 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550041 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:32:33.550885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550043 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:32:33.550885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550046 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:32:33.550885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550055 6841 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:32:33.550885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550058 6841 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:32:33.550885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550061 6841 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:32:33.550885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550064 6841 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:32:33.550885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550067 6841 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:32:33.550885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550070 6841 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:32:33.550885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550072 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:32:33.550885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550075 6841 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:32:33.550885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550078 6841 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:32:33.550885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550080 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:32:33.550885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550083 6841 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:32:33.550885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550086 6841 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:32:33.550885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550089 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:32:33.550885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550092 6841 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:32:33.550885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550095 6841 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:32:33.550885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550098 6841 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:32:33.550885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550101 6841 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:32:33.551359 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550104 6841 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:32:33.551359 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550106 6841 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:32:33.551359 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550109 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:32:33.551359 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550115 6841 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:32:33.551359 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550118 6841 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:32:33.551359 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550122 6841 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:32:33.551359 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550125 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:32:33.551359 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550128 6841 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:32:33.551359 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550131 6841 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:32:33.551359 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550134 6841 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:32:33.551359 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550137 6841 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:32:33.551359 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550140 6841 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:32:33.551359 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550143 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:32:33.551359 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550146 6841 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:32:33.551359 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550148 6841 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:32:33.551359 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550152 6841 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:32:33.551359 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550156 6841 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:32:33.551359 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550159 6841 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:32:33.551359 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550162 6841 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:32:33.551808 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550164 6841 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:32:33.551808 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550167 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:32:33.551808 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550171 6841 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:32:33.551808 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550174 6841 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:32:33.551808 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550177 6841 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:32:33.551808 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550180 6841 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:32:33.551808 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550183 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:32:33.551808 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550185 6841 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:32:33.551808 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550188 6841 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:32:33.551808 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550191 6841 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:32:33.551808 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550193 6841 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:32:33.551808 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550196 6841 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:32:33.551808 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550199 6841 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:32:33.551808 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550201 6841 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:32:33.551808 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550204 6841 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:32:33.551808 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550207 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:32:33.551808 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550210 6841 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:32:33.551808 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550213 6841 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:32:33.551808 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550216 6841 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:32:33.551808 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550218 6841 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:32:33.552277 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550221 6841 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:32:33.552277 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550224 6841 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:32:33.552277 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550227 6841 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:32:33.552277 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550230 6841 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:32:33.552277 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550232 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:32:33.552277 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550235 6841 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:32:33.552277 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550238 6841 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:32:33.552277 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.550993 6841 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:32:33.552277 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551003 6841 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:32:33.552277 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551006 6841 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:32:33.552277 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551010 6841 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:32:33.552277 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551013 6841 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:32:33.552277 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551016 6841 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:32:33.552277 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551019 6841 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:32:33.552277 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551022 6841 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:32:33.552277 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551025 6841 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:32:33.552277 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551027 6841 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:32:33.552277 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551030 6841 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:32:33.552277 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551033 6841 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:32:33.552748 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551036 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:32:33.552748 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551038 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:32:33.552748 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551041 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:32:33.552748 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551043 6841 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:32:33.552748 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551046 6841 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:32:33.552748 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551049 6841 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:32:33.552748 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551052 6841 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:32:33.552748 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551054 6841 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:32:33.552748 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551059 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:32:33.552748 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551062 6841 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:32:33.552748 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551065 6841 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:32:33.552748 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551067 6841 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:32:33.552748 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551070 6841 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:32:33.552748 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551073 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:32:33.552748 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551075 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:32:33.552748 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551078 6841 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:32:33.552748 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551081 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:32:33.552748 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551083 6841 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:32:33.552748 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551086 6841 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:32:33.552748 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551089 6841 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:32:33.553253 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551091 6841 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:32:33.553253 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551094 6841 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:32:33.553253 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551097 6841 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:32:33.553253 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551099 6841 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:32:33.553253 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551103 6841 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:32:33.553253 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551106 6841 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:32:33.553253 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551108 6841 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:32:33.553253 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551111 6841 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:32:33.553253 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551113 6841 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:32:33.553253 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551116 6841 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:32:33.553253 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551119 6841 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:32:33.553253 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551122 6841 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:32:33.553253 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551125 6841 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:32:33.553253 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551128 6841 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:32:33.553253 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551131 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:32:33.553253 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551134 6841 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:32:33.553253 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551136 6841 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:32:33.553253 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551139 6841 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:32:33.553253 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551142 6841 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:32:33.553731 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551144 6841 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:32:33.553731 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551148 6841 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:32:33.553731 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551151 6841 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:32:33.553731 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551154 6841 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:32:33.553731 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551157 6841 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:32:33.553731 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551160 6841 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:32:33.553731 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551168 6841 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:32:33.553731 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551172 6841 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:32:33.553731 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551174 6841 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:32:33.553731 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551177 6841 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:32:33.553731 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551180 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:32:33.553731 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551183 6841 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:32:33.553731 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551185 6841 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:32:33.553731 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551188 6841 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:32:33.553731 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551191 6841 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:32:33.553731 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551194 6841 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:32:33.553731 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551197 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:32:33.553731 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551199 6841 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:32:33.553731 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551202 6841 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:32:33.553731 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551206 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:32:33.554197 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551208 6841 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:32:33.554197 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551211 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:32:33.554197 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551214 6841 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:32:33.554197 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551217 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:32:33.554197 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551220 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:32:33.554197 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551223 6841 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:32:33.554197 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551225 6841 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:32:33.554197 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551228 6841 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:32:33.554197 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551231 6841 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:32:33.554197 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551234 6841 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:32:33.554197 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551236 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:32:33.554197 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551239 6841 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:32:33.554197 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551242 6841 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:32:33.554197 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551245 6841 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:32:33.554197 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551247 6841 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:32:33.554197 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551250 6841 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:32:33.554197 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551253 6841 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:32:33.554197 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551256 6841 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:32:33.554197 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551258 6841 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:32:33.554197 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551261 6841 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:32:33.554659 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551264 6841 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:32:33.554659 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551266 6841 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:32:33.554659 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551269 6841 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:32:33.554659 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551272 6841 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:32:33.554659 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551276 6841 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:32:33.554659 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551280 6841 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:32:33.554659 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551283 6841 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:32:33.554659 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551286 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:32:33.554659 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551289 6841 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:32:33.554659 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551292 6841 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:32:33.554659 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551295 6841 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:32:33.554659 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551301 6841 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:32:33.554659 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551304 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:32:33.554659 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551307 6841 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:32:33.554659 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551310 6841 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:32:33.554659 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551796 6841 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:32:33.554659 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551803 6841 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:32:33.554659 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551806 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:32:33.554659 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551809 6841 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:32:33.554659 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551812 6841 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:32:33.555146 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551815 6841 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:32:33.555146 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551818 6841 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:32:33.555146 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551822 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:32:33.555146 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551824 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:32:33.555146 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551827 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:32:33.555146 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551830 6841 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:32:33.555146 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551833 6841 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:32:33.555146 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551835 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:32:33.555146 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551838 6841 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:32:33.555146 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551840 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:32:33.555146 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551843 6841 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:32:33.555146 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551846 6841 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:32:33.555146 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551849 6841 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:32:33.555146 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551851 6841 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:32:33.555146 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551854 6841 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:32:33.555146 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551860 6841 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:32:33.555146 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551864 6841 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:32:33.555146 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551868 6841 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:32:33.555146 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551872 6841 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:32:33.555594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551875 6841 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:32:33.555594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551877 6841 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:32:33.555594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551880 6841 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:32:33.555594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551883 6841 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:32:33.555594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551886 6841 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:32:33.555594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551889 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:32:33.555594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551892 6841 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:32:33.555594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551895 6841 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:32:33.555594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551897 6841 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:32:33.555594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551900 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:32:33.555594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551903 6841 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:32:33.555594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551906 6841 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:32:33.555594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551909 6841 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:32:33.555594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551911 6841 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:32:33.555594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551914 6841 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:32:33.555594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551917 6841 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:32:33.555594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551919 6841 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:32:33.555594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551922 6841 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:32:33.555594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551925 6841 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:32:33.555594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551927 6841 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:32:33.556075 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551930 6841 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:32:33.556075 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551933 6841 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:32:33.556075 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551935 6841 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:32:33.556075 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551938 6841 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:32:33.556075 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551941 6841 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:32:33.556075 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551943 6841 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:32:33.556075 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551946 6841 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:32:33.556075 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551949 6841 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:32:33.556075 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551953 6841 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:32:33.556075 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551955 6841 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:32:33.556075 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551958 6841 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:32:33.556075 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551962 6841 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:32:33.556075 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551965 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:32:33.556075 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551968 6841 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:32:33.556075 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551970 6841 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:32:33.556075 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551973 6841 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:32:33.556075 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551975 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:32:33.556075 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551993 6841 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:32:33.556075 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.551997 6841 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:32:33.556075 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552000 6841 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:32:33.556537 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552003 6841 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:32:33.556537 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552006 6841 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:32:33.556537 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552008 6841 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:32:33.556537 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552011 6841 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:32:33.556537 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552014 6841 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:32:33.556537 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552016 6841 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:32:33.556537 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552019 6841 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:32:33.556537 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552022 6841 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:32:33.556537 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552025 6841 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:32:33.556537 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552028 6841 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:32:33.556537 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552032 6841 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:32:33.556537 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552036 6841 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:32:33.556537 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552039 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:32:33.556537 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552043 6841 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:32:33.556537 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552046 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:32:33.556537 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552049 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:32:33.556537 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552051 6841 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:32:33.556537 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552054 6841 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:32:33.556537 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552057 6841 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:32:33.557011 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552060 6841 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:32:33.557011 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552064 6841 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:32:33.557011 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552068 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:32:33.557011 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552070 6841 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:32:33.557011 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552074 6841 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:32:33.557011 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552077 6841 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:32:33.557011 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552080 6841 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:32:33.557011 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552082 6841 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:32:33.557011 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552085 6841 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:32:33.557011 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552088 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:32:33.557011 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552090 6841 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:32:33.557011 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552097 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:32:33.557011 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552100 6841 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:32:33.557011 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552103 6841 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:32:33.557011 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552106 6841 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:32:33.557011 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552108 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:32:33.557011 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552111 6841 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:32:33.557011 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552114 6841 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:32:33.557011 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552116 6841 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:32:33.557011 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552119 6841 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:32:33.557480 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552121 6841 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:32:33.557480 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552124 6841 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:32:33.557480 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552127 6841 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:32:33.557480 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552216 6841 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 12:32:33.557480 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552223 6841 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 12:32:33.557480 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552229 6841 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 12:32:33.557480 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552233 6841 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 12:32:33.557480 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552238 6841 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 12:32:33.557480 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552241 6841 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 12:32:33.557480 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552245 6841 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 12:32:33.557480 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552249 6841 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 12:32:33.557480 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552253 6841 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 12:32:33.557480 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552256 6841 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 12:32:33.557480 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552261 6841 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 12:32:33.557480 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552264 6841 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 12:32:33.557480 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552267 6841 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 12:32:33.557480 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552272 6841 flags.go:64] FLAG: --cgroup-root="" Oct 31 12:32:33.557480 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552275 6841 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 12:32:33.557480 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552278 6841 flags.go:64] FLAG: --client-ca-file="" Oct 31 12:32:33.557480 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552282 6841 flags.go:64] FLAG: --cloud-provider="external" Oct 31 12:32:33.557480 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552285 6841 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 12:32:33.557480 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552292 6841 flags.go:64] FLAG: --cluster-domain="" Oct 31 12:32:33.557480 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552296 6841 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 12:32:33.557480 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552299 6841 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 12:32:33.558059 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552310 6841 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 12:32:33.558059 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552313 6841 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 12:32:33.558059 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552317 6841 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 12:32:33.558059 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552321 6841 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 12:32:33.558059 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552324 6841 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 12:32:33.558059 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552327 6841 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 12:32:33.558059 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552330 6841 flags.go:64] FLAG: --contention-profiling="false" Oct 31 12:32:33.558059 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552333 6841 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 12:32:33.558059 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552336 6841 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 12:32:33.558059 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552339 6841 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 12:32:33.558059 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552343 6841 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 12:32:33.558059 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552347 6841 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 12:32:33.558059 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552350 6841 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 12:32:33.558059 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552353 6841 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 12:32:33.558059 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552356 6841 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 12:32:33.558059 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552359 6841 flags.go:64] FLAG: --enable-server="true" Oct 31 12:32:33.558059 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552362 6841 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 12:32:33.558059 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552369 6841 flags.go:64] FLAG: --event-burst="100" Oct 31 12:32:33.558059 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552372 6841 flags.go:64] FLAG: --event-qps="50" Oct 31 12:32:33.558059 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552375 6841 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 12:32:33.558059 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552379 6841 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 12:32:33.558059 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552382 6841 flags.go:64] FLAG: --eviction-hard="" Oct 31 12:32:33.558059 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552386 6841 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 12:32:33.558059 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552390 6841 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 12:32:33.558649 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552394 6841 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 12:32:33.558649 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552397 6841 flags.go:64] FLAG: --eviction-soft="" Oct 31 12:32:33.558649 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552400 6841 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 12:32:33.558649 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552403 6841 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 12:32:33.558649 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552406 6841 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 12:32:33.558649 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552410 6841 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 12:32:33.558649 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552413 6841 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 12:32:33.558649 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552416 6841 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 12:32:33.558649 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552419 6841 flags.go:64] FLAG: --feature-gates="" Oct 31 12:32:33.558649 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552429 6841 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 12:32:33.558649 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552432 6841 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 12:32:33.558649 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552436 6841 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 12:32:33.558649 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552439 6841 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 12:32:33.558649 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552442 6841 flags.go:64] FLAG: --healthz-port="10248" Oct 31 12:32:33.558649 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552445 6841 flags.go:64] FLAG: --help="false" Oct 31 12:32:33.558649 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552448 6841 flags.go:64] FLAG: --hostname-override="ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:32:33.558649 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552451 6841 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 12:32:33.558649 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552455 6841 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 12:32:33.558649 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552458 6841 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 12:32:33.558649 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552461 6841 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 12:32:33.558649 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552465 6841 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 12:32:33.558649 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552468 6841 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 12:32:33.558649 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552471 6841 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 12:32:33.559222 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552474 6841 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 12:32:33.559222 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552477 6841 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 12:32:33.559222 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552480 6841 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 12:32:33.559222 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552484 6841 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 12:32:33.559222 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552487 6841 flags.go:64] FLAG: --kube-reserved="" Oct 31 12:32:33.559222 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552490 6841 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 12:32:33.559222 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552493 6841 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 12:32:33.559222 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552496 6841 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 12:32:33.559222 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552499 6841 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 12:32:33.559222 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552504 6841 flags.go:64] FLAG: --lock-file="" Oct 31 12:32:33.559222 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552507 6841 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 12:32:33.559222 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552510 6841 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 12:32:33.559222 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552513 6841 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 12:32:33.559222 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552519 6841 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 12:32:33.559222 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552522 6841 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 12:32:33.559222 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552525 6841 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 12:32:33.559222 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552528 6841 flags.go:64] FLAG: --logging-format="text" Oct 31 12:32:33.559222 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552531 6841 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 12:32:33.559222 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552535 6841 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 12:32:33.559222 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552543 6841 flags.go:64] FLAG: --manifest-url="" Oct 31 12:32:33.559222 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552547 6841 flags.go:64] FLAG: --manifest-url-header="" Oct 31 12:32:33.559222 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552551 6841 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 12:32:33.559222 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552554 6841 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 12:32:33.559222 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552558 6841 flags.go:64] FLAG: --max-pods="110" Oct 31 12:32:33.559222 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552561 6841 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 12:32:33.559789 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552564 6841 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 12:32:33.559789 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552567 6841 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 12:32:33.559789 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552570 6841 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 12:32:33.559789 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552573 6841 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 12:32:33.559789 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552576 6841 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 12:32:33.559789 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552580 6841 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 12:32:33.559789 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552586 6841 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 12:32:33.559789 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552589 6841 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 12:32:33.559789 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552593 6841 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 12:32:33.559789 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552596 6841 flags.go:64] FLAG: --pod-cidr="" Oct 31 12:32:33.559789 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552599 6841 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 12:32:33.559789 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552601 6841 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 12:32:33.559789 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552605 6841 flags.go:64] FLAG: --pods-per-core="0" Oct 31 12:32:33.559789 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552607 6841 flags.go:64] FLAG: --port="10250" Oct 31 12:32:33.559789 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552611 6841 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 12:32:33.559789 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552614 6841 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0214ebc941630a7b8" Oct 31 12:32:33.559789 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552619 6841 flags.go:64] FLAG: --qos-reserved="" Oct 31 12:32:33.559789 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552622 6841 flags.go:64] FLAG: --read-only-port="10255" Oct 31 12:32:33.559789 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552625 6841 flags.go:64] FLAG: --register-node="true" Oct 31 12:32:33.559789 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552628 6841 flags.go:64] FLAG: --register-with-taints="" Oct 31 12:32:33.559789 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552632 6841 flags.go:64] FLAG: --registry-burst="10" Oct 31 12:32:33.559789 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552635 6841 flags.go:64] FLAG: --registry-qps="5" Oct 31 12:32:33.559789 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552638 6841 flags.go:64] FLAG: --reserved-cpus="" Oct 31 12:32:33.559789 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552640 6841 flags.go:64] FLAG: --reserved-memory="" Oct 31 12:32:33.559789 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552644 6841 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 12:32:33.559789 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552647 6841 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 12:32:33.560689 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552651 6841 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 12:32:33.560689 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552654 6841 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 12:32:33.560689 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552658 6841 flags.go:64] FLAG: --runonce="false" Oct 31 12:32:33.560689 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552660 6841 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 12:32:33.560689 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552664 6841 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 12:32:33.560689 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552667 6841 flags.go:64] FLAG: --seccomp-default="false" Oct 31 12:32:33.560689 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552670 6841 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 12:32:33.560689 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552673 6841 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 12:32:33.560689 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552676 6841 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 12:32:33.560689 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552679 6841 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 12:32:33.560689 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552682 6841 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 12:32:33.560689 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552685 6841 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 12:32:33.560689 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552688 6841 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 12:32:33.560689 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552691 6841 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 12:32:33.560689 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552694 6841 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 12:32:33.560689 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552697 6841 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 12:32:33.560689 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552700 6841 flags.go:64] FLAG: --system-cgroups="" Oct 31 12:32:33.560689 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552703 6841 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 12:32:33.560689 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552708 6841 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 12:32:33.560689 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552712 6841 flags.go:64] FLAG: --tls-cert-file="" Oct 31 12:32:33.560689 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552715 6841 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 12:32:33.560689 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552719 6841 flags.go:64] FLAG: --tls-min-version="" Oct 31 12:32:33.560689 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552723 6841 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 12:32:33.560689 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552726 6841 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 12:32:33.560689 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552729 6841 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 12:32:33.561524 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552732 6841 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 12:32:33.561524 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552735 6841 flags.go:64] FLAG: --v="2" Oct 31 12:32:33.561524 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552740 6841 flags.go:64] FLAG: --version="false" Oct 31 12:32:33.561524 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552744 6841 flags.go:64] FLAG: --vmodule="" Oct 31 12:32:33.561524 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552748 6841 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 12:32:33.561524 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.552753 6841 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 12:32:33.561524 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552905 6841 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:32:33.561524 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552909 6841 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:32:33.561524 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552912 6841 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:32:33.561524 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552915 6841 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:32:33.561524 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552918 6841 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:32:33.561524 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552921 6841 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:32:33.561524 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552924 6841 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:32:33.561524 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552927 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:32:33.561524 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552930 6841 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:32:33.561524 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552933 6841 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:32:33.561524 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552935 6841 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:32:33.561524 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552938 6841 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:32:33.561524 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552941 6841 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:32:33.561524 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552944 6841 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:32:33.561524 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552946 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:32:33.562041 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552949 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:32:33.562041 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552952 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:32:33.562041 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552955 6841 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:32:33.562041 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552958 6841 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:32:33.562041 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552960 6841 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:32:33.562041 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552963 6841 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:32:33.562041 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552966 6841 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:32:33.562041 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552969 6841 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:32:33.562041 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552973 6841 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:32:33.562041 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552976 6841 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:32:33.562041 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552992 6841 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:32:33.562041 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.552997 6841 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:32:33.562041 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553001 6841 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:32:33.562041 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553005 6841 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:32:33.562041 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553008 6841 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:32:33.562041 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553011 6841 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:32:33.562041 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553015 6841 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:32:33.562041 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553018 6841 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:32:33.562041 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553021 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:32:33.562484 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553024 6841 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:32:33.562484 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553026 6841 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:32:33.562484 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553030 6841 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:32:33.562484 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553033 6841 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:32:33.562484 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553036 6841 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:32:33.562484 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553038 6841 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:32:33.562484 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553041 6841 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:32:33.562484 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553044 6841 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:32:33.562484 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553046 6841 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:32:33.562484 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553049 6841 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:32:33.562484 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553052 6841 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:32:33.562484 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553055 6841 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:32:33.562484 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553057 6841 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:32:33.562484 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553060 6841 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:32:33.562484 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553063 6841 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:32:33.562484 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553067 6841 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:32:33.562484 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553071 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:32:33.562484 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553074 6841 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:32:33.562484 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553077 6841 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:32:33.562916 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553080 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:32:33.562916 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553083 6841 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:32:33.562916 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553087 6841 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:32:33.562916 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553090 6841 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:32:33.562916 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553094 6841 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:32:33.562916 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553097 6841 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:32:33.562916 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553100 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:32:33.562916 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553103 6841 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:32:33.562916 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553106 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:32:33.562916 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553109 6841 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:32:33.562916 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553113 6841 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:32:33.562916 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553116 6841 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:32:33.562916 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553118 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:32:33.562916 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553121 6841 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:32:33.562916 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553124 6841 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:32:33.562916 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553127 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:32:33.562916 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553130 6841 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:32:33.562916 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553133 6841 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:32:33.562916 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553136 6841 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:32:33.563386 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553138 6841 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:32:33.563386 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553141 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:32:33.563386 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553144 6841 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:32:33.563386 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553147 6841 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:32:33.563386 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553150 6841 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:32:33.563386 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553153 6841 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:32:33.563386 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553155 6841 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:32:33.563386 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553158 6841 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:32:33.563386 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553161 6841 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:32:33.563386 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553163 6841 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:32:33.563386 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553166 6841 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:32:33.563386 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553169 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:32:33.563386 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553172 6841 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:32:33.563386 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553174 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:32:33.563386 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553178 6841 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:32:33.563386 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553182 6841 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:32:33.563386 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553185 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:32:33.563386 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553188 6841 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:32:33.563386 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553191 6841 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:32:33.563386 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553193 6841 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:32:33.563925 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553196 6841 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:32:33.563925 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553199 6841 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:32:33.563925 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553202 6841 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:32:33.563925 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553206 6841 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:32:33.563925 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553209 6841 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:32:33.563925 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553212 6841 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:32:33.563925 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553215 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:32:33.563925 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553218 6841 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:32:33.563925 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553220 6841 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:32:33.563925 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553223 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:32:33.563925 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553226 6841 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:32:33.563925 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553229 6841 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:32:33.563925 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553232 6841 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:32:33.563925 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553234 6841 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:32:33.563925 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.553260 6841 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 12:32:33.563925 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553296 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:32:33.564426 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553300 6841 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:32:33.564426 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553303 6841 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:32:33.564426 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553306 6841 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:32:33.564426 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553309 6841 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:32:33.564426 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553311 6841 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:32:33.564426 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553314 6841 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:32:33.564426 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553317 6841 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:32:33.564426 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553320 6841 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:32:33.564426 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553323 6841 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:32:33.564426 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553326 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:32:33.564426 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553329 6841 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:32:33.564426 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553332 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:32:33.564426 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553336 6841 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:32:33.564426 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553338 6841 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:32:33.564426 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553341 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:32:33.564426 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553344 6841 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:32:33.564426 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553347 6841 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:32:33.564426 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553350 6841 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:32:33.564426 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553353 6841 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:32:33.564426 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553357 6841 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:32:33.564934 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553359 6841 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:32:33.564934 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553362 6841 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:32:33.564934 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553365 6841 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:32:33.564934 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553368 6841 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:32:33.564934 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553370 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:32:33.564934 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553373 6841 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:32:33.564934 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553376 6841 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:32:33.564934 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553379 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:32:33.564934 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553382 6841 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:32:33.564934 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553384 6841 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:32:33.564934 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553387 6841 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:32:33.564934 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553390 6841 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:32:33.564934 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553393 6841 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:32:33.564934 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553396 6841 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:32:33.564934 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553398 6841 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:32:33.564934 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553401 6841 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:32:33.564934 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553404 6841 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:32:33.564934 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553407 6841 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:32:33.564934 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553410 6841 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:32:33.564934 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553413 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:32:33.565427 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553416 6841 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:32:33.565427 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553418 6841 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:32:33.565427 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553421 6841 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:32:33.565427 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553424 6841 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:32:33.565427 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553426 6841 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:32:33.565427 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553429 6841 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:32:33.565427 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553432 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:32:33.565427 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553434 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:32:33.565427 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553437 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:32:33.565427 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553441 6841 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:32:33.565427 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553443 6841 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:32:33.565427 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553447 6841 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:32:33.565427 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553450 6841 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:32:33.565427 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553453 6841 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:32:33.565427 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553456 6841 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:32:33.565427 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553459 6841 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:32:33.565427 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553461 6841 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:32:33.565427 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553464 6841 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:32:33.565427 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553467 6841 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:32:33.565427 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553469 6841 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:32:33.565885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553472 6841 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:32:33.565885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553475 6841 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:32:33.565885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553477 6841 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:32:33.565885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553480 6841 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:32:33.565885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553483 6841 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:32:33.565885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553485 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:32:33.565885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553488 6841 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:32:33.565885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553490 6841 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:32:33.565885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553493 6841 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:32:33.565885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553496 6841 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:32:33.565885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553498 6841 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:32:33.565885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553501 6841 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:32:33.565885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553504 6841 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:32:33.565885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553506 6841 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:32:33.565885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553509 6841 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:32:33.565885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553512 6841 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:32:33.565885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553514 6841 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:32:33.565885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553517 6841 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:32:33.565885 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553520 6841 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:32:33.566615 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553523 6841 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:32:33.566615 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553525 6841 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:32:33.566615 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553529 6841 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:32:33.566615 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553533 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:32:33.566615 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553538 6841 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:32:33.566615 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553541 6841 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:32:33.566615 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553543 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:32:33.566615 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553546 6841 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:32:33.566615 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553549 6841 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:32:33.566615 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553551 6841 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:32:33.566615 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553556 6841 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:32:33.566615 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553560 6841 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:32:33.566615 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553563 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:32:33.566615 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553565 6841 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:32:33.566615 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553568 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:32:33.566615 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553571 6841 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:32:33.566615 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553574 6841 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:32:33.566615 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553577 6841 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:32:33.566615 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553579 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:32:33.567273 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553582 6841 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:32:33.567273 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553585 6841 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:32:33.567273 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553587 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:32:33.567273 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553590 6841 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:32:33.567273 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553593 6841 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:32:33.567273 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553595 6841 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:32:33.567273 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.553598 6841 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:32:33.567273 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.557378 6841 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 12:32:33.567273 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.557393 6841 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 12:32:33.567273 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.557410 6841 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 12:32:33.567273 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.557415 6841 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 12:32:33.567273 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557478 6841 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:32:33.567273 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557482 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:32:33.567273 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557486 6841 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:32:33.567273 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557489 6841 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:32:33.567273 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557492 6841 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:32:33.567273 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557495 6841 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:32:33.567273 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557500 6841 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:32:33.567273 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557503 6841 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:32:33.568056 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557506 6841 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:32:33.568056 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557508 6841 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:32:33.568056 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557511 6841 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:32:33.568056 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557514 6841 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:32:33.568056 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557518 6841 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:32:33.568056 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557521 6841 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:32:33.568056 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557524 6841 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:32:33.568056 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557527 6841 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:32:33.568056 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557548 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:32:33.568056 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557552 6841 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:32:33.568056 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557556 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:32:33.568056 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557559 6841 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:32:33.568056 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557562 6841 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:32:33.568056 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557565 6841 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:32:33.568056 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557568 6841 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:32:33.568056 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557571 6841 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:32:33.568056 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557574 6841 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:32:33.568056 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557577 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:32:33.568056 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557579 6841 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:32:33.568811 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557582 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:32:33.568811 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557585 6841 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:32:33.568811 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557587 6841 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:32:33.568811 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557590 6841 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:32:33.568811 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557593 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:32:33.568811 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557596 6841 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:32:33.568811 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557599 6841 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:32:33.568811 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557601 6841 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:32:33.568811 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557604 6841 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:32:33.568811 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557607 6841 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:32:33.568811 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557609 6841 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:32:33.568811 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557612 6841 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:32:33.568811 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557615 6841 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:32:33.568811 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557618 6841 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:32:33.568811 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557620 6841 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:32:33.568811 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557623 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:32:33.568811 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557626 6841 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:32:33.568811 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557628 6841 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:32:33.568811 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557631 6841 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:32:33.568811 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557634 6841 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:32:33.569640 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557636 6841 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:32:33.569640 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557639 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:32:33.569640 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557649 6841 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:32:33.569640 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557652 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:32:33.569640 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557654 6841 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:32:33.569640 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557657 6841 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:32:33.569640 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557660 6841 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:32:33.569640 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557663 6841 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:32:33.569640 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557666 6841 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:32:33.569640 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557668 6841 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:32:33.569640 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557671 6841 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:32:33.569640 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557674 6841 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:32:33.569640 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557677 6841 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:32:33.569640 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557679 6841 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:32:33.569640 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557682 6841 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:32:33.569640 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557684 6841 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:32:33.569640 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557687 6841 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:32:33.569640 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557690 6841 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:32:33.569640 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557692 6841 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:32:33.569640 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557695 6841 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:32:33.570594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557697 6841 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:32:33.570594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557700 6841 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:32:33.570594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557703 6841 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:32:33.570594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557706 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:32:33.570594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557709 6841 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:32:33.570594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557711 6841 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:32:33.570594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557714 6841 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:32:33.570594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557716 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:32:33.570594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557719 6841 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:32:33.570594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557722 6841 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:32:33.570594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557724 6841 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:32:33.570594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557727 6841 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:32:33.570594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557730 6841 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:32:33.570594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557732 6841 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:32:33.570594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557735 6841 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:32:33.570594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557743 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:32:33.570594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557746 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:32:33.570594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557749 6841 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:32:33.570594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557752 6841 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:32:33.570594 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557754 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:32:33.571428 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557757 6841 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:32:33.571428 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557760 6841 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:32:33.571428 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557762 6841 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:32:33.571428 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557765 6841 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:32:33.571428 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557769 6841 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:32:33.571428 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557773 6841 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:32:33.571428 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557776 6841 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:32:33.571428 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557779 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:32:33.571428 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557782 6841 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:32:33.571428 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557785 6841 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:32:33.571428 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557788 6841 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:32:33.571428 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557796 6841 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:32:33.571428 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557799 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:32:33.571428 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557802 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:32:33.571428 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557805 6841 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:32:33.571428 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557808 6841 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:32:33.571428 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557810 6841 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:32:33.571428 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557813 6841 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:32:33.571428 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.557816 6841 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:32:33.572295 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.557843 6841 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 12:32:33.572295 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558059 6841 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:32:33.572295 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558065 6841 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:32:33.572295 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558069 6841 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:32:33.572295 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558072 6841 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:32:33.572295 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558075 6841 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:32:33.572295 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558078 6841 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:32:33.572295 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558081 6841 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:32:33.572295 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558083 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:32:33.572295 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558093 6841 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:32:33.572295 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558096 6841 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:32:33.572295 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558099 6841 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:32:33.572295 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558101 6841 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:32:33.572295 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558104 6841 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:32:33.572295 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558107 6841 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:32:33.572295 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558109 6841 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:32:33.573178 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558112 6841 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:32:33.573178 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558115 6841 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:32:33.573178 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558118 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:32:33.573178 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558121 6841 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:32:33.573178 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558123 6841 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:32:33.573178 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558128 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:32:33.573178 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558131 6841 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:32:33.573178 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558134 6841 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:32:33.573178 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558137 6841 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:32:33.573178 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558140 6841 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:32:33.573178 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558142 6841 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:32:33.573178 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558145 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:32:33.573178 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558148 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:32:33.573178 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558152 6841 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:32:33.573178 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558155 6841 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:32:33.573178 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558160 6841 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:32:33.573178 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558163 6841 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:32:33.573178 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558166 6841 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:32:33.573178 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558169 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:32:33.573958 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558172 6841 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:32:33.573958 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558176 6841 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:32:33.573958 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558180 6841 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:32:33.573958 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558183 6841 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:32:33.573958 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558186 6841 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:32:33.573958 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558189 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:32:33.573958 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558192 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:32:33.573958 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558200 6841 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:32:33.573958 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558203 6841 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:32:33.573958 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558206 6841 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:32:33.573958 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558209 6841 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:32:33.573958 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558211 6841 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:32:33.573958 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558214 6841 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:32:33.573958 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558217 6841 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:32:33.573958 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558220 6841 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:32:33.573958 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558223 6841 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:32:33.573958 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558225 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:32:33.573958 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558228 6841 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:32:33.573958 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558232 6841 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:32:33.573958 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558235 6841 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:32:33.574802 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558237 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:32:33.574802 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558240 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:32:33.574802 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558243 6841 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:32:33.574802 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558246 6841 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:32:33.574802 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558249 6841 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:32:33.574802 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558252 6841 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:32:33.574802 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558254 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:32:33.574802 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558257 6841 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:32:33.574802 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558261 6841 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:32:33.574802 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558264 6841 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:32:33.574802 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558267 6841 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:32:33.574802 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558269 6841 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:32:33.574802 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558272 6841 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:32:33.574802 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558275 6841 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:32:33.574802 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558278 6841 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:32:33.574802 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558280 6841 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:32:33.574802 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558283 6841 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:32:33.574802 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558286 6841 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:32:33.574802 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558288 6841 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:32:33.575669 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558294 6841 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:32:33.575669 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558303 6841 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:32:33.575669 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558306 6841 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:32:33.575669 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558309 6841 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:32:33.575669 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558311 6841 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:32:33.575669 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558314 6841 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:32:33.575669 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558317 6841 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:32:33.575669 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558320 6841 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:32:33.575669 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558322 6841 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:32:33.575669 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558325 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:32:33.575669 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558328 6841 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:32:33.575669 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558332 6841 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:32:33.575669 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558335 6841 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:32:33.575669 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558338 6841 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:32:33.575669 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558340 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:32:33.575669 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558343 6841 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:32:33.575669 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558346 6841 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:32:33.575669 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558349 6841 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:32:33.575669 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558352 6841 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:32:33.576501 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558355 6841 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:32:33.576501 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558357 6841 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:32:33.576501 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558361 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:32:33.576501 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558364 6841 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:32:33.576501 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558367 6841 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:32:33.576501 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558370 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:32:33.576501 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558372 6841 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:32:33.576501 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558375 6841 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:32:33.576501 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558378 6841 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:32:33.576501 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558381 6841 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:32:33.576501 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558383 6841 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:32:33.576501 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558386 6841 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:32:33.576501 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558389 6841 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:32:33.576501 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558391 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:32:33.576501 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.558414 6841 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 12:32:33.576501 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558463 6841 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:32:33.577264 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558468 6841 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:32:33.577264 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558470 6841 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:32:33.577264 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558473 6841 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:32:33.577264 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558476 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:32:33.577264 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558479 6841 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:32:33.577264 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558482 6841 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:32:33.577264 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558485 6841 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:32:33.577264 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558487 6841 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:32:33.577264 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558490 6841 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:32:33.577264 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558496 6841 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:32:33.577264 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558499 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:32:33.577264 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558502 6841 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:32:33.577264 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558504 6841 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:32:33.577264 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558507 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:32:33.577264 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558510 6841 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:32:33.577264 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558512 6841 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:32:33.577264 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558515 6841 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:32:33.577264 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558518 6841 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:32:33.577264 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558522 6841 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:32:33.577264 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558525 6841 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:32:33.578236 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558528 6841 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:32:33.578236 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558531 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:32:33.578236 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558533 6841 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:32:33.578236 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558536 6841 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:32:33.578236 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558539 6841 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:32:33.578236 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558541 6841 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:32:33.578236 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558544 6841 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:32:33.578236 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558547 6841 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:32:33.578236 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558550 6841 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:32:33.578236 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558552 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:32:33.578236 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558555 6841 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:32:33.578236 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558557 6841 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:32:33.578236 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558566 6841 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:32:33.578236 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558569 6841 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:32:33.578236 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558572 6841 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:32:33.578236 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558574 6841 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:32:33.578236 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558577 6841 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:32:33.578236 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558580 6841 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:32:33.578236 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558583 6841 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:32:33.579116 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558585 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:32:33.579116 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558588 6841 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:32:33.579116 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558592 6841 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:32:33.579116 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558595 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:32:33.579116 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558598 6841 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:32:33.579116 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558600 6841 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:32:33.579116 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558603 6841 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:32:33.579116 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558606 6841 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:32:33.579116 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558609 6841 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:32:33.579116 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558611 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:32:33.579116 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558614 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:32:33.579116 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558621 6841 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:32:33.579116 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558624 6841 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:32:33.579116 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558627 6841 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:32:33.579116 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558630 6841 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:32:33.579116 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558633 6841 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:32:33.579116 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558636 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:32:33.579116 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558638 6841 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:32:33.579925 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558642 6841 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:32:33.579925 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558646 6841 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:32:33.579925 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558649 6841 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:32:33.579925 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558651 6841 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:32:33.579925 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558654 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:32:33.579925 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558657 6841 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:32:33.579925 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558660 6841 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:32:33.579925 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558662 6841 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:32:33.579925 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558672 6841 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:32:33.579925 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558675 6841 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:32:33.579925 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558678 6841 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:32:33.579925 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558681 6841 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:32:33.579925 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558683 6841 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:32:33.579925 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558686 6841 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:32:33.579925 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558689 6841 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:32:33.579925 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558691 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:32:33.579925 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558696 6841 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:32:33.579925 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558699 6841 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:32:33.579925 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558702 6841 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:32:33.579925 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558704 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:32:33.580848 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558707 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:32:33.580848 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558710 6841 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:32:33.580848 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558712 6841 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:32:33.580848 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558715 6841 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:32:33.580848 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558718 6841 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:32:33.580848 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558722 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:32:33.580848 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558725 6841 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:32:33.580848 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558728 6841 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:32:33.580848 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558731 6841 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:32:33.580848 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558733 6841 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:32:33.580848 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558736 6841 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:32:33.580848 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558739 6841 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:32:33.580848 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558741 6841 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:32:33.580848 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558744 6841 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:32:33.580848 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558747 6841 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:32:33.580848 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558750 6841 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:32:33.580848 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558752 6841 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:32:33.580848 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558755 6841 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:32:33.580848 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558758 6841 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:32:33.580848 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558761 6841 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:32:33.581836 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558764 6841 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:32:33.581836 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558772 6841 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:32:33.581836 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558775 6841 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:32:33.581836 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558779 6841 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:32:33.581836 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558782 6841 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:32:33.581836 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558784 6841 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:32:33.581836 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558787 6841 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:32:33.581836 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:33.558790 6841 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:32:33.581836 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.558837 6841 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 12:32:33.581836 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.559890 6841 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 12:32:33.581836 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.559948 6841 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 12:32:33.581836 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.561269 6841 server.go:1014] "Starting client certificate rotation" Oct 31 12:32:33.581836 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.561473 6841 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 12:32:33.581836 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.561520 6841 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 12:26:24 +0000 UTC" deadline="2028-03-27 06:59:31.403831075 +0000 UTC" Oct 31 12:32:33.581836 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.561574 6841 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12306h26m57.842260002s" Oct 31 12:32:33.582493 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.561823 6841 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 12:32:33.582493 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.562861 6841 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 12:32:33.582493 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.564008 6841 log.go:25] "Validated CRI v1 runtime API" Oct 31 12:32:33.582493 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.565874 6841 log.go:25] "Validated CRI v1 image API" Oct 31 12:32:33.582493 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.566700 6841 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 12:32:33.582493 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.570024 6841 fs.go:135] Filesystem UUIDs: map[01febce6-dac2-4843-81a5-19674d633042:/dev/nvme0n1p4 20bba8e4-2e3a-4312-9ae5-7078c5fbf135:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 31 12:32:33.582778 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.570044 6841 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/08bf3028d74ba1be22b8eff52dfa80da4415491c5d2ace59cfe6855fe3f62de3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/08bf3028d74ba1be22b8eff52dfa80da4415491c5d2ace59cfe6855fe3f62de3/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0b2ae9cdc4c4abb0ffde53243a0fb5f7beaf231c85f9e7703d2b3ed9882eff3d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0b2ae9cdc4c4abb0ffde53243a0fb5f7beaf231c85f9e7703d2b3ed9882eff3d/userdata/shm major:0 minor:259 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/25a2eb84a3073687d9fcce0cbad1b493db4f67bc08c85ebb41007424685dee7d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/25a2eb84a3073687d9fcce0cbad1b493db4f67bc08c85ebb41007424685dee7d/userdata/shm major:0 minor:467 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/28a49097c3af65785e13df0b750453b22cda3a77c1091663e4b32d1a34931038/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/28a49097c3af65785e13df0b750453b22cda3a77c1091663e4b32d1a34931038/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/293c6f8c3cdeab4b9c8a75f939fcb0aa32173b995bbb9e800a75e477f96cd418/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/293c6f8c3cdeab4b9c8a75f939fcb0aa32173b995bbb9e800a75e477f96cd418/userdata/shm major:0 minor:263 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/29662ec5981eedb63451d94253c121403437cf209e58fdd35b582cc07cd6083e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/29662ec5981eedb63451d94253c121403437cf209e58fdd35b582cc07cd6083e/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2a5593f17ffaf057c3130cf9bb1de5849fbd82eddb2fbfe51ddf9144fd876f77/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2a5593f17ffaf057c3130cf9bb1de5849fbd82eddb2fbfe51ddf9144fd876f77/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2aada7e02f979a289459c877f627fbcbe636ec005ad4901058d33e5b4fcc38a6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2aada7e02f979a289459c877f627fbcbe636ec005ad4901058d33e5b4fcc38a6/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d16696a4281ee3226f6a8b5d61f1b670e057fa5bb3b46bcfc96f53442467faf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d16696a4281ee3226f6a8b5d61f1b670e057fa5bb3b46bcfc96f53442467faf/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2fad3db0afe9655284a9a370727d43c125b97c26b3561f4a17f974fc4544524e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2fad3db0afe9655284a9a370727d43c125b97c26b3561f4a17f974fc4544524e/userdata/shm major:0 minor:341 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3aae7470827bb34e14998980bf978982122239e729acb001001ec90d56250f17/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3aae7470827bb34e14998980bf978982122239e729acb001001ec90d56250f17/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3e52ebafbf95c6d424b658edb726e95b68be80a1b2bb1f3d634696893f68eca4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3e52ebafbf95c6d424b658edb726e95b68be80a1b2bb1f3d634696893f68eca4/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4d9d1b5cbd5ebbbc65fc88798331a76530c991bd2afc75059b31e8c8f1e4fbdb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4d9d1b5cbd5ebbbc65fc88798331a76530c991bd2afc75059b31e8c8f1e4fbdb/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/80fd5c23535b507f55088f79647078b229734ff457ca4877f93320bfe66d162d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/80fd5c23535b507f55088f79647078b229734ff457ca4877f93320bfe66d162d/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a34f6736e45de3c0c200bfcd0855b13be4f4622974d6499c8418c62d33cc800/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a34f6736e45de3c0c200bfcd0855b13be4f4622974d6499c8418c62d33cc800/userdata/shm major:0 minor:261 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d00465d2d3bd7107ed192550f20c47c88dd6e2d52bff4bd6d44a90e4036a013/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d00465d2d3bd7107ed192550f20c47c88dd6e2d52bff4bd6d44a90e4036a013/userdata/shm major:0 minor:412 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/91495a0e37b2ab21647b1e27e9cac114cf4fde7363c472c369a23b030bbb3fc0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/91495a0e37b2ab21647b1e27e9cac114cf4fde7363c472c369a23b030bbb3fc0/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9688828c9124c834c36dd87c69743ea0641dd9b01c80effe16b52064f6f407bf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9688828c9124c834c36dd87c69743ea0641dd9b01c80effe16b52064f6f407bf/userdata/shm major:0 minor:463 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a45582cbe71480fd292ad3a12002be22ce33917475324ae2cc1b8d3156c7363/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a45582cbe71480fd292ad3a12002be22ce33917475324ae2cc1b8d3156c7363/userdata/shm major:0 minor:58 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a63c715a7cc2b7b1d0a46462feb927341b0329e4ab774c81fee0450104ad97ac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a63c715a7cc2b7b1d0a46462feb927341b0329e4ab774c81fee0450104ad97ac/userdata/shm major:0 minor:314 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c2d8995e2f9691d5a668296466755363b80f2a33df4df094e9fdd205125a1a1e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c2d8995e2f9691d5a668296466755363b80f2a33df4df094e9fdd205125a1a1e/userdata/shm major:0 minor:464 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5b7285bb83af01ab1e07e8abf8b7bcf1af2e34737a034a5d74b33e7a8b68f1b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5b7285bb83af01ab1e07e8abf8b7bcf1af2e34737a034a5d74b33e7a8b68f1b/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e9887e89089491f30a483df7856be6b371343b0a442b63fd84dba8e610efe12a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e9887e89089491f30a483df7856be6b371343b0a442b63fd84dba8e610efe12a/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08d2ae45-d701-4555-bc6b-670feece190f/volumes/kubernetes.io~projected/kube-api-access-pl8lr:{mountpoint:/var/lib/kubelet/pods/08d2ae45-d701-4555-bc6b-670feece190f/volumes/kubernetes.io~projected/kube-api-access-pl8lr major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08d2ae45-d701-4555-bc6b-670feece190f/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/08d2ae45-d701-4555-bc6b-670feece190f/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08d2ae45-d701-4555-bc6b-670feece190f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/08d2ae45-d701-4555-bc6b-670feece190f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08d2ae45-d701-4555-bc6b-670feece190f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/08d2ae45-d701-4555-bc6b-670feece190f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08d2ae45-d701-4555-bc6b-670feece190f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/08d2ae45-d701-4555-bc6b-670feece190f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08d2ae45-d701-4555-bc6b-670feece190f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/08d2ae45-d701-4555-bc6b-670feece190f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08d2ae45-d701-4555-bc6b-670feece190f/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/08d2ae45-d701-4555-bc6b-670feece190f/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/178dd4e3-f913-4638-b4c7-8c9e96c82053/volumes/kubernetes.io~projected/kube-api-access-865nz:{mountpoint:/var/lib/kubelet/pods/178dd4e3-f913-4638-b4c7-8c9e96c82053/volumes/kubernetes.io~projected/kube-api-access-865nz major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/178dd4e3-f913-4638-b4c7-8c9e96c82053/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/178dd4e3-f913-4638-b4c7-8c9e96c82053/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/volumes/kubernetes.io~secret/agent-certs major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b6cb291-3443-496d-8598-3b253102832f/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/2b6cb291-3443-496d-8598-3b253102832f/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b6cb291-3443-496d-8598-3b253102832f/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/2b6cb291-3443-496d-8598-3b253102832f/volumes/kubernetes.io~empty-dir/tmp major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b6cb291-3443-496d-8598-3b253102832f/volumes/kubernetes.io~projected/kube-api-access-xgk87:{mountpoint:/var/lib/kubelet/pods/2b6cb291-3443-496d-8598-3b253102832f/volumes/kubernetes.io~projected/kube-api-access-xgk87 major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b7d2569-9fcf-4e91-9e2f-42beb94593a8/volumes/kubernetes.io~projected/kube-api-access-6jrdh:{mountpoint:/var/lib/kubelet/pods/2b7d2569-9fcf-4e91-9e2f-42beb94593a8/volumes/kubernetes.io~projected/kube-api-access-6jrdh major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2da30edd-1262-4438-b719-da8055c3a763/volumes/kubernetes.io~projected/kube-api-access-9c69t:{mountpoint:/var/lib/kubelet/pods/2da30edd-1262-4438-b719-da8055c3a763/volumes/kubernetes.io~projected/kube-api-access-9c69t major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2da30edd-1262-4438-b719-da8055c3a763/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/2da30edd-1262-4438-b719-da8055c3a763/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2da30edd-1262-4438-b719-da8055c3a763/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/2da30edd-1262-4438-b719-da8055c3a763/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f51d774-d999-4957-a165-0346972d65bc/volumes/kubernetes.io~projected/kube-api-access-fhn4r:{mountpoint:/var/lib/kubelet/pods/4f51d774-d999-4957-a165-0346972d65bc/volumes/kubernetes.io~projected/kube-api-access-fhn4r major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5dd720ed-3a46-454c-80c2-4850993e6b76/volumes/kubernetes.io~projected/kube-api-access-t5sww:{mountpoint:/var/lib/kubelet/pods/5dd720ed-3a46-454c-80c2-4850993e6b76/volumes/kubernetes.io~projected/kube-api-access-t5sww major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5dd720ed-3a46-454c-80c2-4850993e6b76/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/5dd720ed-3a46-454c-80c2-4850993e6b76/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ab66c79-2b0a-4671-9bf8-6aafea4e7454/volumes/kubernetes.io~projected/kube-api-access-qzs6g:{mountpoint:/var/lib/kubelet/pods/6ab66c79-2b0a-4671-9bf8-6aafea4e7454/volumes/kubernetes.io~projected/kube-api-access-qzs6g major:0 minor:462 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8087a5a6-949e-491f-98b9-3360b93a6419/volumes/kubernetes.io~projected/kube-api-access-ggkt5:{mountpoint:/var/lib/kubelet/pods/8087a5a6-949e-491f-98b9-3360b93a6419/volumes/kubernetes.io~projected/kube-api-access-ggkt5 major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/856f4c46-f3c3-4afe-823e-1b77842932a4/volumes/kubernetes.io~projected/kube-api-access-n4hhg:{mountpoint:/var/lib/kubelet/pods/856f4c46-f3c3-4afe-823e-1b77842932a4/volumes/kubernetes.io~projected/kube-api-access-n4hhg major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/856f4c46-f3c3-4afe-823e-1b77842932a4/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/856f4c46-f3c3-4afe-823e-1b77842932a4/volumes/kubernetes.io~secret/metrics-certs major:0 minor:461 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ef4a37a-e3ae-4c25-abd8-8132d592be9e/volumes/kubernetes.io~projected/kube-api-access-5h58w:{mountpoint:/var/lib/kubelet/pods/8ef4a37a-e3ae-4c25-abd8-8132d592be9e/volumes/kubernetes.io~projected/kube-api-access-5h58w major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ef4a37a-e3ae-4c25-abd8-8132d592be9e/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/8ef4a37a-e3ae-4c25-abd8-8132d592be9e/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:256 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ef4a37a-e3ae-4c25-abd8-8132d592be9e/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/8ef4a37a-e3ae-4c25-abd8-8132d592be9e/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9fcef325-e046-4c6d-8599-6c67115f382c/volumes/kubernetes.io~projected/kube-api-access-5jd8l:{mountpoint:/var/lib/kubelet/pods/9fcef325-e046-4c6d-8599-6c67115f382c/volumes/kubernetes.io~projected/kube-api-access-5jd8l major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a12ea3cb-3a3e-4fab-a040-4b4d00926c07/volumes/kubernetes.io~projected/kube-api-access-fj9cr:{mountpoint:/var/lib/kubelet/pods/a12ea3cb-3a3e-4fab-a040-4b4d00926c07/volumes/kubernetes.io~projected/kube-api-access-fj9cr major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a12ea3cb-3a3e-4fab-a040-4b4d00926c07/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/a12ea3cb-3a3e-4fab-a040-4b4d00926c07/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a12ea3cb-3a3e-4fab-a040-4b4d00926c07/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/a12ea3cb-3a3e-4fab-a040-4b4d00926c07/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca9b156d-7499-4b53-b17f-01dc63ae0c5e/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/ca9b156d-7499-4b53-b17f-01dc63ae0c5e/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca9b156d-7499-4b53-b17f-01dc63ae0c5e/volumes/kubernetes.io~projected/kube-api-access-cw546:{mountpoint:/var/lib/kubelet/pods/ca9b156d-7499-4b53-b17f-01dc63ae0c5e/volumes/kubernetes.io~projected/kube-api-access-cw546 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca9b156d-7499-4b53-b17f-01dc63ae0c5e/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/ca9b156d-7499-4b53-b17f-01dc63ae0c5e/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d89912fc-3948-4569-bfea-87c8dceedd36/volumes/kubernetes.io~projected/kube-api-access-g8wjg:{mountpoint:/var/lib/kubelet/pods/d89912fc-3948-4569-bfea-87c8dceedd36/volumes/kubernetes.io~projected/kube-api-access-g8wjg major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volume-subpaths/web-config/config-reloader/2 major:0 minor:392 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:392 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volume-subpaths/web-config/prometheus/15 major:0 minor:392 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes/kubernetes.io~empty-dir/config-out major:0 minor:395 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes/kubernetes.io~projected/kube-api-access-r52xd:{mountpoint:/var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes/kubernetes.io~projected/kube-api-access-r52xd major:0 minor:400 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes/kubernetes.io~projected/tls-assets major:0 minor:390 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes/kubernetes.io~secret/config major:0 minor:391 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:394 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:393 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:396 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:397 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:398 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:388 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:399 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes/kubernetes.io~secret/web-config major:0 minor:392 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9e8e847-5b80-44de-9450-c5ae29a5b57b/volumes/kubernetes.io~projected/kube-api-access-bnpbz:{mountpoint:/var/lib/kubelet/pods/e9e8e847-5b80-44de-9450-c5ae29a5b57b/volumes/kubernetes.io~projected/kube-api-access-bnpbz major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9e8e847-5b80-44de-9450-c5ae29a5b57b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/e9e8e847-5b80-44de-9450-c5ae29a5b57b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volume-subpaths/web-config/alertmanager/9 major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volume-subpaths/web-config/config-reloader/6 major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volumes/kubernetes.io~empty-dir/config-out major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volumes/kubernetes.io~projected/kube-api-access-8wkrd:{mountpoint:/var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volumes/kubernetes.io~projected/kube-api-access-8wkrd major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volumes/kubernetes.io~projected/tls-assets major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volumes/kubernetes.io~secret/config-volume major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volumes/kubernetes.io~secret/web-config major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:457 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f23a1c72-8427-4b6f-ad2b-cc76a88709c9/volumes/kubernetes.io~projected/kube-api-access-fwz6m:{mountpoint:/var/lib/kubelet/pods/f23a1c72-8427-4b6f-ad2b-cc76a88709c9/volumes/kubernetes.io~projected/kube-api-access-fwz6m major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f84bcfff-6360-4722-965f-8b3f152f2bdf/volumes/kubernetes.io~projected/kube-api-access-b8gsv:{mountpoint:/var/lib/kubelet/pods/f84bcfff-6360-4722-965f-8b3f152f2bdf/volumes/kubernetes.io~projected/kube-api-access-b8gsv major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f84bcfff-6360-4722-965f-8b3f152f2bdf/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/f84bcfff-6360-4722-965f-8b3f152f2bdf/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f84bcfff-6360-4722-965f-8b3f152f2bdf/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/f84bcfff-6360-4722-965f-8b3f152f2bdf/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:338 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/f851f6a72b6fb44d4d81981e9b4d55348a112ab4d7eae0215ae6302b69160d68/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/d318688ac8784a032e66af532c801b80b352e313fd253d4b602886413e0f355b/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/7dc873ae2d757061e0cd6db03f471ac45166498a345224a3217a88189d36065a/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/4394d1320e0938a87281e4e99062882bd75ae5e054e5699140a57de6d1a96af0/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/952578e5a0878fd227a71411b63f0d332bdd66dc771585584705cacbf07a38f5/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/8f3e91fcce514b7e80cb844983c2b9b8b82fb087366bd159f5b5827db28e2e0b/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/6143eb76299cf8775cb45a14e5a47617cfddaee650f979a2fbf4adc02f36a393/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/402cc6e1b24af4a8adb8b442b70dd69ede3788cec8f78ff7a711128dc7224dbb/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/df19e0ca3cd51ac7ebe4569ad122bd9b5437d9398999e00bc9d1fe81d8989fff/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/e5941a108ca40669165f58f3f7280e4af40e10ccdd369c817359699d9c74d014/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/1fe601dea0774c14bf5596c4c924d37b3108e939d4e19c4173be744711e2945a/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/c6fb24fe85cd76c9a0be29995a5ebcf53f72dc3c2544ee5d360c63f0f052e7de/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/c2606a6d696cde3535141c2966845bfa7053e638a163831f59df084a1d90d8cb/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/540cbec88b4d3765260f1330cd40fcdf1c358d24ba552737a845fc49ebcec70f/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/0ed83f337c5d459e11d10382df963763c8e64e662d1ae6570229e1d2dd31f8be/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/37fe83b2deaf18b7691bf4516e7e87d7b9f9ee47ffb3b3ceaccb1b6c2434abb8/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/1664443510c3f03ca66f7f66a2f6bb89977d74c64d255d374cebb00cc450dfa4/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/172e3dd81543949a7f162de436a826d37893215feb6fea64c5732fe9ce429ee8/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/10067056ca12d58c979230608986c85271fde7b7335eea8281cf38f7bf473cdc/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/335adab70846b4b4a5cba1a7af175d5c95ef27619357eb1dcd63d06ef8824eb3/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/aeb5167d4a404ebb1204ff29d2cca16cdc2ffcebd17e49488698c53c7319f360/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/7f475ded567da7ab9ec49d98e5a8288a5752b901021553532e64088e36333f09/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/61cbcc65246d100ee9601006d66a2c46fe27babfa2172686166fd5c964295d69/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/dbbca78668f0eef331cd8decfcfb437fb7792b008ca1f96a562eef8225996f0e/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/2c517c490999354a419a41adb7c25d8d5af1dbfa2cb2c1d41e324ccd5d2fb1da/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/92127740378fe9f9dd546ec8221867f7a9366ba69f97891c1ec4e29bc37f7a71/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/848911b6dfb3c6fd027cb7874d0c713262221ae41b4e8a33ab1c93940b2604f1/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/b1a734025a3d21b71412346d3e986397007d73758baadf539f29f2a8f7585712/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/526b92d42d82768b70035668a07f757dbde071ea9aabbe2a1ac2c4ff36355692/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/5de1a2b5cd9fc324098d425819ef6d21fca4dac12c958c56dbf0ecdac3050291/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/3074f7eddf042b338e5e64408458cc1e3ffe4d9826a2de71e666063ad96d8ddc/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/932e587a76817bd0c43d40c703f79033aa5d35d7a6850079f99b5bdd0c326d92/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/34608b28c789dda382f6d0828ff52d2f8c31c6e2f89045c0c95d41967bf3fc7c/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/9eaceaa8f3b13eafe779cb07fcb40a1642ac1c5bf78a15c0807cd160b07dbbf9/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/6f3da07393d6363df68810d105d609b4f680d352b811b4a96dc55ba0d66d48d5/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/fc642c36fcf876c205a5d8bbe57c04d1f712cdc7497ddf8fbe78ce2a6932481b/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/cdc8b5268477ed5a476424e994a8541575c41e4c473f0aee98ccdc39c60e7cd3/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/398e560942a6cbddacb5393e96aa4b2674b3bbb1e4d540005742c40943255c15/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/9f183b234622831918f8254f003aebf59c999c0156f76012a20729967d2ef552/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/49e63366e7e6247ef8d955c65c64a059e6814f09164916aaeb0482eb119c32c1/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/ce3bd62f90820bc59d7459cd1700aa3be035e8272b30cbcec24bfd160c300873/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/2e96b9b9a8b1545a212ee530777c8073f7ddab94ced92e888b61228f78edcfac/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/025a77734e80bdb969b98b9bab24f1577f792613107b795e68a1670141a267f0/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/70ec6c3d68d3a49709f20b1e65cbc662db6f7d3de06ad66a878be20083cf03e7/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-345:{mountpoint:/var/lib/containers/storage/overlay/da31dff54dff6719dcbe3553181539b2973a1cfc591cdc4c0f14ab8e9b483aea/merged major:0 minor:345 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/23f57879c1d039b37ff1c7f9ff4a9b183f5593baf631f02149055335555e8d03/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/16a9c1ee7e3503a1a8affff0d83d52bec6f808cc95c36d7513a77e4fa513e82a/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/59a489c36c4c9bffe775ae10fa7389a0e0c0ea656eadf02a99da50a59b226a2c/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/2b9eb0fee4aa83ff39dc744a343c39621912968e565e73c45362c383a625c6f8/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/1149df0e2ba9e71e614e5f7992b393ec49565829bf87e97f5516ee01001053b3/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-373:{mountpoint:/var/lib/containers/storage/overlay/de1f69743fd7d32bd85c490883634d0d586acda5061f33923e0314b6a483a42f/merged major:0 minor:373 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/00f2c99663373a171c25e0b9235e2744fb4ca9bca7a113c313b3e94be37c823c/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-384:{mountpoint:/var/lib/containers/storage/overlay/19a4372efb74f37c34b820a03d2e1e7c86da225d094ab55866322ae77f321ad5/merged major:0 minor:384 fsType:overlay blockSize:0} overlay_0-401:{mountpoint:/var/lib/containers/storage/overlay/ec361d441f4ef47200ac31197406e9de13257e5b7d483226640b62f9e683b993/merged major:0 minor:401 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/e2ffc004a866792afba9969432494cbf7ff4aad34831ed0fc1517d789685ce6d/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/6d8ceac3ffe835d5fde2a784dd26da71e997075c0938898e4bd9dd37d90c5c9f/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-418:{mountpoint:/var/lib/containers/storage/overlay/d010de31dc6c04c91f4ec4eb363b2ee9eff84c4ebfc3f6d6b2ff72f538995910/merged major:0 minor:418 fsType:overlay blockSize:0} overlay_0-423:{mountpoint:/var/lib/containers/storage/overlay/817b03896eeb46d6e0b6cab8a1b48c1a56bbd736f43ea3276990eea6e26d7f81/merged major:0 minor:423 fsType:overlay blockSize:0} overlay_0-425:{mountpoint:/var/lib/containers/storage/overlay/eca64bc5224752cf356cee8513921bf323c9fbb42d0fed7f9b6746fe91f43606/merged major:0 minor:425 fsType:overlay blockSize:0} overlay_0-431:{mountpoint:/var/lib/containers/storage/overlay/c9b1cec667c151b8b2a08d4f557c017b842f894f2603b32e44f187d36a0dba4e/merged major:0 minor:431 fsType:overlay blockSize:0} overlay_0-437:{mountpoint:/var/lib/containers/storage/overlay/c6238f0e13ee421c3a18ef49cb95f4b0e15c8fc5d9d264a646dbc82bb35cc4ee/merged major:0 minor:437 fsType:overlay blockSize:0} overlay_0-442:{mountpoint:/var/lib/containers/storage/overlay/2c3364fe55458c47d7d498de2a9a3392256c40a16cf5f0f28c60a274a7dac287/merged major:0 minor:442 fsType:overlay blockSize:0} overlay_0-447:{mountpoint:/var/lib/containers/storage/overlay/ed3b5cf9e35cb538d4a096d832281f8c5761868cc97aaf880112fa4237aa1bce/merged major:0 minor:447 fsType:overlay blockSize:0} overlay_0-452:{mountpoint:/var/lib/containers/storage/overlay/a238faa43707947dd599ccca10192f9660b82691ce503b3e897a076ef08d0086/merged major:0 minor:452 fsType:overlay blockSize:0} overlay_0-469:{mountpoint:/var/lib/containers/storage/overlay/c2ef04f6dee5dab0a456f6af4e0db0e3f6099b279874cfe5930280a8b2a05281/merged major:0 minor:469 fsType:overlay blockSize:0} overlay_0-471:{mountpoint:/var/lib/containers/storage/overlay/fb4989dcc4f92822ea8cef44c845f9d9f35a13fef24f8d4a1fc427ccc4f1cfa5/merged major:0 minor:471 fsType:overlay blockSize:0} overlay_0-473:{mountpoint:/var/lib/containers/storage/overlay/86bee89c4f4632c2de731f1065acd43d3796a4492b17aeb3c738795a163babef/merged major:0 minor:473 fsType:overlay blockSize:0} overlay_0-475:{mountpoint:/var/lib/containers/storage/overlay/3830711587dc229eded6578858985eb2171b4df08ec76b72d7fbcf7152fad9d6/merged major:0 minor:475 fsType:overlay blockSize:0} overlay_0-477:{mountpoint:/var/lib/containers/storage/overlay/271ae1a2654a805b2e0b022929034f73f350859bd3d1cdaffd739e40750d0ad6/merged major:0 minor:477 fsType:overlay blockSize:0} overlay_0-483:{mountpoint:/var/lib/containers/storage/overlay/07fc388cc1ef77fdb7aa026a8ad06039cf838d755ac2515749c9dcf7ece62e07/merged major:0 minor:483 fsType:overlay blockSize:0} overlay_0-488:{mountpoint:/var/lib/containers/storage/overlay/9b3cde1d32a8aa6761702cf7607f220820e34f43a9340fd737e3809c870eca56/merged major:0 minor:488 fsType:overlay blockSize:0} overlay_0-62:{mountpoint:/var/lib/containers/storage/overlay/e12b0a9775cca7c1fe58ac51d6c0cdd16d9999c754d4a6636a27b5c2e1bc276c/merged major:0 minor:62 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/248c23406444f971379000782ed6f4d61da65b010a7c1ade5ccd6726e9e8fd0a/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/0e28cf7dcaa94defc4b7dffa404aad1ec4a0d415355d8a886543adef2973fe87/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/2b01bc0a912fd0c484f3f290ede8d06e6a8d54b03ccc459abba69134959c4798/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/fe0eaef9fca3c560566faaacc4396ca25d88a5266b52eafe7fd61c469f9af4db/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/afc588b177d7579c743b2a76445d5939a83728e43a49ab57d10beb3d86acfc9f/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/7bef0b8948e89fd16db09882d9a3a02e06d0aff23645373e90b6b8d0fe902a72/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/238981fdf35698e2e1f17003748e5ec41933f32afa62c928e3dfc5d3f6c53952/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/5148a98c587b7b8ce2a771a5807cd94b545c9fbcfcd23e23fab399f86e353620/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/ea2421e259de90487b6375479197bdc5eaf7882cbd10977773685ba686496329/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 31 12:32:33.583954 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.583323 6841 manager.go:217] Machine: {Timestamp:2026-10-31 12:32:33.582148471 +0000 UTC m=+0.065271368 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103982 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f22714cfba4a925c95ace1069530b SystemUUID:ec2f2271-4cfb-a4a9-25c9-5ace1069530b BootID:b97e6b3e-70b3-495a-879f-92f4cde9d215 Filesystems:[{Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f84bcfff-6360-4722-965f-8b3f152f2bdf/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:338 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3e52ebafbf95c6d424b658edb726e95b68be80a1b2bb1f3d634696893f68eca4/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/08d2ae45-d701-4555-bc6b-670feece190f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:330 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:287 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2b6cb291-3443-496d-8598-3b253102832f/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:50 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:279 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:392 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a12ea3cb-3a3e-4fab-a040-4b4d00926c07/volumes/kubernetes.io~projected/kube-api-access-fj9cr DeviceMajor:0 DeviceMinor:255 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:287 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volumes/kubernetes.io~projected/kube-api-access-8wkrd DeviceMajor:0 DeviceMinor:291 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d00465d2d3bd7107ed192550f20c47c88dd6e2d52bff4bd6d44a90e4036a013/userdata/shm DeviceMajor:0 DeviceMinor:412 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:392 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-473 DeviceMajor:0 DeviceMinor:473 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2da30edd-1262-4438-b719-da8055c3a763/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:253 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:391 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/856f4c46-f3c3-4afe-823e-1b77842932a4/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:461 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-471 DeviceMajor:0 DeviceMinor:471 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3aae7470827bb34e14998980bf978982122239e729acb001001ec90d56250f17/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4d9d1b5cbd5ebbbc65fc88798331a76530c991bd2afc75059b31e8c8f1e4fbdb/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a63c715a7cc2b7b1d0a46462feb927341b0329e4ab774c81fee0450104ad97ac/userdata/shm DeviceMajor:0 DeviceMinor:314 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:388 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6ab66c79-2b0a-4671-9bf8-6aafea4e7454/volumes/kubernetes.io~projected/kube-api-access-qzs6g DeviceMajor:0 DeviceMinor:462 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-477 DeviceMajor:0 DeviceMinor:477 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-373 DeviceMajor:0 DeviceMinor:373 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/178dd4e3-f913-4638-b4c7-8c9e96c82053/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-401 DeviceMajor:0 DeviceMinor:401 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8087a5a6-949e-491f-98b9-3360b93a6419/volumes/kubernetes.io~projected/kube-api-access-ggkt5 DeviceMajor:0 DeviceMinor:47 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a45582cbe71480fd292ad3a12002be22ce33917475324ae2cc1b8d3156c7363/userdata/shm DeviceMajor:0 DeviceMinor:58 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e9887e89089491f30a483df7856be6b371343b0a442b63fd84dba8e610efe12a/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:279 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5dd720ed-3a46-454c-80c2-4850993e6b76/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2da30edd-1262-4438-b719-da8055c3a763/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:252 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-447 DeviceMajor:0 DeviceMinor:447 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-437 DeviceMajor:0 DeviceMinor:437 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2b6cb291-3443-496d-8598-3b253102832f/volumes/kubernetes.io~projected/kube-api-access-xgk87 DeviceMajor:0 DeviceMinor:48 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:285 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-384 DeviceMajor:0 DeviceMinor:384 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e9e8e847-5b80-44de-9450-c5ae29a5b57b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/178dd4e3-f913-4638-b4c7-8c9e96c82053/volumes/kubernetes.io~projected/kube-api-access-865nz DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:289 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4f51d774-d999-4957-a165-0346972d65bc/volumes/kubernetes.io~projected/kube-api-access-fhn4r DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ca9b156d-7499-4b53-b17f-01dc63ae0c5e/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8ef4a37a-e3ae-4c25-abd8-8132d592be9e/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:256 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/08d2ae45-d701-4555-bc6b-670feece190f/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:331 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f23a1c72-8427-4b6f-ad2b-cc76a88709c9/volumes/kubernetes.io~projected/kube-api-access-fwz6m DeviceMajor:0 DeviceMinor:54 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:284 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:396 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:457 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2a5593f17ffaf057c3130cf9bb1de5849fbd82eddb2fbfe51ddf9144fd876f77/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-423 DeviceMajor:0 DeviceMinor:423 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-425 DeviceMajor:0 DeviceMinor:425 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0b2ae9cdc4c4abb0ffde53243a0fb5f7beaf231c85f9e7703d2b3ed9882eff3d/userdata/shm DeviceMajor:0 DeviceMinor:259 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:279 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:290 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/08d2ae45-d701-4555-bc6b-670feece190f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:329 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:390 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:392 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2aada7e02f979a289459c877f627fbcbe636ec005ad4901058d33e5b4fcc38a6/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ca9b156d-7499-4b53-b17f-01dc63ae0c5e/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a34f6736e45de3c0c200bfcd0855b13be4f4622974d6499c8418c62d33cc800/userdata/shm DeviceMajor:0 DeviceMinor:261 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/08d2ae45-d701-4555-bc6b-670feece190f/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:332 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/08bf3028d74ba1be22b8eff52dfa80da4415491c5d2ace59cfe6855fe3f62de3/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2fad3db0afe9655284a9a370727d43c125b97c26b3561f4a17f974fc4544524e/userdata/shm DeviceMajor:0 DeviceMinor:341 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9fcef325-e046-4c6d-8599-6c67115f382c/volumes/kubernetes.io~projected/kube-api-access-5jd8l DeviceMajor:0 DeviceMinor:52 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5b7285bb83af01ab1e07e8abf8b7bcf1af2e34737a034a5d74b33e7a8b68f1b/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-345 DeviceMajor:0 DeviceMinor:345 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:393 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:394 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-418 DeviceMajor:0 DeviceMinor:418 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d16696a4281ee3226f6a8b5d61f1b670e057fa5bb3b46bcfc96f53442467faf/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/29662ec5981eedb63451d94253c121403437cf209e58fdd35b582cc07cd6083e/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8ef4a37a-e3ae-4c25-abd8-8132d592be9e/volumes/kubernetes.io~projected/kube-api-access-5h58w DeviceMajor:0 DeviceMinor:258 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:398 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-442 DeviceMajor:0 DeviceMinor:442 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9688828c9124c834c36dd87c69743ea0641dd9b01c80effe16b52064f6f407bf/userdata/shm DeviceMajor:0 DeviceMinor:463 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-483 DeviceMajor:0 DeviceMinor:483 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-488 DeviceMajor:0 DeviceMinor:488 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a12ea3cb-3a3e-4fab-a040-4b4d00926c07/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:251 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:397 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-452 DeviceMajor:0 DeviceMinor:452 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f84bcfff-6360-4722-965f-8b3f152f2bdf/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:339 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:279 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c2d8995e2f9691d5a668296466755363b80f2a33df4df094e9fdd205125a1a1e/userdata/shm DeviceMajor:0 DeviceMinor:464 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e9e8e847-5b80-44de-9450-c5ae29a5b57b/volumes/kubernetes.io~projected/kube-api-access-bnpbz DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/08d2ae45-d701-4555-bc6b-670feece190f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:322 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:399 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/80fd5c23535b507f55088f79647078b229734ff457ca4877f93320bfe66d162d/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5dd720ed-3a46-454c-80c2-4850993e6b76/volumes/kubernetes.io~projected/kube-api-access-t5sww DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/08d2ae45-d701-4555-bc6b-670feece190f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:328 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:51 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:283 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:392 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/25a2eb84a3073687d9fcce0cbad1b493db4f67bc08c85ebb41007424685dee7d/userdata/shm DeviceMajor:0 DeviceMinor:467 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/856f4c46-f3c3-4afe-823e-1b77842932a4/volumes/kubernetes.io~projected/kube-api-access-n4hhg DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-62 DeviceMajor:0 DeviceMinor:62 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/91495a0e37b2ab21647b1e27e9cac114cf4fde7363c472c369a23b030bbb3fc0/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8ef4a37a-e3ae-4c25-abd8-8132d592be9e/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:257 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-431 DeviceMajor:0 DeviceMinor:431 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ca9b156d-7499-4b53-b17f-01dc63ae0c5e/volumes/kubernetes.io~projected/kube-api-access-cw546 DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/293c6f8c3cdeab4b9c8a75f939fcb0aa32173b995bbb9e800a75e477f96cd418/userdata/shm DeviceMajor:0 DeviceMinor:263 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f84bcfff-6360-4722-965f-8b3f152f2bdf/volumes/kubernetes.io~projected/kube-api-access-b8gsv DeviceMajor:0 DeviceMinor:340 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:395 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes/kubernetes.io~projected/kube-api-access-r52xd DeviceMajor:0 DeviceMinor:400 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-469 DeviceMajor:0 DeviceMinor:469 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2b7d2569-9fcf-4e91-9e2f-42beb94593a8/volumes/kubernetes.io~projected/kube-api-access-6jrdh DeviceMajor:0 DeviceMinor:53 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d89912fc-3948-4569-bfea-87c8dceedd36/volumes/kubernetes.io~projected/kube-api-access-g8wjg DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/28a49097c3af65785e13df0b750453b22cda3a77c1091663e4b32d1a34931038/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a12ea3cb-3a3e-4fab-a040-4b4d00926c07/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:246 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2da30edd-1262-4438-b719-da8055c3a763/volumes/kubernetes.io~projected/kube-api-access-9c69t DeviceMajor:0 DeviceMinor:254 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-475 DeviceMajor:0 DeviceMinor:475 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2b6cb291-3443-496d-8598-3b253102832f/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:49 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/08d2ae45-d701-4555-bc6b-670feece190f/volumes/kubernetes.io~projected/kube-api-access-pl8lr DeviceMajor:0 DeviceMinor:333 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:08bf3028d74ba1b MacAddress:2e:61:90:d0:71:74 Speed:10000 Mtu:8901} {Name:0b2ae9cdc4c4abb MacAddress:ea:f4:94:e8:a2:b7 Speed:10000 Mtu:8901} {Name:25a2eb84a307368 MacAddress:2a:f4:b1:87:bc:b0 Speed:10000 Mtu:8901} {Name:2fad3db0afe9655 MacAddress:32:21:64:3f:24:6f Speed:10000 Mtu:8901} {Name:3e52ebafbf95c6d MacAddress:0a:02:08:f7:21:78 Speed:10000 Mtu:8901} {Name:4d9d1b5cbd5ebbb MacAddress:66:89:bd:31:40:a9 Speed:10000 Mtu:8901} {Name:8a34f6736e45de3 MacAddress:f6:ec:df:9a:e9:f3 Speed:10000 Mtu:8901} {Name:8d00465d2d3bd71 MacAddress:d2:2f:5b:af:18:0a Speed:10000 Mtu:8901} {Name:9688828c9124c83 MacAddress:62:cd:87:85:5d:02 Speed:10000 Mtu:8901} {Name:a63c715a7cc2b7b MacAddress:ae:f3:cc:2d:3f:8f Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:cc:05:24:4d Speed:0 Mtu:9001} {Name:br-int MacAddress:c6:98:2c:e7:7d:19 Speed:0 Mtu:8901} {Name:c2d8995e2f9691d MacAddress:9e:77:7a:13:8e:1c Speed:10000 Mtu:8901} {Name:e9887e89089491f MacAddress:5a:3b:9e:65:47:ae Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:cc:05:24:4d Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:d6:fd:af:b4:92:01 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:1a:d9:9b:96:33:27 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 12:32:33.583954 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.583958 6841 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 12:32:33.584334 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.584043 6841 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 12:32:33.584413 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.584387 6841 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 12:32:33.584592 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.584415 6841 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-70.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 12:32:33.584719 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.584596 6841 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 12:32:33.584719 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.584609 6841 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 12:32:33.584719 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.584626 6841 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 12:32:33.584719 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.584644 6841 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 12:32:33.584908 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.584752 6841 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 12:32:33.585054 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.584973 6841 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 12:32:33.585144 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.585132 6841 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 12:32:33.585195 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.585179 6841 kubelet.go:493] "Attempting to sync node with API server" Oct 31 12:32:33.585253 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.585197 6841 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 12:32:33.585253 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.585212 6841 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 12:32:33.585253 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.585226 6841 kubelet.go:405] "Adding apiserver pod source" Oct 31 12:32:33.585253 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.585237 6841 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 12:32:33.585913 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.585895 6841 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 12:32:33.585913 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.585913 6841 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 12:32:33.586383 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.586368 6841 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 12:32:33.589184 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.589169 6841 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 12:32:33.589184 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.589183 6841 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 12:32:33.589289 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.589279 6841 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 12:32:33.589469 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.589458 6841 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 12:32:33.589499 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.589487 6841 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 12:32:33.589575 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.589567 6841 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 12:32:33.589604 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.589581 6841 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 12:32:33.589604 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.589588 6841 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 12:32:33.589604 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.589593 6841 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 12:32:33.589604 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.589599 6841 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 12:32:33.589604 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.589604 6841 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 12:32:33.589767 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.589610 6841 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 12:32:33.589767 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.589622 6841 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 12:32:33.589767 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.589629 6841 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 12:32:33.589767 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.589635 6841 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 12:32:33.589767 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.589648 6841 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 12:32:33.589767 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.589657 6841 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 12:32:33.589767 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.589692 6841 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 12:32:33.589767 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.589698 6841 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 12:32:33.593159 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.593135 6841 server.go:1263] "Started kubelet" Oct 31 12:32:33.593677 ip-10-0-138-70 systemd[1]: Started Kubernetes Kubelet. Oct 31 12:32:33.593872 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.593700 6841 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 12:32:33.594942 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.594917 6841 server.go:317] "Adding debug handlers to kubelet server" Oct 31 12:32:33.595430 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.595364 6841 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 12:32:33.595524 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.595455 6841 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 12:32:33.595767 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.595744 6841 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 12:32:33.601085 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.599138 6841 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 12:32:33.601085 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.599495 6841 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 12:32:33.601085 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.599623 6841 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 12:32:33.601085 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.599641 6841 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 12:32:33.601085 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.599705 6841 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 12:26:25 +0000 UTC" deadline="2028-04-06 13:06:58.180274499 +0000 UTC" Oct 31 12:32:33.601085 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.599751 6841 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12552h34m24.58052756s" Oct 31 12:32:33.602090 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.602065 6841 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 12:32:33.602448 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.602429 6841 factory.go:55] Registering systemd factory Oct 31 12:32:33.602530 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.602453 6841 factory.go:223] Registration of the systemd container factory successfully Oct 31 12:32:33.602602 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:33.602567 6841 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-70.us-east-2.compute.internal\" not found" Oct 31 12:32:33.603760 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.603724 6841 factory.go:153] Registering CRI-O factory Oct 31 12:32:33.603760 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.603743 6841 factory.go:223] Registration of the crio container factory successfully Oct 31 12:32:33.604318 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.603799 6841 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 12:32:33.604318 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.603819 6841 factory.go:103] Registering Raw factory Oct 31 12:32:33.604318 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.603835 6841 manager.go:1199] Started watching for new ooms in manager Oct 31 12:32:33.604318 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.604214 6841 manager.go:319] Starting recovery of all containers Oct 31 12:32:33.605089 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605038 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db5feef6-b3fc-4263-bfc1-66b070df97b4" volumeName="kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-web-config" seLinuxMountContext="" Oct 31 12:32:33.605215 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605092 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9e8e847-5b80-44de-9450-c5ae29a5b57b" volumeName="kubernetes.io/empty-dir/e9e8e847-5b80-44de-9450-c5ae29a5b57b-data-volume" seLinuxMountContext="" Oct 31 12:32:33.605215 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605114 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f84bcfff-6360-4722-965f-8b3f152f2bdf" volumeName="kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-service-ca" seLinuxMountContext="" Oct 31 12:32:33.605215 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605128 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f84bcfff-6360-4722-965f-8b3f152f2bdf" volumeName="kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-trusted-ca-bundle" seLinuxMountContext="" Oct 31 12:32:33.605215 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605140 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f84bcfff-6360-4722-965f-8b3f152f2bdf" volumeName="kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-oauth-config" seLinuxMountContext="" Oct 31 12:32:33.605215 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605152 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d89912fc-3948-4569-bfea-87c8dceedd36" volumeName="kubernetes.io/projected/d89912fc-3948-4569-bfea-87c8dceedd36-kube-api-access-g8wjg" seLinuxMountContext="" Oct 31 12:32:33.605215 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605162 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="178dd4e3-f913-4638-b4c7-8c9e96c82053" volumeName="kubernetes.io/projected/178dd4e3-f913-4638-b4c7-8c9e96c82053-kube-api-access-865nz" seLinuxMountContext="" Oct 31 12:32:33.605215 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605169 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ab66c79-2b0a-4671-9bf8-6aafea4e7454" volumeName="kubernetes.io/projected/6ab66c79-2b0a-4671-9bf8-6aafea4e7454-kube-api-access-qzs6g" seLinuxMountContext="" Oct 31 12:32:33.605215 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605180 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db5feef6-b3fc-4263-bfc1-66b070df97b4" volumeName="kubernetes.io/projected/db5feef6-b3fc-4263-bfc1-66b070df97b4-tls-assets" seLinuxMountContext="" Oct 31 12:32:33.605215 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605188 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db5feef6-b3fc-4263-bfc1-66b070df97b4" volumeName="kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 31 12:32:33.605215 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605195 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea2d5c42-8213-489b-ba92-369c0435c6e5" volumeName="kubernetes.io/empty-dir/ea2d5c42-8213-489b-ba92-369c0435c6e5-config-out" seLinuxMountContext="" Oct 31 12:32:33.605215 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605204 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b6cb291-3443-496d-8598-3b253102832f" volumeName="kubernetes.io/empty-dir/2b6cb291-3443-496d-8598-3b253102832f-tmp" seLinuxMountContext="" Oct 31 12:32:33.605215 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605211 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea2d5c42-8213-489b-ba92-369c0435c6e5" volumeName="kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 31 12:32:33.605215 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605219 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08d2ae45-d701-4555-bc6b-670feece190f" volumeName="kubernetes.io/empty-dir/08d2ae45-d701-4555-bc6b-670feece190f-tmp" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605230 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b7d2569-9fcf-4e91-9e2f-42beb94593a8" volumeName="kubernetes.io/configmap/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-cni-binary-copy" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605237 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8087a5a6-949e-491f-98b9-3360b93a6419" volumeName="kubernetes.io/projected/8087a5a6-949e-491f-98b9-3360b93a6419-kube-api-access-ggkt5" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605245 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ef4a37a-e3ae-4c25-abd8-8132d592be9e" volumeName="kubernetes.io/configmap/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605254 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db5feef6-b3fc-4263-bfc1-66b070df97b4" volumeName="kubernetes.io/empty-dir/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-db" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605261 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9e8e847-5b80-44de-9450-c5ae29a5b57b" volumeName="kubernetes.io/projected/e9e8e847-5b80-44de-9450-c5ae29a5b57b-kube-api-access-bnpbz" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605269 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea2d5c42-8213-489b-ba92-369c0435c6e5" volumeName="kubernetes.io/empty-dir/ea2d5c42-8213-489b-ba92-369c0435c6e5-alertmanager-main-db" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605278 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea2d5c42-8213-489b-ba92-369c0435c6e5" volumeName="kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605286 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b6cb291-3443-496d-8598-3b253102832f" volumeName="kubernetes.io/projected/2b6cb291-3443-496d-8598-3b253102832f-kube-api-access-xgk87" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605294 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a12ea3cb-3a3e-4fab-a040-4b4d00926c07" volumeName="kubernetes.io/projected/a12ea3cb-3a3e-4fab-a040-4b4d00926c07-kube-api-access-fj9cr" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605302 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a12ea3cb-3a3e-4fab-a040-4b4d00926c07" volumeName="kubernetes.io/secret/a12ea3cb-3a3e-4fab-a040-4b4d00926c07-kube-state-metrics-tls" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605322 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea2d5c42-8213-489b-ba92-369c0435c6e5" volumeName="kubernetes.io/projected/ea2d5c42-8213-489b-ba92-369c0435c6e5-tls-assets" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605330 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f84bcfff-6360-4722-965f-8b3f152f2bdf" volumeName="kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-oauth-serving-cert" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605339 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db5feef6-b3fc-4263-bfc1-66b070df97b4" volumeName="kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605368 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2da30edd-1262-4438-b719-da8055c3a763" volumeName="kubernetes.io/projected/2da30edd-1262-4438-b719-da8055c3a763-kube-api-access-9c69t" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605393 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2da30edd-1262-4438-b719-da8055c3a763" volumeName="kubernetes.io/secret/2da30edd-1262-4438-b719-da8055c3a763-openshift-state-metrics-tls" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605401 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca9b156d-7499-4b53-b17f-01dc63ae0c5e" volumeName="kubernetes.io/configmap/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-ovnkube-config" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605422 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca9b156d-7499-4b53-b17f-01dc63ae0c5e" volumeName="kubernetes.io/projected/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-kube-api-access-cw546" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605431 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db5feef6-b3fc-4263-bfc1-66b070df97b4" volumeName="kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605439 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db5feef6-b3fc-4263-bfc1-66b070df97b4" volumeName="kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605447 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea2d5c42-8213-489b-ba92-369c0435c6e5" volumeName="kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-cluster-tls-config" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605454 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f84bcfff-6360-4722-965f-8b3f152f2bdf" volumeName="kubernetes.io/empty-dir/f84bcfff-6360-4722-965f-8b3f152f2bdf-tmp" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605463 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08d2ae45-d701-4555-bc6b-670feece190f" volumeName="kubernetes.io/secret/08d2ae45-d701-4555-bc6b-670feece190f-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605470 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="178dd4e3-f913-4638-b4c7-8c9e96c82053" volumeName="kubernetes.io/secret/178dd4e3-f913-4638-b4c7-8c9e96c82053-cert" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605480 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f51d774-d999-4957-a165-0346972d65bc" volumeName="kubernetes.io/projected/4f51d774-d999-4957-a165-0346972d65bc-kube-api-access-fhn4r" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605489 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5dd720ed-3a46-454c-80c2-4850993e6b76" volumeName="kubernetes.io/projected/5dd720ed-3a46-454c-80c2-4850993e6b76-kube-api-access-t5sww" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605497 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ef4a37a-e3ae-4c25-abd8-8132d592be9e" volumeName="kubernetes.io/secret/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-node-exporter-tls" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605504 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db5feef6-b3fc-4263-bfc1-66b070df97b4" volumeName="kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605523 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db5feef6-b3fc-4263-bfc1-66b070df97b4" volumeName="kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605540 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea2d5c42-8213-489b-ba92-369c0435c6e5" volumeName="kubernetes.io/configmap/ea2d5c42-8213-489b-ba92-369c0435c6e5-metrics-client-ca" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605549 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5dd720ed-3a46-454c-80c2-4850993e6b76" volumeName="kubernetes.io/secret/5dd720ed-3a46-454c-80c2-4850993e6b76-metrics-tls" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605562 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db5feef6-b3fc-4263-bfc1-66b070df97b4" volumeName="kubernetes.io/projected/db5feef6-b3fc-4263-bfc1-66b070df97b4-kube-api-access-r52xd" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605570 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea2d5c42-8213-489b-ba92-369c0435c6e5" volumeName="kubernetes.io/projected/ea2d5c42-8213-489b-ba92-369c0435c6e5-kube-api-access-8wkrd" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605580 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea2d5c42-8213-489b-ba92-369c0435c6e5" volumeName="kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-config-volume" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605589 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed7f6a91-3f6e-409f-bc57-b5d0f8e67770" volumeName="kubernetes.io/secret/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-original-pull-secret" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605596 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f23a1c72-8427-4b6f-ad2b-cc76a88709c9" volumeName="kubernetes.io/configmap/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605603 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f84bcfff-6360-4722-965f-8b3f152f2bdf" volumeName="kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-serving-cert" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605611 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08d2ae45-d701-4555-bc6b-670feece190f" volumeName="kubernetes.io/configmap/08d2ae45-d701-4555-bc6b-670feece190f-metrics-client-ca" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605618 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29bc026b-84aa-4cbf-ba44-bb5c499e6bdc" volumeName="kubernetes.io/configmap/29bc026b-84aa-4cbf-ba44-bb5c499e6bdc-konnectivity-ca" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605626 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b7d2569-9fcf-4e91-9e2f-42beb94593a8" volumeName="kubernetes.io/projected/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-kube-api-access-6jrdh" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605634 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5dd720ed-3a46-454c-80c2-4850993e6b76" volumeName="kubernetes.io/empty-dir/5dd720ed-3a46-454c-80c2-4850993e6b76-tmp-dir" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605642 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ef4a37a-e3ae-4c25-abd8-8132d592be9e" volumeName="kubernetes.io/projected/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-kube-api-access-5h58w" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605649 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a12ea3cb-3a3e-4fab-a040-4b4d00926c07" volumeName="kubernetes.io/configmap/a12ea3cb-3a3e-4fab-a040-4b4d00926c07-metrics-client-ca" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605678 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a12ea3cb-3a3e-4fab-a040-4b4d00926c07" volumeName="kubernetes.io/configmap/a12ea3cb-3a3e-4fab-a040-4b4d00926c07-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605688 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db5feef6-b3fc-4263-bfc1-66b070df97b4" volumeName="kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-grpc-tls" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605697 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08d2ae45-d701-4555-bc6b-670feece190f" volumeName="kubernetes.io/projected/08d2ae45-d701-4555-bc6b-670feece190f-kube-api-access-pl8lr" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605704 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08d2ae45-d701-4555-bc6b-670feece190f" volumeName="kubernetes.io/secret/08d2ae45-d701-4555-bc6b-670feece190f-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605712 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08d2ae45-d701-4555-bc6b-670feece190f" volumeName="kubernetes.io/secret/08d2ae45-d701-4555-bc6b-670feece190f-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605720 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2da30edd-1262-4438-b719-da8055c3a763" volumeName="kubernetes.io/configmap/2da30edd-1262-4438-b719-da8055c3a763-metrics-client-ca" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605729 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8087a5a6-949e-491f-98b9-3360b93a6419" volumeName="kubernetes.io/configmap/8087a5a6-949e-491f-98b9-3360b93a6419-iptables-alerter-script" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605737 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ef4a37a-e3ae-4c25-abd8-8132d592be9e" volumeName="kubernetes.io/configmap/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-metrics-client-ca" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605746 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fcef325-e046-4c6d-8599-6c67115f382c" volumeName="kubernetes.io/configmap/9fcef325-e046-4c6d-8599-6c67115f382c-serviceca" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605754 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca9b156d-7499-4b53-b17f-01dc63ae0c5e" volumeName="kubernetes.io/configmap/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-env-overrides" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605762 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08d2ae45-d701-4555-bc6b-670feece190f" volumeName="kubernetes.io/secret/08d2ae45-d701-4555-bc6b-670feece190f-secret-grpc-tls" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605772 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08d2ae45-d701-4555-bc6b-670feece190f" volumeName="kubernetes.io/secret/08d2ae45-d701-4555-bc6b-670feece190f-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605780 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b7d2569-9fcf-4e91-9e2f-42beb94593a8" volumeName="kubernetes.io/configmap/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-multus-daemon-config" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605788 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fcef325-e046-4c6d-8599-6c67115f382c" volumeName="kubernetes.io/projected/9fcef325-e046-4c6d-8599-6c67115f382c-kube-api-access-5jd8l" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605796 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db5feef6-b3fc-4263-bfc1-66b070df97b4" volumeName="kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605804 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca9b156d-7499-4b53-b17f-01dc63ae0c5e" volumeName="kubernetes.io/secret/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605811 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea2d5c42-8213-489b-ba92-369c0435c6e5" volumeName="kubernetes.io/configmap/ea2d5c42-8213-489b-ba92-369c0435c6e5-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605818 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea2d5c42-8213-489b-ba92-369c0435c6e5" volumeName="kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-web-config" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605826 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29bc026b-84aa-4cbf-ba44-bb5c499e6bdc" volumeName="kubernetes.io/secret/29bc026b-84aa-4cbf-ba44-bb5c499e6bdc-agent-certs" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605834 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f51d774-d999-4957-a165-0346972d65bc" volumeName="kubernetes.io/empty-dir/4f51d774-d999-4957-a165-0346972d65bc-tmp-dir" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605845 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="856f4c46-f3c3-4afe-823e-1b77842932a4" volumeName="kubernetes.io/secret/856f4c46-f3c3-4afe-823e-1b77842932a4-metrics-certs" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605853 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ef4a37a-e3ae-4c25-abd8-8132d592be9e" volumeName="kubernetes.io/secret/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605862 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca9b156d-7499-4b53-b17f-01dc63ae0c5e" volumeName="kubernetes.io/configmap/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-ovnkube-script-lib" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605869 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db5feef6-b3fc-4263-bfc1-66b070df97b4" volumeName="kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-config" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605878 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db5feef6-b3fc-4263-bfc1-66b070df97b4" volumeName="kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605885 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea2d5c42-8213-489b-ba92-369c0435c6e5" volumeName="kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605904 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5dd720ed-3a46-454c-80c2-4850993e6b76" volumeName="kubernetes.io/configmap/5dd720ed-3a46-454c-80c2-4850993e6b76-config-volume" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605913 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a12ea3cb-3a3e-4fab-a040-4b4d00926c07" volumeName="kubernetes.io/empty-dir/a12ea3cb-3a3e-4fab-a040-4b4d00926c07-volume-directive-shadow" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605921 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a12ea3cb-3a3e-4fab-a040-4b4d00926c07" volumeName="kubernetes.io/secret/a12ea3cb-3a3e-4fab-a040-4b4d00926c07-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605929 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db5feef6-b3fc-4263-bfc1-66b070df97b4" volumeName="kubernetes.io/empty-dir/db5feef6-b3fc-4263-bfc1-66b070df97b4-config-out" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605938 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db5feef6-b3fc-4263-bfc1-66b070df97b4" volumeName="kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-metrics-client-certs" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605946 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9e8e847-5b80-44de-9450-c5ae29a5b57b" volumeName="kubernetes.io/configmap/e9e8e847-5b80-44de-9450-c5ae29a5b57b-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605952 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9e8e847-5b80-44de-9450-c5ae29a5b57b" volumeName="kubernetes.io/secret/e9e8e847-5b80-44de-9450-c5ae29a5b57b-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.605969 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea2d5c42-8213-489b-ba92-369c0435c6e5" volumeName="kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.606009 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2da30edd-1262-4438-b719-da8055c3a763" volumeName="kubernetes.io/secret/2da30edd-1262-4438-b719-da8055c3a763-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.606020 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b6cb291-3443-496d-8598-3b253102832f" volumeName="kubernetes.io/empty-dir/2b6cb291-3443-496d-8598-3b253102832f-etc-tuned" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.606032 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="856f4c46-f3c3-4afe-823e-1b77842932a4" volumeName="kubernetes.io/projected/856f4c46-f3c3-4afe-823e-1b77842932a4-kube-api-access-n4hhg" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.606042 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db5feef6-b3fc-4263-bfc1-66b070df97b4" volumeName="kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.606082 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db5feef6-b3fc-4263-bfc1-66b070df97b4" volumeName="kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.606093 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f23a1c72-8427-4b6f-ad2b-cc76a88709c9" volumeName="kubernetes.io/configmap/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-cni-binary-copy" seLinuxMountContext="" Oct 31 12:32:33.606091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.606150 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f23a1c72-8427-4b6f-ad2b-cc76a88709c9" volumeName="kubernetes.io/projected/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-kube-api-access-fwz6m" seLinuxMountContext="" Oct 31 12:32:33.609242 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.606164 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ef4a37a-e3ae-4c25-abd8-8132d592be9e" volumeName="kubernetes.io/empty-dir/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-node-exporter-textfile" seLinuxMountContext="" Oct 31 12:32:33.609242 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.606173 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f23a1c72-8427-4b6f-ad2b-cc76a88709c9" volumeName="kubernetes.io/configmap/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 12:32:33.609242 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.606181 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f84bcfff-6360-4722-965f-8b3f152f2bdf" volumeName="kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-config" seLinuxMountContext="" Oct 31 12:32:33.609242 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.606188 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f84bcfff-6360-4722-965f-8b3f152f2bdf" volumeName="kubernetes.io/projected/f84bcfff-6360-4722-965f-8b3f152f2bdf-kube-api-access-b8gsv" seLinuxMountContext="" Oct 31 12:32:33.609242 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.606197 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08d2ae45-d701-4555-bc6b-670feece190f" volumeName="kubernetes.io/secret/08d2ae45-d701-4555-bc6b-670feece190f-secret-thanos-querier-tls" seLinuxMountContext="" Oct 31 12:32:33.609242 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.606206 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db5feef6-b3fc-4263-bfc1-66b070df97b4" volumeName="kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-configmap-metrics-client-ca" seLinuxMountContext="" Oct 31 12:32:33.609242 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.606213 6841 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db5feef6-b3fc-4263-bfc1-66b070df97b4" volumeName="kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 31 12:32:33.609242 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.606221 6841 reconstruct.go:97] "Volume reconstruction finished" Oct 31 12:32:33.609242 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.606226 6841 reconciler.go:29] "Reconciler: start to sync state" Oct 31 12:32:33.609242 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.606662 6841 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:32:33.609691 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.609550 6841 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:32:33.610033 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.609977 6841 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:32:33.612346 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.612326 6841 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 12:32:33.632466 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.632423 6841 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 12:32:33.634186 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.634165 6841 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 12:32:33.634186 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.634186 6841 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 12:32:33.634294 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.634209 6841 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 12:32:33.634294 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:33.634249 6841 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 12:32:33.638629 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.638604 6841 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:32:33.659154 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.659126 6841 generic.go:356] "Generic (PLEG): container finished" podID="62138b49cad483f3e9534fc842796713" containerID="87ed63087d2abd6a68e84ca1a585e203c250221bc2fe933f4f7aaa8069e2801f" exitCode=0 Oct 31 12:32:33.664680 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.664651 6841 generic.go:356] "Generic (PLEG): container finished" podID="ea2d5c42-8213-489b-ba92-369c0435c6e5" containerID="475e94bf7c5e862bc04c010bf619e58932a88ded1c15f91acf581c4058959760" exitCode=0 Oct 31 12:32:33.678685 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.678656 6841 generic.go:356] "Generic (PLEG): container finished" podID="db5feef6-b3fc-4263-bfc1-66b070df97b4" containerID="4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc" exitCode=0 Oct 31 12:32:33.688310 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.688282 6841 generic.go:356] "Generic (PLEG): container finished" podID="f23a1c72-8427-4b6f-ad2b-cc76a88709c9" containerID="faf4539a6b9874b2afb3379486b68d8d3171b59b8f76a0951adb6c948e9454ec" exitCode=0 Oct 31 12:32:33.688392 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.688311 6841 generic.go:356] "Generic (PLEG): container finished" podID="f23a1c72-8427-4b6f-ad2b-cc76a88709c9" containerID="d85c9a834a2f716d2344f49a91500854101fd89a1168b80c86dbc48d5e5f09e6" exitCode=0 Oct 31 12:32:33.688392 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.688323 6841 generic.go:356] "Generic (PLEG): container finished" podID="f23a1c72-8427-4b6f-ad2b-cc76a88709c9" containerID="f31c0fb13238d5ddf59f35a74a3e0a0329b2343a58b850d64e58191a9b4735c3" exitCode=0 Oct 31 12:32:33.688392 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.688333 6841 generic.go:356] "Generic (PLEG): container finished" podID="f23a1c72-8427-4b6f-ad2b-cc76a88709c9" containerID="e82d1cf1b7a57a0964a18b86e1a23532f19f760215e3e9dd233db273b50b8302" exitCode=0 Oct 31 12:32:33.688392 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.688342 6841 generic.go:356] "Generic (PLEG): container finished" podID="f23a1c72-8427-4b6f-ad2b-cc76a88709c9" containerID="e8f134b1aa5d56622a42f23af37aa6503b139b7177e7620fdc45dd62710bbab5" exitCode=0 Oct 31 12:32:33.688392 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.688351 6841 generic.go:356] "Generic (PLEG): container finished" podID="f23a1c72-8427-4b6f-ad2b-cc76a88709c9" containerID="780ab5a0a4d2e2eb28ff2c5025383863f3ad065385433f87859d20140f5aac9f" exitCode=0 Oct 31 12:32:33.698167 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.698142 6841 generic.go:356] "Generic (PLEG): container finished" podID="8ef4a37a-e3ae-4c25-abd8-8132d592be9e" containerID="a3af927eaa09f018b068508d377b6fb747af174a3e060b91151cd573964c941b" exitCode=0 Oct 31 12:32:33.721781 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.721730 6841 manager.go:324] Recovery completed Oct 31 12:32:33.734436 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:33.734405 6841 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 31 12:32:33.746538 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.746515 6841 cpu_manager.go:226] "Starting" policy="none" Oct 31 12:32:33.746538 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.746534 6841 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 12:32:33.746667 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.746554 6841 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 12:32:33.746708 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.746683 6841 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 12:32:33.746708 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.746694 6841 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 12:32:33.746762 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.746710 6841 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 12:32:33.746762 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.746715 6841 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 12:32:33.746762 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.746720 6841 policy_none.go:50] "Start" Oct 31 12:32:33.746762 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.746729 6841 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 12:32:33.746762 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.746738 6841 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 12:32:33.748300 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.748288 6841 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 12:32:33.748333 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.748300 6841 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 12:32:33.748333 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.748306 6841 policy_none.go:44] "Start" Oct 31 12:32:33.752863 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.752838 6841 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 12:32:33.755380 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.755367 6841 manager.go:341] "Starting Device Plugin manager" Oct 31 12:32:33.755844 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:33.755395 6841 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 12:32:33.755916 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.755847 6841 server.go:87] "Starting device plugin registration server" Oct 31 12:32:33.756156 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.756143 6841 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 12:32:33.756221 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.756157 6841 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 12:32:33.756269 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.756240 6841 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 12:32:33.756374 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.756339 6841 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 12:32:33.756444 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.756383 6841 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 12:32:33.756444 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.756390 6841 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 12:32:33.756523 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.756443 6841 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T12:32:33.756372874Z","UUID":"02060e16-4d87-4e8b-b829-c031d054e377","Handler":null,"Name":"","Endpoint":""} Oct 31 12:32:33.757801 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.757783 6841 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 12:32:33.757894 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.757819 6841 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 12:32:33.758470 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:33.758445 6841 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 12:32:33.856266 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.856236 6841 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:32:33.858065 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.858045 6841 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-70.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:32:33.858159 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.858077 6841 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-70.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:32:33.858159 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.858094 6841 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-70.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:32:33.858159 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.858137 6841 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:32:33.873890 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.873872 6841 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:32:33.873943 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.873926 6841 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:32:33.935266 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.935224 6841 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-70.us-east-2.compute.internal"] Oct 31 12:32:33.935806 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.935772 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal" event={"ID":"62138b49cad483f3e9534fc842796713","Type":"ContainerStarted","Data":"fdcb1dbcaa299f2d31b8a91fa7f00b1f7a70dc61717d74c315491fadfd2022d2"} Oct 31 12:32:33.935882 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.935813 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal" event={"ID":"62138b49cad483f3e9534fc842796713","Type":"ContainerDied","Data":"87ed63087d2abd6a68e84ca1a585e203c250221bc2fe933f4f7aaa8069e2801f"} Oct 31 12:32:33.935882 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.935833 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal" event={"ID":"62138b49cad483f3e9534fc842796713","Type":"ContainerStarted","Data":"9a45582cbe71480fd292ad3a12002be22ce33917475324ae2cc1b8d3156c7363"} Oct 31 12:32:33.935882 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.935873 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-70.us-east-2.compute.internal" event={"ID":"578d51cfd2278b7149c7a4e11f7c55d2","Type":"ContainerStarted","Data":"e0ffd45d57c973c009fe120efa680e684f44c0512316092e2d8bc4b2881c35c9"} Oct 31 12:32:33.936104 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.935891 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-70.us-east-2.compute.internal" event={"ID":"578d51cfd2278b7149c7a4e11f7c55d2","Type":"ContainerStarted","Data":"2a5593f17ffaf057c3130cf9bb1de5849fbd82eddb2fbfe51ddf9144fd876f77"} Oct 31 12:32:33.936104 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.935927 6841 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:32:33.936104 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.935933 6841 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:32:33.950119 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.950099 6841 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 12:32:33.950197 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:33.950141 6841 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-70.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:32:33.950596 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:33.950582 6841 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 12:32:33.950645 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:33.950615 6841 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:32:34.014933 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.014876 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/62138b49cad483f3e9534fc842796713-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal\" (UID: \"62138b49cad483f3e9534fc842796713\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:32:34.014933 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.014906 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62138b49cad483f3e9534fc842796713-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal\" (UID: \"62138b49cad483f3e9534fc842796713\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:32:34.014933 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.014922 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/578d51cfd2278b7149c7a4e11f7c55d2-config\") pod \"kube-apiserver-proxy-ip-10-0-138-70.us-east-2.compute.internal\" (UID: \"578d51cfd2278b7149c7a4e11f7c55d2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:32:34.115467 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.115434 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/578d51cfd2278b7149c7a4e11f7c55d2-config\") pod \"kube-apiserver-proxy-ip-10-0-138-70.us-east-2.compute.internal\" (UID: \"578d51cfd2278b7149c7a4e11f7c55d2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:32:34.115595 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.115477 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/62138b49cad483f3e9534fc842796713-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal\" (UID: \"62138b49cad483f3e9534fc842796713\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:32:34.115595 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.115507 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62138b49cad483f3e9534fc842796713-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal\" (UID: \"62138b49cad483f3e9534fc842796713\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:32:34.115595 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.115535 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/578d51cfd2278b7149c7a4e11f7c55d2-config\") pod \"kube-apiserver-proxy-ip-10-0-138-70.us-east-2.compute.internal\" (UID: \"578d51cfd2278b7149c7a4e11f7c55d2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:32:34.115595 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.115555 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62138b49cad483f3e9534fc842796713-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal\" (UID: \"62138b49cad483f3e9534fc842796713\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:32:34.115728 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.115595 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/62138b49cad483f3e9534fc842796713-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal\" (UID: \"62138b49cad483f3e9534fc842796713\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:32:34.586083 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.586035 6841 apiserver.go:52] "Watching apiserver" Oct 31 12:32:34.602494 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.602465 6841 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 12:32:34.603234 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.603205 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-h9pd6","openshift-ingress-canary/ingress-canary-tm5c4","openshift-multus/network-metrics-daemon-8bvmr","openshift-dns/node-resolver-7p252","openshift-monitoring/thanos-querier-56dc986dcb-tghc2","openshift-multus/multus-additional-cni-plugins-plzf9","openshift-network-diagnostics/network-check-target-7sf98","openshift-dns/dns-default-4kn7k","openshift-multus/multus-rdcdb","openshift-cluster-node-tuning-operator/tuned-g98lz","openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4","openshift-console/console-668c5b5566-4scvp","openshift-monitoring/openshift-state-metrics-7bd7489957-v2x29","kube-system/global-pull-secret-syncer-blcxw","openshift-monitoring/prometheus-k8s-0","openshift-network-operator/iptables-alerter-27p2l","kube-system/konnectivity-agent-2wtkj","openshift-monitoring/alertmanager-main-0","kube-system/kube-apiserver-proxy-ip-10-0-138-70.us-east-2.compute.internal","openshift-image-registry/node-ca-chf5k","openshift-insights/insights-runtime-extractor-9txq2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal","openshift-monitoring/node-exporter-26lht"] Oct 31 12:32:34.607732 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.607698 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 12:32:34.607846 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.607776 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 12:32:34.607846 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.607801 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 12:32:34.607846 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.607814 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 12:32:34.607846 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.607843 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 12:32:34.609045 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.609016 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 12:32:34.609358 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.609336 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bkb2g\"" Oct 31 12:32:34.609475 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.609374 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l77g8\"" Oct 31 12:32:34.609475 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.609411 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 12:32:34.609582 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.609484 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-l5d6v\"" Oct 31 12:32:34.609819 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.609802 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 12:32:34.609877 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.609833 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zfd9q\"" Oct 31 12:32:34.610601 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.610579 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 12:32:34.610601 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.610595 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 12:32:34.610737 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.610585 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 12:32:34.610737 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.610655 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 12:32:34.610737 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.610660 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 12:32:34.610737 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.610711 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zzvpb\"" Oct 31 12:32:34.610917 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.610765 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 12:32:34.610917 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.610770 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 12:32:34.610917 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.610781 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 12:32:34.610917 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.610875 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7phhl\"" Oct 31 12:32:34.612543 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.612520 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 12:32:34.612717 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.612700 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 12:32:34.612879 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.612858 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 12:32:34.612947 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.612901 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zbx6v\"" Oct 31 12:32:34.613079 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.613065 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 12:32:34.613172 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.613143 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 12:32:34.613284 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.613265 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8c8p7\"" Oct 31 12:32:34.613661 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.613641 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 12:32:34.614133 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.614117 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 12:32:34.614238 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.614217 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 12:32:34.614324 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.614289 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 12:32:34.614729 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.614706 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w8vnc\"" Oct 31 12:32:34.614812 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.614736 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 12:32:34.614938 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.614919 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 12:32:34.615085 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.615066 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 12:32:34.615085 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.615079 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 12:32:34.615184 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.615086 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hk7cx\"" Oct 31 12:32:34.615358 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.615341 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 12:32:34.615427 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.615388 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fmbl2\"" Oct 31 12:32:34.615427 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.615410 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 12:32:34.615552 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.615536 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 12:32:34.615607 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.615549 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 12:32:34.615607 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.615553 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-52dg7\"" Oct 31 12:32:34.615706 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.615641 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6k457\"" Oct 31 12:32:34.615757 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.615700 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 12:32:34.615851 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.615834 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 12:32:34.615906 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.615874 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 12:32:34.615951 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.615905 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 12:32:34.616313 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.616292 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 12:32:34.616898 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.616878 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 12:32:34.616898 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.616887 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 12:32:34.617046 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.616997 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 12:32:34.617591 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.617567 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 12:32:34.617667 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.617575 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 31 12:32:34.617667 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.617581 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mrn76\"" Oct 31 12:32:34.617667 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.617583 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 12:32:34.617667 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.617623 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4c2ww\"" Oct 31 12:32:34.617667 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.617614 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 31 12:32:34.617667 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.617641 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-8bz5h\"" Oct 31 12:32:34.617872 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.617707 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nthx6\"" Oct 31 12:32:34.617872 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.617762 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 31 12:32:34.618007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.617968 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 31 12:32:34.618515 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.618502 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 12:32:34.618589 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.618577 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 31 12:32:34.618668 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.618656 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zhwtd\"" Oct 31 12:32:34.618726 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.618713 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 12:32:34.618757 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.618721 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 12:32:34.625309 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.625287 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 12:32:34.625423 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.625316 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 12:32:34.625423 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.625354 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 12:32:34.625423 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.625372 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 12:32:34.626780 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.626758 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 12:32:34.626780 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.626764 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 12:32:34.626917 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.626820 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 12:32:34.626917 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.626857 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 12:32:34.626917 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.626864 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 31 12:32:34.626917 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.626883 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 12:32:34.627140 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.626932 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 12:32:34.627140 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.626958 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jklxq\"" Oct 31 12:32:34.627140 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.627114 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 12:32:34.628566 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.628547 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 12:32:34.628703 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.628690 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 12:32:34.640094 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.640077 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 12:32:34.640221 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.640205 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 12:32:34.702771 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.702747 6841 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:32:34.702935 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.702901 6841 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:32:34.704720 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.704702 6841 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 12:32:34.709175 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.709147 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-xcxdk\"" Oct 31 12:32:34.709268 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.709147 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 12:32:34.709268 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.709250 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-bgz64\"" Oct 31 12:32:34.709268 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.709257 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-c2m7gphuijb36\"" Oct 31 12:32:34.709382 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.709345 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 31 12:32:34.709814 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.709791 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 31 12:32:34.709930 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.709867 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 12:32:34.709930 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.709890 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-crggt40j4ekbj\"" Oct 31 12:32:34.709930 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.709900 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 12:32:34.709930 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.709919 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 12:32:34.709930 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.709932 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 31 12:32:34.710195 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.710176 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 31 12:32:34.710270 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.710183 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 12:32:34.710270 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.710213 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 12:32:34.710270 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.710212 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 12:32:34.710270 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.710214 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 12:32:34.710270 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.710268 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 12:32:34.714328 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.714311 6841 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 12:32:34.714416 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:34.714361 6841 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-70.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:32:34.714901 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.714886 6841 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 12:32:34.714971 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:34.714926 6841 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.us-east-2.compute.internal" Oct 31 12:32:34.717186 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.717166 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d89912fc-3948-4569-bfea-87c8dceedd36-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:32:34.717246 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.717196 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5dd720ed-3a46-454c-80c2-4850993e6b76-metrics-tls\") pod \"dns-default-4kn7k\" (UID: \"5dd720ed-3a46-454c-80c2-4850993e6b76\") " pod="openshift-dns/dns-default-4kn7k" Oct 31 12:32:34.717246 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.717212 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08d2ae45-d701-4555-bc6b-670feece190f-tmp\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:34.717246 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.717227 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-865nz\" (UniqueName: \"kubernetes.io/projected/178dd4e3-f913-4638-b4c7-8c9e96c82053-kube-api-access-865nz\") pod \"ingress-canary-tm5c4\" (UID: \"178dd4e3-f913-4638-b4c7-8c9e96c82053\") " pod="openshift-ingress-canary/ingress-canary-tm5c4" Oct 31 12:32:34.717246 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.717244 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea2d5c42-8213-489b-ba92-369c0435c6e5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:34.717397 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.717293 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-var-lib-kubelet\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.717397 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.717373 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj9cr\" (UniqueName: \"kubernetes.io/projected/a12ea3cb-3a3e-4fab-a040-4b4d00926c07-kube-api-access-fj9cr\") pod \"kube-state-metrics-5dc8d7d89b-lvmr6\" (UID: \"a12ea3cb-3a3e-4fab-a040-4b4d00926c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" Oct 31 12:32:34.717494 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.717451 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08d2ae45-d701-4555-bc6b-670feece190f-tmp\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:34.717494 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.717471 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5dd720ed-3a46-454c-80c2-4850993e6b76-metrics-tls\") pod \"dns-default-4kn7k\" (UID: \"5dd720ed-3a46-454c-80c2-4850993e6b76\") " pod="openshift-dns/dns-default-4kn7k" Oct 31 12:32:34.717560 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.717525 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-cni-binary-copy\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:32:34.717614 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.717586 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/db5feef6-b3fc-4263-bfc1-66b070df97b4-config-out\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.717690 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.717673 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea2d5c42-8213-489b-ba92-369c0435c6e5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:34.717740 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.717727 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-config-volume\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:34.717780 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.717740 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/db5feef6-b3fc-4263-bfc1-66b070df97b4-config-out\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.717814 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.717775 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea2d5c42-8213-489b-ba92-369c0435c6e5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:34.717814 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.717787 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-cni-binary-copy\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:32:34.717875 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.717854 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.717946 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.717929 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-config-volume\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:34.718014 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.717938 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-run\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.718048 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.718011 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a12ea3cb-3a3e-4fab-a040-4b4d00926c07-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-lvmr6\" (UID: \"a12ea3cb-3a3e-4fab-a040-4b4d00926c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" Oct 31 12:32:34.718048 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.718035 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea2d5c42-8213-489b-ba92-369c0435c6e5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:34.718142 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.718049 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-root\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:34.718142 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.718078 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.718242 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.718171 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-etc-systemd\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.718242 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.718193 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/29bc026b-84aa-4cbf-ba44-bb5c499e6bdc-konnectivity-ca\") pod \"konnectivity-agent-2wtkj\" (UID: \"29bc026b-84aa-4cbf-ba44-bb5c499e6bdc\") " pod="kube-system/konnectivity-agent-2wtkj" Oct 31 12:32:34.718242 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.718211 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2da30edd-1262-4438-b719-da8055c3a763-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-v2x29\" (UID: \"2da30edd-1262-4438-b719-da8055c3a763\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2x29" Oct 31 12:32:34.718242 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.718236 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-system-cni-dir\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:32:34.718404 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.718260 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.718404 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.718300 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-node-exporter-textfile\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:34.718404 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.718307 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a12ea3cb-3a3e-4fab-a040-4b4d00926c07-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-lvmr6\" (UID: \"a12ea3cb-3a3e-4fab-a040-4b4d00926c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" Oct 31 12:32:34.718404 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.718365 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-node-exporter-textfile\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:34.718561 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.718414 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-oauth-config\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:34.718561 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.718435 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2da30edd-1262-4438-b719-da8055c3a763-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-v2x29\" (UID: \"2da30edd-1262-4438-b719-da8055c3a763\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2x29" Oct 31 12:32:34.718561 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.718445 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-config\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:34.718561 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.718471 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-var-lib-openvswitch\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.718561 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.718435 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/29bc026b-84aa-4cbf-ba44-bb5c499e6bdc-konnectivity-ca\") pod \"konnectivity-agent-2wtkj\" (UID: \"29bc026b-84aa-4cbf-ba44-bb5c499e6bdc\") " pod="kube-system/konnectivity-agent-2wtkj" Oct 31 12:32:34.718561 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.718494 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-host-run-multus-certs\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.718561 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:34.718496 6841 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 31 12:32:34.718561 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:34.718503 6841 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 31 12:32:34.718561 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.718526 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4f51d774-d999-4957-a165-0346972d65bc-hosts-file\") pod \"node-resolver-7p252\" (UID: \"4f51d774-d999-4957-a165-0346972d65bc\") " pod="openshift-dns/node-resolver-7p252" Oct 31 12:32:34.718911 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:34.718617 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-config podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:32:35.218586385 +0000 UTC m=+1.701709265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-config") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : object "openshift-console"/"console-config" not registered Oct 31 12:32:34.718911 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.718631 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8087a5a6-949e-491f-98b9-3360b93a6419-host-slash\") pod \"iptables-alerter-27p2l\" (UID: \"8087a5a6-949e-491f-98b9-3360b93a6419\") " pod="openshift-network-operator/iptables-alerter-27p2l" Oct 31 12:32:34.718911 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:34.718657 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-oauth-config podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:32:35.218640746 +0000 UTC m=+1.701763650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-oauth-config") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : object "openshift-console"/"console-oauth-config" not registered Oct 31 12:32:34.718911 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.718719 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4hhg\" (UniqueName: \"kubernetes.io/projected/856f4c46-f3c3-4afe-823e-1b77842932a4-kube-api-access-n4hhg\") pod \"network-metrics-daemon-8bvmr\" (UID: \"856f4c46-f3c3-4afe-823e-1b77842932a4\") " pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:32:34.718911 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.718748 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.718911 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.718832 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-original-pull-secret\") pod \"global-pull-secret-syncer-blcxw\" (UID: \"ed7f6a91-3f6e-409f-bc57-b5d0f8e67770\") " pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:32:34.718911 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.718862 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ea2d5c42-8213-489b-ba92-369c0435c6e5-config-out\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:34.718911 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.718888 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.718911 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.718914 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:34.719361 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.718940 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:34.719361 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.718965 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-etc-modprobe-d\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.719361 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.718969 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.719361 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.719009 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a12ea3cb-3a3e-4fab-a040-4b4d00926c07-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-lvmr6\" (UID: \"a12ea3cb-3a3e-4fab-a040-4b4d00926c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" Oct 31 12:32:34.719361 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.719056 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ea2d5c42-8213-489b-ba92-369c0435c6e5-config-out\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:34.719361 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.719055 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:32:34.719361 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.719105 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwz6m\" (UniqueName: \"kubernetes.io/projected/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-kube-api-access-fwz6m\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:32:34.719361 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.719150 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-node-exporter-wtmp\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:34.719361 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.719174 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-config\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.719361 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.719202 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:34.719361 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.719219 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.719361 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.719221 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:32:34.719361 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.719234 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a12ea3cb-3a3e-4fab-a040-4b4d00926c07-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-lvmr6\" (UID: \"a12ea3cb-3a3e-4fab-a040-4b4d00926c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" Oct 31 12:32:34.719361 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.719235 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:34.719361 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.719259 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-ovnkube-config\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.719361 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.719276 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-original-pull-secret\") pod \"global-pull-secret-syncer-blcxw\" (UID: \"ed7f6a91-3f6e-409f-bc57-b5d0f8e67770\") " pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:32:34.719361 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.719286 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-env-overrides\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.719361 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.719311 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgk87\" (UniqueName: \"kubernetes.io/projected/2b6cb291-3443-496d-8598-3b253102832f-kube-api-access-xgk87\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.720017 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.719453 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.720017 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.719459 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-ovnkube-config\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.720017 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.719493 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:32:34.720017 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.719518 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-env-overrides\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.720017 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.719535 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/08d2ae45-d701-4555-bc6b-670feece190f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:34.720017 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.719588 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.720017 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.719640 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-config\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.720017 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.719638 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8gsv\" (UniqueName: \"kubernetes.io/projected/f84bcfff-6360-4722-965f-8b3f152f2bdf-kube-api-access-b8gsv\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:34.720017 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.719688 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:32:34.720017 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.719751 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-hostroot\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.720017 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.719783 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f51d774-d999-4957-a165-0346972d65bc-tmp-dir\") pod \"node-resolver-7p252\" (UID: \"4f51d774-d999-4957-a165-0346972d65bc\") " pod="openshift-dns/node-resolver-7p252" Oct 31 12:32:34.720017 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.719786 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/08d2ae45-d701-4555-bc6b-670feece190f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:34.720017 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.719815 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dd720ed-3a46-454c-80c2-4850993e6b76-tmp-dir\") pod \"dns-default-4kn7k\" (UID: \"5dd720ed-3a46-454c-80c2-4850993e6b76\") " pod="openshift-dns/dns-default-4kn7k" Oct 31 12:32:34.720017 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.719858 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f51d774-d999-4957-a165-0346972d65bc-tmp-dir\") pod \"node-resolver-7p252\" (UID: \"4f51d774-d999-4957-a165-0346972d65bc\") " pod="openshift-dns/node-resolver-7p252" Oct 31 12:32:34.720017 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.719886 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzs6g\" (UniqueName: \"kubernetes.io/projected/6ab66c79-2b0a-4671-9bf8-6aafea4e7454-kube-api-access-qzs6g\") pod \"network-check-target-7sf98\" (UID: \"6ab66c79-2b0a-4671-9bf8-6aafea4e7454\") " pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:32:34.720017 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.719903 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-run-ovn-kubernetes\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.720017 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720014 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dd720ed-3a46-454c-80c2-4850993e6b76-tmp-dir\") pod \"dns-default-4kn7k\" (UID: \"5dd720ed-3a46-454c-80c2-4850993e6b76\") " pod="openshift-dns/dns-default-4kn7k" Oct 31 12:32:34.720509 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720040 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-multus-cni-dir\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.720509 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720066 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhn4r\" (UniqueName: \"kubernetes.io/projected/4f51d774-d999-4957-a165-0346972d65bc-kube-api-access-fhn4r\") pod \"node-resolver-7p252\" (UID: \"4f51d774-d999-4957-a165-0346972d65bc\") " pod="openshift-dns/node-resolver-7p252" Oct 31 12:32:34.720509 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720092 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h58w\" (UniqueName: \"kubernetes.io/projected/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-kube-api-access-5h58w\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:34.720509 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720222 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.720509 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720248 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d89912fc-3948-4569-bfea-87c8dceedd36-registration-dir\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:32:34.720509 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720269 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/08d2ae45-d701-4555-bc6b-670feece190f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:34.720509 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720293 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:34.720509 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720315 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-ovn-node-metrics-cert\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.720509 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720331 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-cnibin\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.720509 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720347 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-etc-kubernetes\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.720509 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720362 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jrdh\" (UniqueName: \"kubernetes.io/projected/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-kube-api-access-6jrdh\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.720509 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720381 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-lib-modules\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.720509 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720396 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e9e8e847-5b80-44de-9450-c5ae29a5b57b-data-volume\") pod \"insights-runtime-extractor-9txq2\" (UID: \"e9e8e847-5b80-44de-9450-c5ae29a5b57b\") " pod="openshift-insights/insights-runtime-extractor-9txq2" Oct 31 12:32:34.720509 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720411 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a12ea3cb-3a3e-4fab-a040-4b4d00926c07-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-lvmr6\" (UID: \"a12ea3cb-3a3e-4fab-a040-4b4d00926c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" Oct 31 12:32:34.720509 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720431 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.720509 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720447 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-run-systemd\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.720509 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720462 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-host-run-k8s-cni-cncf-io\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.720509 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720477 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f84bcfff-6360-4722-965f-8b3f152f2bdf-tmp\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:34.720509 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720492 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/db5feef6-b3fc-4263-bfc1-66b070df97b4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.720509 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720507 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720520 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-node-log\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720535 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-host-var-lib-kubelet\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720549 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-multus-daemon-config\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720573 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-etc-sysctl-d\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720587 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b6cb291-3443-496d-8598-3b253102832f-tmp\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720601 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/29bc026b-84aa-4cbf-ba44-bb5c499e6bdc-agent-certs\") pod \"konnectivity-agent-2wtkj\" (UID: \"29bc026b-84aa-4cbf-ba44-bb5c499e6bdc\") " pod="kube-system/konnectivity-agent-2wtkj" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720618 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-kubelet-config\") pod \"global-pull-secret-syncer-blcxw\" (UID: \"ed7f6a91-3f6e-409f-bc57-b5d0f8e67770\") " pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720633 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720647 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-kubelet\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720664 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw546\" (UniqueName: \"kubernetes.io/projected/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-kube-api-access-cw546\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720682 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-cni-binary-copy\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720697 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2da30edd-1262-4438-b719-da8055c3a763-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-v2x29\" (UID: \"2da30edd-1262-4438-b719-da8055c3a763\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2x29" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720714 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/08d2ae45-d701-4555-bc6b-670feece190f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720730 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08d2ae45-d701-4555-bc6b-670feece190f-metrics-client-ca\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720751 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-metrics-client-ca\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720766 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-trusted-ca-bundle\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720781 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-slash\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720795 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-run-netns\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720808 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-host-var-lib-cni-bin\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720824 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d89912fc-3948-4569-bfea-87c8dceedd36-socket-dir\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720849 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720866 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d89912fc-3948-4569-bfea-87c8dceedd36-etc-selinux\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.720881 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-web-config\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721106 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9e8e847-5b80-44de-9450-c5ae29a5b57b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9txq2\" (UID: \"e9e8e847-5b80-44de-9450-c5ae29a5b57b\") " pod="openshift-insights/insights-runtime-extractor-9txq2" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721125 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9fcef325-e046-4c6d-8599-6c67115f382c-host\") pod \"node-ca-chf5k\" (UID: \"9fcef325-e046-4c6d-8599-6c67115f382c\") " pod="openshift-image-registry/node-ca-chf5k" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721139 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-etc-openvswitch\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721155 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-etc-kubernetes\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721171 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e9e8e847-5b80-44de-9450-c5ae29a5b57b-crio-socket\") pod \"insights-runtime-extractor-9txq2\" (UID: \"e9e8e847-5b80-44de-9450-c5ae29a5b57b\") " pod="openshift-insights/insights-runtime-extractor-9txq2" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721187 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2da30edd-1262-4438-b719-da8055c3a763-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-v2x29\" (UID: \"2da30edd-1262-4438-b719-da8055c3a763\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2x29" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721206 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c69t\" (UniqueName: \"kubernetes.io/projected/2da30edd-1262-4438-b719-da8055c3a763-kube-api-access-9c69t\") pod \"openshift-state-metrics-7bd7489957-v2x29\" (UID: \"2da30edd-1262-4438-b719-da8055c3a763\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2x29" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721223 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a12ea3cb-3a3e-4fab-a040-4b4d00926c07-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-lvmr6\" (UID: \"a12ea3cb-3a3e-4fab-a040-4b4d00926c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721239 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5sww\" (UniqueName: \"kubernetes.io/projected/5dd720ed-3a46-454c-80c2-4850993e6b76-kube-api-access-t5sww\") pod \"dns-default-4kn7k\" (UID: \"5dd720ed-3a46-454c-80c2-4850993e6b76\") " pod="openshift-dns/dns-default-4kn7k" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721255 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721271 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggkt5\" (UniqueName: \"kubernetes.io/projected/8087a5a6-949e-491f-98b9-3360b93a6419-kube-api-access-ggkt5\") pod \"iptables-alerter-27p2l\" (UID: \"8087a5a6-949e-491f-98b9-3360b93a6419\") " pod="openshift-network-operator/iptables-alerter-27p2l" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721287 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/08d2ae45-d701-4555-bc6b-670feece190f-secret-grpc-tls\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721286 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-web-config\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721305 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl8lr\" (UniqueName: \"kubernetes.io/projected/08d2ae45-d701-4555-bc6b-670feece190f-kube-api-access-pl8lr\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721327 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/178dd4e3-f913-4638-b4c7-8c9e96c82053-cert\") pod \"ingress-canary-tm5c4\" (UID: \"178dd4e3-f913-4638-b4c7-8c9e96c82053\") " pod="openshift-ingress-canary/ingress-canary-tm5c4" Oct 31 12:32:34.721329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721351 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721368 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-node-exporter-accelerators-collector-config\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721395 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721415 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-cni-bin\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721431 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-service-ca\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721447 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ea2d5c42-8213-489b-ba92-369c0435c6e5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721465 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wkrd\" (UniqueName: \"kubernetes.io/projected/ea2d5c42-8213-489b-ba92-369c0435c6e5-kube-api-access-8wkrd\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721465 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-multus-daemon-config\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721481 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-systemd-units\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721498 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a12ea3cb-3a3e-4fab-a040-4b4d00926c07-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-lvmr6\" (UID: \"a12ea3cb-3a3e-4fab-a040-4b4d00926c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721515 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8087a5a6-949e-491f-98b9-3360b93a6419-iptables-alerter-script\") pod \"iptables-alerter-27p2l\" (UID: \"8087a5a6-949e-491f-98b9-3360b93a6419\") " pod="openshift-network-operator/iptables-alerter-27p2l" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721519 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b6cb291-3443-496d-8598-3b253102832f-tmp\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721533 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/08d2ae45-d701-4555-bc6b-670feece190f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721550 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-sys\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721567 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721582 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-host-run-netns\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721599 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d89912fc-3948-4569-bfea-87c8dceedd36-sys-fs\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721614 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-etc-sysctl-conf\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721631 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721647 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d89912fc-3948-4569-bfea-87c8dceedd36-device-dir\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721664 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5dd720ed-3a46-454c-80c2-4850993e6b76-config-volume\") pod \"dns-default-4kn7k\" (UID: \"5dd720ed-3a46-454c-80c2-4850993e6b76\") " pod="openshift-dns/dns-default-4kn7k" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721747 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/29bc026b-84aa-4cbf-ba44-bb5c499e6bdc-agent-certs\") pod \"konnectivity-agent-2wtkj\" (UID: \"29bc026b-84aa-4cbf-ba44-bb5c499e6bdc\") " pod="kube-system/konnectivity-agent-2wtkj" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721813 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-web-config\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721830 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-run-openvswitch\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721847 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-run-ovn\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721865 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-ovnkube-script-lib\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721882 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721899 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r52xd\" (UniqueName: \"kubernetes.io/projected/db5feef6-b3fc-4263-bfc1-66b070df97b4-kube-api-access-r52xd\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721914 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-dbus\") pod \"global-pull-secret-syncer-blcxw\" (UID: \"ed7f6a91-3f6e-409f-bc57-b5d0f8e67770\") " pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721929 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ea2d5c42-8213-489b-ba92-369c0435c6e5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721946 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721964 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9fcef325-e046-4c6d-8599-6c67115f382c-serviceca\") pod \"node-ca-chf5k\" (UID: \"9fcef325-e046-4c6d-8599-6c67115f382c\") " pod="openshift-image-registry/node-ca-chf5k" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721959 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a12ea3cb-3a3e-4fab-a040-4b4d00926c07-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-lvmr6\" (UID: \"a12ea3cb-3a3e-4fab-a040-4b4d00926c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.721993 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-cni-netd\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.722011 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-etc-sysconfig\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.722027 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-host\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.722045 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnpbz\" (UniqueName: \"kubernetes.io/projected/e9e8e847-5b80-44de-9450-c5ae29a5b57b-kube-api-access-bnpbz\") pod \"insights-runtime-extractor-9txq2\" (UID: \"e9e8e847-5b80-44de-9450-c5ae29a5b57b\") " pod="openshift-insights/insights-runtime-extractor-9txq2" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.722062 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-serving-cert\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.722079 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.722097 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-multus-socket-dir-parent\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.722113 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-host-var-lib-cni-multus\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.722127 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-sys\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.722144 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/856f4c46-f3c3-4afe-823e-1b77842932a4-metrics-certs\") pod \"network-metrics-daemon-8bvmr\" (UID: \"856f4c46-f3c3-4afe-823e-1b77842932a4\") " pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.722157 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8087a5a6-949e-491f-98b9-3360b93a6419-iptables-alerter-script\") pod \"iptables-alerter-27p2l\" (UID: \"8087a5a6-949e-491f-98b9-3360b93a6419\") " pod="openshift-network-operator/iptables-alerter-27p2l" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.722160 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jd8l\" (UniqueName: \"kubernetes.io/projected/9fcef325-e046-4c6d-8599-6c67115f382c-kube-api-access-5jd8l\") pod \"node-ca-chf5k\" (UID: \"9fcef325-e046-4c6d-8599-6c67115f382c\") " pod="openshift-image-registry/node-ca-chf5k" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.722203 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-log-socket\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.722234 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-system-cni-dir\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.722265 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2b6cb291-3443-496d-8598-3b253102832f-etc-tuned\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.722299 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9e8e847-5b80-44de-9450-c5ae29a5b57b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9txq2\" (UID: \"e9e8e847-5b80-44de-9450-c5ae29a5b57b\") " pod="openshift-insights/insights-runtime-extractor-9txq2" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.722346 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-cnibin\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.722378 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8wjg\" (UniqueName: \"kubernetes.io/projected/d89912fc-3948-4569-bfea-87c8dceedd36-kube-api-access-g8wjg\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.722407 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5dd720ed-3a46-454c-80c2-4850993e6b76-config-volume\") pod \"dns-default-4kn7k\" (UID: \"5dd720ed-3a46-454c-80c2-4850993e6b76\") " pod="openshift-dns/dns-default-4kn7k" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.722420 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-node-exporter-tls\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.722464 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-oauth-serving-cert\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.722534 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.722579 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/08d2ae45-d701-4555-bc6b-670feece190f-secret-thanos-querier-tls\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.722605 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.722624 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-multus-conf-dir\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.722646 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-cni-binary-copy\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.722663 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9e8e847-5b80-44de-9450-c5ae29a5b57b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9txq2\" (UID: \"e9e8e847-5b80-44de-9450-c5ae29a5b57b\") " pod="openshift-insights/insights-runtime-extractor-9txq2" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:34.722486 6841 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:34.722781 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-service-ca podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:32:35.222769518 +0000 UTC m=+1.705892396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-service-ca") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : object "openshift-console"/"service-ca" not registered Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.722791 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-ovnkube-script-lib\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.722829 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/08d2ae45-d701-4555-bc6b-670feece190f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.722858 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2da30edd-1262-4438-b719-da8055c3a763-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-v2x29\" (UID: \"2da30edd-1262-4438-b719-da8055c3a763\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2x29" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.722863 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-node-exporter-tls\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.722929 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.722943 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2b6cb291-3443-496d-8598-3b253102832f-etc-tuned\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:34.722709 6841 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:34.723000 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-oauth-serving-cert podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:32:35.222976075 +0000 UTC m=+1.706098953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-oauth-serving-cert") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : object "openshift-console"/"oauth-serving-cert" not registered Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.723003 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.723061 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08d2ae45-d701-4555-bc6b-670feece190f-metrics-client-ca\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.723062 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/08d2ae45-d701-4555-bc6b-670feece190f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.722662 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.723140 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.723150 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/08d2ae45-d701-4555-bc6b-670feece190f-secret-thanos-querier-tls\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.723150 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/08d2ae45-d701-4555-bc6b-670feece190f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.723247 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-metrics-client-ca\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.723262 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.723297 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/08d2ae45-d701-4555-bc6b-670feece190f-secret-grpc-tls\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:34.723326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.723330 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ea2d5c42-8213-489b-ba92-369c0435c6e5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:34.725965 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.723410 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-ovn-node-metrics-cert\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.725965 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:34.723431 6841 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 12:32:34.725965 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.723433 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ea2d5c42-8213-489b-ba92-369c0435c6e5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:34.725965 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:34.723459 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-trusted-ca-bundle podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:32:35.223450875 +0000 UTC m=+1.706573754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-trusted-ca-bundle") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 12:32:34.725965 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.723483 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2da30edd-1262-4438-b719-da8055c3a763-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-v2x29\" (UID: \"2da30edd-1262-4438-b719-da8055c3a763\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2x29" Oct 31 12:32:34.725965 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.723518 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/178dd4e3-f913-4638-b4c7-8c9e96c82053-cert\") pod \"ingress-canary-tm5c4\" (UID: \"178dd4e3-f913-4638-b4c7-8c9e96c82053\") " pod="openshift-ingress-canary/ingress-canary-tm5c4" Oct 31 12:32:34.725965 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.723575 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e9e8e847-5b80-44de-9450-c5ae29a5b57b-data-volume\") pod \"insights-runtime-extractor-9txq2\" (UID: \"e9e8e847-5b80-44de-9450-c5ae29a5b57b\") " pod="openshift-insights/insights-runtime-extractor-9txq2" Oct 31 12:32:34.725965 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.723612 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a12ea3cb-3a3e-4fab-a040-4b4d00926c07-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-lvmr6\" (UID: \"a12ea3cb-3a3e-4fab-a040-4b4d00926c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" Oct 31 12:32:34.725965 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.723649 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:34.725965 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.723655 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:34.725965 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.723705 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.725965 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.723770 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-node-exporter-accelerators-collector-config\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:34.725965 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.723776 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.725965 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.723836 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a12ea3cb-3a3e-4fab-a040-4b4d00926c07-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-lvmr6\" (UID: \"a12ea3cb-3a3e-4fab-a040-4b4d00926c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" Oct 31 12:32:34.725965 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.723951 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.725965 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.724030 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/856f4c46-f3c3-4afe-823e-1b77842932a4-metrics-certs\") pod \"network-metrics-daemon-8bvmr\" (UID: \"856f4c46-f3c3-4afe-823e-1b77842932a4\") " pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:32:34.725965 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.724104 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f84bcfff-6360-4722-965f-8b3f152f2bdf-tmp\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:34.725965 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.724191 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:34.725965 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:34.724232 6841 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 31 12:32:34.725965 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.724242 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/db5feef6-b3fc-4263-bfc1-66b070df97b4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.725965 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:34.724273 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-serving-cert podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:32:35.224262851 +0000 UTC m=+1.707385729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-serving-cert") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : object "openshift-console"/"console-serving-cert" not registered Oct 31 12:32:34.725965 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.724295 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9fcef325-e046-4c6d-8599-6c67115f382c-serviceca\") pod \"node-ca-chf5k\" (UID: \"9fcef325-e046-4c6d-8599-6c67115f382c\") " pod="openshift-image-registry/node-ca-chf5k" Oct 31 12:32:34.725965 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.724333 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.725965 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.724421 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9e8e847-5b80-44de-9450-c5ae29a5b57b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9txq2\" (UID: \"e9e8e847-5b80-44de-9450-c5ae29a5b57b\") " pod="openshift-insights/insights-runtime-extractor-9txq2" Oct 31 12:32:34.725965 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.724440 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.725965 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.724526 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.725965 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.724530 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-web-config\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.725965 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.724844 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.727636 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.727619 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-865nz\" (UniqueName: \"kubernetes.io/projected/178dd4e3-f913-4638-b4c7-8c9e96c82053-kube-api-access-865nz\") pod \"ingress-canary-tm5c4\" (UID: \"178dd4e3-f913-4638-b4c7-8c9e96c82053\") " pod="openshift-ingress-canary/ingress-canary-tm5c4" Oct 31 12:32:34.727699 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.727683 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj9cr\" (UniqueName: \"kubernetes.io/projected/a12ea3cb-3a3e-4fab-a040-4b4d00926c07-kube-api-access-fj9cr\") pod \"kube-state-metrics-5dc8d7d89b-lvmr6\" (UID: \"a12ea3cb-3a3e-4fab-a040-4b4d00926c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lvmr6" Oct 31 12:32:34.735800 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.735779 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4hhg\" (UniqueName: \"kubernetes.io/projected/856f4c46-f3c3-4afe-823e-1b77842932a4-kube-api-access-n4hhg\") pod \"network-metrics-daemon-8bvmr\" (UID: \"856f4c46-f3c3-4afe-823e-1b77842932a4\") " pod="openshift-multus/network-metrics-daemon-8bvmr" Oct 31 12:32:34.735880 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.735807 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h58w\" (UniqueName: \"kubernetes.io/projected/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-kube-api-access-5h58w\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:34.735931 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.735886 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhn4r\" (UniqueName: \"kubernetes.io/projected/4f51d774-d999-4957-a165-0346972d65bc-kube-api-access-fhn4r\") pod \"node-resolver-7p252\" (UID: \"4f51d774-d999-4957-a165-0346972d65bc\") " pod="openshift-dns/node-resolver-7p252" Oct 31 12:32:34.735931 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.735913 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgk87\" (UniqueName: \"kubernetes.io/projected/2b6cb291-3443-496d-8598-3b253102832f-kube-api-access-xgk87\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.736175 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.736160 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw546\" (UniqueName: \"kubernetes.io/projected/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-kube-api-access-cw546\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.740591 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.740572 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5sww\" (UniqueName: \"kubernetes.io/projected/5dd720ed-3a46-454c-80c2-4850993e6b76-kube-api-access-t5sww\") pod \"dns-default-4kn7k\" (UID: \"5dd720ed-3a46-454c-80c2-4850993e6b76\") " pod="openshift-dns/dns-default-4kn7k" Oct 31 12:32:34.740830 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.740795 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8wjg\" (UniqueName: \"kubernetes.io/projected/d89912fc-3948-4569-bfea-87c8dceedd36-kube-api-access-g8wjg\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:32:34.741138 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.741119 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c69t\" (UniqueName: \"kubernetes.io/projected/2da30edd-1262-4438-b719-da8055c3a763-kube-api-access-9c69t\") pod \"openshift-state-metrics-7bd7489957-v2x29\" (UID: \"2da30edd-1262-4438-b719-da8055c3a763\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2x29" Oct 31 12:32:34.741192 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.741147 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggkt5\" (UniqueName: \"kubernetes.io/projected/8087a5a6-949e-491f-98b9-3360b93a6419-kube-api-access-ggkt5\") pod \"iptables-alerter-27p2l\" (UID: \"8087a5a6-949e-491f-98b9-3360b93a6419\") " pod="openshift-network-operator/iptables-alerter-27p2l" Oct 31 12:32:34.741871 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.741855 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnpbz\" (UniqueName: \"kubernetes.io/projected/e9e8e847-5b80-44de-9450-c5ae29a5b57b-kube-api-access-bnpbz\") pod \"insights-runtime-extractor-9txq2\" (UID: \"e9e8e847-5b80-44de-9450-c5ae29a5b57b\") " pod="openshift-insights/insights-runtime-extractor-9txq2" Oct 31 12:32:34.742092 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.742077 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jrdh\" (UniqueName: \"kubernetes.io/projected/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-kube-api-access-6jrdh\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.742661 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.742645 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwz6m\" (UniqueName: \"kubernetes.io/projected/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-kube-api-access-fwz6m\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:32:34.743012 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.742978 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wkrd\" (UniqueName: \"kubernetes.io/projected/ea2d5c42-8213-489b-ba92-369c0435c6e5-kube-api-access-8wkrd\") pod \"alertmanager-main-0\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:34.743393 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.743372 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jd8l\" (UniqueName: \"kubernetes.io/projected/9fcef325-e046-4c6d-8599-6c67115f382c-kube-api-access-5jd8l\") pod \"node-ca-chf5k\" (UID: \"9fcef325-e046-4c6d-8599-6c67115f382c\") " pod="openshift-image-registry/node-ca-chf5k" Oct 31 12:32:34.743606 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.743588 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl8lr\" (UniqueName: \"kubernetes.io/projected/08d2ae45-d701-4555-bc6b-670feece190f-kube-api-access-pl8lr\") pod \"thanos-querier-56dc986dcb-tghc2\" (UID: \"08d2ae45-d701-4555-bc6b-670feece190f\") " pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:34.744526 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:34.744512 6841 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 31 12:32:34.744570 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:34.744529 6841 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 31 12:32:34.744570 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:34.744539 6841 projected.go:196] Error preparing data for projected volume kube-api-access-b8gsv for pod openshift-console/console-668c5b5566-4scvp: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 12:32:34.744625 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:34.744586 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f84bcfff-6360-4722-965f-8b3f152f2bdf-kube-api-access-b8gsv podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:32:35.244576052 +0000 UTC m=+1.727698976 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-b8gsv" (UniqueName: "kubernetes.io/projected/f84bcfff-6360-4722-965f-8b3f152f2bdf-kube-api-access-b8gsv") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 12:32:34.748738 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.748703 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r52xd\" (UniqueName: \"kubernetes.io/projected/db5feef6-b3fc-4263-bfc1-66b070df97b4-kube-api-access-r52xd\") pod \"prometheus-k8s-0\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:34.752064 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.752039 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzs6g\" (UniqueName: \"kubernetes.io/projected/6ab66c79-2b0a-4671-9bf8-6aafea4e7454-kube-api-access-qzs6g\") pod \"network-check-target-7sf98\" (UID: \"6ab66c79-2b0a-4671-9bf8-6aafea4e7454\") " pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:32:34.771457 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.771421 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:34.823178 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823137 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-run-ovn-kubernetes\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.823178 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823168 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-multus-cni-dir\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.823381 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823188 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d89912fc-3948-4569-bfea-87c8dceedd36-registration-dir\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:32:34.823381 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823213 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-cnibin\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.823381 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823234 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-etc-kubernetes\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.823381 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823245 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-run-ovn-kubernetes\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.823381 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823259 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-lib-modules\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.823381 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823294 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-etc-kubernetes\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.823381 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823289 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-run-systemd\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.823381 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823308 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d89912fc-3948-4569-bfea-87c8dceedd36-registration-dir\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:32:34.823381 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823327 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-host-run-k8s-cni-cncf-io\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.823381 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823306 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-cnibin\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.823381 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823358 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-node-log\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.823381 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823322 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-multus-cni-dir\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.823381 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823377 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-host-var-lib-kubelet\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.823381 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823377 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-host-run-k8s-cni-cncf-io\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.823381 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823365 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-run-systemd\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.824149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823417 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-etc-sysctl-d\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.824149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823420 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-lib-modules\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.824149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823428 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-host-var-lib-kubelet\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.824149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823481 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-node-log\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.824149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823531 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-kubelet-config\") pod \"global-pull-secret-syncer-blcxw\" (UID: \"ed7f6a91-3f6e-409f-bc57-b5d0f8e67770\") " pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:32:34.824149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823550 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-etc-sysctl-d\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.824149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823579 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-kubelet\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.824149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823651 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-kubelet-config\") pod \"global-pull-secret-syncer-blcxw\" (UID: \"ed7f6a91-3f6e-409f-bc57-b5d0f8e67770\") " pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:32:34.824149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823662 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-kubelet\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.824149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823686 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-slash\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.824149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823717 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-slash\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.824149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823717 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-run-netns\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.824149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823742 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-run-netns\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.824149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823755 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-host-var-lib-cni-bin\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.824149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823774 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d89912fc-3948-4569-bfea-87c8dceedd36-socket-dir\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:32:34.824149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823799 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d89912fc-3948-4569-bfea-87c8dceedd36-etc-selinux\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:32:34.824149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823803 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-host-var-lib-cni-bin\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.824149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823823 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9fcef325-e046-4c6d-8599-6c67115f382c-host\") pod \"node-ca-chf5k\" (UID: \"9fcef325-e046-4c6d-8599-6c67115f382c\") " pod="openshift-image-registry/node-ca-chf5k" Oct 31 12:32:34.824149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823847 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-etc-openvswitch\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.824149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823861 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d89912fc-3948-4569-bfea-87c8dceedd36-socket-dir\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:32:34.824149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823876 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-etc-kubernetes\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.824149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823901 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-etc-openvswitch\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.824149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823903 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e9e8e847-5b80-44de-9450-c5ae29a5b57b-crio-socket\") pod \"insights-runtime-extractor-9txq2\" (UID: \"e9e8e847-5b80-44de-9450-c5ae29a5b57b\") " pod="openshift-insights/insights-runtime-extractor-9txq2" Oct 31 12:32:34.824149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823904 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d89912fc-3948-4569-bfea-87c8dceedd36-etc-selinux\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:32:34.824149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823925 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9fcef325-e046-4c6d-8599-6c67115f382c-host\") pod \"node-ca-chf5k\" (UID: \"9fcef325-e046-4c6d-8599-6c67115f382c\") " pod="openshift-image-registry/node-ca-chf5k" Oct 31 12:32:34.824149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823945 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e9e8e847-5b80-44de-9450-c5ae29a5b57b-crio-socket\") pod \"insights-runtime-extractor-9txq2\" (UID: \"e9e8e847-5b80-44de-9450-c5ae29a5b57b\") " pod="openshift-insights/insights-runtime-extractor-9txq2" Oct 31 12:32:34.824149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.823950 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-cni-bin\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.824149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824002 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-cni-bin\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.824149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824013 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-etc-kubernetes\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.824149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824076 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-systemd-units\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.824149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824111 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-sys\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:34.824149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824126 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-host-run-netns\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.824149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824141 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-systemd-units\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.824149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824146 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d89912fc-3948-4569-bfea-87c8dceedd36-sys-fs\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824198 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-sys\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824203 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-etc-sysctl-conf\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824223 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-host-run-netns\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824232 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d89912fc-3948-4569-bfea-87c8dceedd36-sys-fs\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824254 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824301 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d89912fc-3948-4569-bfea-87c8dceedd36-device-dir\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824335 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-etc-sysctl-conf\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824339 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-run-openvswitch\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824360 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824376 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d89912fc-3948-4569-bfea-87c8dceedd36-device-dir\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824375 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-run-ovn\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824407 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-run-openvswitch\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824409 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-run-ovn\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824450 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-dbus\") pod \"global-pull-secret-syncer-blcxw\" (UID: \"ed7f6a91-3f6e-409f-bc57-b5d0f8e67770\") " pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824469 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-cni-netd\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824483 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-etc-sysconfig\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824500 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-host\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824521 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-cni-netd\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824543 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-multus-socket-dir-parent\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824533 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed7f6a91-3f6e-409f-bc57-b5d0f8e67770-dbus\") pod \"global-pull-secret-syncer-blcxw\" (UID: \"ed7f6a91-3f6e-409f-bc57-b5d0f8e67770\") " pod="kube-system/global-pull-secret-syncer-blcxw" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824570 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-host-var-lib-cni-multus\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824559 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-host\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824568 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-etc-sysconfig\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824603 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-multus-socket-dir-parent\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824610 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-sys\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824612 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-host-var-lib-cni-multus\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824638 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-log-socket\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824664 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-system-cni-dir\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824683 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-log-socket\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824691 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-cnibin\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824686 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-sys\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824710 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-system-cni-dir\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824741 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-cnibin\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824779 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-multus-conf-dir\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824794 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d89912fc-3948-4569-bfea-87c8dceedd36-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824819 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-multus-conf-dir\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824823 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-var-lib-kubelet\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824875 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824889 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d89912fc-3948-4569-bfea-87c8dceedd36-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vpkg4\" (UID: \"d89912fc-3948-4569-bfea-87c8dceedd36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vpkg4" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824903 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-run\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824934 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-root\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824940 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-run\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824950 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-var-lib-kubelet\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.824954 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.825014 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-etc-systemd\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.825019 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-root\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.825043 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-system-cni-dir\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.825073 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-var-lib-openvswitch\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.825095 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-host-run-multus-certs\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.825099 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-etc-systemd\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.825131 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f23a1c72-8427-4b6f-ad2b-cc76a88709c9-system-cni-dir\") pod \"multus-additional-cni-plugins-plzf9\" (UID: \"f23a1c72-8427-4b6f-ad2b-cc76a88709c9\") " pod="openshift-multus/multus-additional-cni-plugins-plzf9" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.825146 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca9b156d-7499-4b53-b17f-01dc63ae0c5e-var-lib-openvswitch\") pod \"ovnkube-node-h9pd6\" (UID: \"ca9b156d-7499-4b53-b17f-01dc63ae0c5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.825161 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4f51d774-d999-4957-a165-0346972d65bc-hosts-file\") pod \"node-resolver-7p252\" (UID: \"4f51d774-d999-4957-a165-0346972d65bc\") " pod="openshift-dns/node-resolver-7p252" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.825186 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-host-run-multus-certs\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.825185 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8087a5a6-949e-491f-98b9-3360b93a6419-host-slash\") pod \"iptables-alerter-27p2l\" (UID: \"8087a5a6-949e-491f-98b9-3360b93a6419\") " pod="openshift-network-operator/iptables-alerter-27p2l" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.825216 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-etc-modprobe-d\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.825216 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8087a5a6-949e-491f-98b9-3360b93a6419-host-slash\") pod \"iptables-alerter-27p2l\" (UID: \"8087a5a6-949e-491f-98b9-3360b93a6419\") " pod="openshift-network-operator/iptables-alerter-27p2l" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.825238 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-node-exporter-wtmp\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.825267 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4f51d774-d999-4957-a165-0346972d65bc-hosts-file\") pod \"node-resolver-7p252\" (UID: \"4f51d774-d999-4957-a165-0346972d65bc\") " pod="openshift-dns/node-resolver-7p252" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.825270 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-hostroot\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.825286 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2b7d2569-9fcf-4e91-9e2f-42beb94593a8-hostroot\") pod \"multus-rdcdb\" (UID: \"2b7d2569-9fcf-4e91-9e2f-42beb94593a8\") " pod="openshift-multus/multus-rdcdb" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.825297 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2b6cb291-3443-496d-8598-3b253102832f-etc-modprobe-d\") pod \"tuned-g98lz\" (UID: \"2b6cb291-3443-496d-8598-3b253102832f\") " pod="openshift-cluster-node-tuning-operator/tuned-g98lz" Oct 31 12:32:34.825731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:34.825342 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8ef4a37a-e3ae-4c25-abd8-8132d592be9e-node-exporter-wtmp\") pod \"node-exporter-26lht\" (UID: \"8ef4a37a-e3ae-4c25-abd8-8132d592be9e\") " pod="openshift-monitoring/node-exporter-26lht" Oct 31 12:32:35.003703 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:35.003670 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:35.122365 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:35.122333 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:35.146637 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:35.146612 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:35.228221 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:35.228189 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-service-ca\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:35.228386 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:35.228235 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-serving-cert\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:35.228386 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:35.228266 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-oauth-serving-cert\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:35.228386 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:35.228295 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-oauth-config\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:35.228386 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:35.228305 6841 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 31 12:32:35.228386 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:35.228372 6841 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 31 12:32:35.228647 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:35.228313 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-config\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:35.228647 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:35.228373 6841 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 31 12:32:35.228647 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:35.228446 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-config podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:32:36.228429307 +0000 UTC m=+2.711552204 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-config") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : object "openshift-console"/"console-config" not registered Oct 31 12:32:35.228647 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:35.228471 6841 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 31 12:32:35.228647 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:35.228444 6841 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 31 12:32:35.228647 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:35.228523 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-service-ca podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:32:36.228507322 +0000 UTC m=+2.711630214 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-service-ca") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : object "openshift-console"/"service-ca" not registered Oct 31 12:32:35.228647 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:35.228547 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-oauth-serving-cert podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:32:36.228538055 +0000 UTC m=+2.711660941 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-oauth-serving-cert") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : object "openshift-console"/"oauth-serving-cert" not registered Oct 31 12:32:35.228647 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:35.228564 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-oauth-config podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:32:36.228556292 +0000 UTC m=+2.711679176 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-oauth-config") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : object "openshift-console"/"console-oauth-config" not registered Oct 31 12:32:35.228647 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:35.228593 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-serving-cert podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:32:36.228572129 +0000 UTC m=+2.711695026 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-serving-cert") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : object "openshift-console"/"console-serving-cert" not registered Oct 31 12:32:35.229156 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:35.228692 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-trusted-ca-bundle\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:35.229156 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:35.228773 6841 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 12:32:35.229156 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:35.228811 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-trusted-ca-bundle podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:32:36.228797773 +0000 UTC m=+2.711920674 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-trusted-ca-bundle") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 12:32:35.291197 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:35.291111 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:35.329297 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:35.329270 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8gsv\" (UniqueName: \"kubernetes.io/projected/f84bcfff-6360-4722-965f-8b3f152f2bdf-kube-api-access-b8gsv\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:35.329454 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:35.329436 6841 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 31 12:32:35.329513 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:35.329461 6841 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 31 12:32:35.329513 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:35.329472 6841 projected.go:196] Error preparing data for projected volume kube-api-access-b8gsv for pod openshift-console/console-668c5b5566-4scvp: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 12:32:35.329605 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:35.329537 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f84bcfff-6360-4722-965f-8b3f152f2bdf-kube-api-access-b8gsv podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:32:36.329518499 +0000 UTC m=+2.812641390 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-b8gsv" (UniqueName: "kubernetes.io/projected/f84bcfff-6360-4722-965f-8b3f152f2bdf-kube-api-access-b8gsv") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 12:32:35.489831 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:35.489794 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:35.581155 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:35.581062 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4kn7k" Oct 31 12:32:35.991813 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:35.991786 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:32:36.238150 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:36.238102 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-trusted-ca-bundle\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:36.238150 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:36.238161 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-service-ca\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:36.238370 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:36.238192 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-serving-cert\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:36.238370 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:36.238213 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-oauth-serving-cert\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:36.238370 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:36.238231 6841 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 12:32:36.238370 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:36.238241 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-oauth-config\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:36.238370 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:36.238315 6841 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 31 12:32:36.238370 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:36.238320 6841 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 31 12:32:36.238370 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:36.238334 6841 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 31 12:32:36.238370 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:36.238325 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-trusted-ca-bundle podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:32:38.238305523 +0000 UTC m=+4.721428408 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-trusted-ca-bundle") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 12:32:36.238370 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:36.238334 6841 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 31 12:32:36.238673 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:36.238415 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-service-ca podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:32:38.238378723 +0000 UTC m=+4.721501602 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-service-ca") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : object "openshift-console"/"service-ca" not registered Oct 31 12:32:36.238673 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:36.238437 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-serving-cert podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:32:38.238427096 +0000 UTC m=+4.721549979 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-serving-cert") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : object "openshift-console"/"console-serving-cert" not registered Oct 31 12:32:36.238673 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:36.238456 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-oauth-serving-cert podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:32:38.238447451 +0000 UTC m=+4.721570331 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-oauth-serving-cert") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : object "openshift-console"/"oauth-serving-cert" not registered Oct 31 12:32:36.238673 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:36.238486 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-config\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:36.238673 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:36.238519 6841 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 31 12:32:36.238673 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:36.238562 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-config podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:32:38.238551072 +0000 UTC m=+4.721673960 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-config") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : object "openshift-console"/"console-config" not registered Oct 31 12:32:36.238673 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:36.238579 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-oauth-config podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:32:38.238570029 +0000 UTC m=+4.721692908 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-oauth-config") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : object "openshift-console"/"console-oauth-config" not registered Oct 31 12:32:36.339575 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:36.339491 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8gsv\" (UniqueName: \"kubernetes.io/projected/f84bcfff-6360-4722-965f-8b3f152f2bdf-kube-api-access-b8gsv\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:36.339710 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:36.339637 6841 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 31 12:32:36.339710 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:36.339656 6841 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 31 12:32:36.339710 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:36.339666 6841 projected.go:196] Error preparing data for projected volume kube-api-access-b8gsv for pod openshift-console/console-668c5b5566-4scvp: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 12:32:36.339817 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:36.339714 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f84bcfff-6360-4722-965f-8b3f152f2bdf-kube-api-access-b8gsv podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:32:38.339700678 +0000 UTC m=+4.822823560 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-b8gsv" (UniqueName: "kubernetes.io/projected/f84bcfff-6360-4722-965f-8b3f152f2bdf-kube-api-access-b8gsv") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 12:32:36.700367 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:36.700337 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-56dc986dcb-tghc2" Oct 31 12:32:37.724100 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:37.724070 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:38.258823 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:38.258788 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-serving-cert\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:38.259075 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:38.258833 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-oauth-serving-cert\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:38.259075 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:38.258876 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-oauth-config\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:38.259075 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:38.258898 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-config\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:38.259075 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:38.258953 6841 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 31 12:32:38.259075 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:38.258977 6841 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 31 12:32:38.259075 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:38.258996 6841 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 31 12:32:38.259075 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:38.259041 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-oauth-config podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:32:42.259023552 +0000 UTC m=+8.742146432 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-oauth-config") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : object "openshift-console"/"console-oauth-config" not registered Oct 31 12:32:38.259075 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:38.259062 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-serving-cert podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:32:42.259053249 +0000 UTC m=+8.742176128 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-serving-cert") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : object "openshift-console"/"console-serving-cert" not registered Oct 31 12:32:38.259075 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:38.259070 6841 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 31 12:32:38.259355 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:38.259078 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-oauth-serving-cert podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:32:42.259070228 +0000 UTC m=+8.742193132 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-oauth-serving-cert") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : object "openshift-console"/"oauth-serving-cert" not registered Oct 31 12:32:38.259355 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:38.259138 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-config podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:32:42.259126246 +0000 UTC m=+8.742249128 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-config") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : object "openshift-console"/"console-config" not registered Oct 31 12:32:38.259355 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:38.259184 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-trusted-ca-bundle\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:38.259355 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:38.259215 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-service-ca\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:38.259355 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:38.259267 6841 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 31 12:32:38.259355 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:38.259285 6841 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 12:32:38.259355 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:38.259299 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-service-ca podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:32:42.259289244 +0000 UTC m=+8.742412134 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-service-ca") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : object "openshift-console"/"service-ca" not registered Oct 31 12:32:38.259355 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:38.259337 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-trusted-ca-bundle podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:32:42.259324865 +0000 UTC m=+8.742447745 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-trusted-ca-bundle") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 12:32:38.360594 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:38.360564 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8gsv\" (UniqueName: \"kubernetes.io/projected/f84bcfff-6360-4722-965f-8b3f152f2bdf-kube-api-access-b8gsv\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:38.360743 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:38.360710 6841 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 31 12:32:38.360743 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:38.360726 6841 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 31 12:32:38.360743 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:38.360737 6841 projected.go:196] Error preparing data for projected volume kube-api-access-b8gsv for pod openshift-console/console-668c5b5566-4scvp: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 12:32:38.360871 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:38.360789 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f84bcfff-6360-4722-965f-8b3f152f2bdf-kube-api-access-b8gsv podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:32:42.360774479 +0000 UTC m=+8.843897373 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-b8gsv" (UniqueName: "kubernetes.io/projected/f84bcfff-6360-4722-965f-8b3f152f2bdf-kube-api-access-b8gsv") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 12:32:40.994569 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:40.994538 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:40.994943 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:40.994741 6841 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 12:32:41.561929 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:41.561881 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:41.760239 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:41.760199 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:42.300095 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:42.300049 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-trusted-ca-bundle\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:42.300095 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:42.300105 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-service-ca\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:42.300633 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:42.300137 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-serving-cert\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:42.300633 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:42.300158 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-oauth-serving-cert\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:42.300633 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:42.300176 6841 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 12:32:42.300633 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:42.300245 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-trusted-ca-bundle podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:32:50.300229834 +0000 UTC m=+16.783352712 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-trusted-ca-bundle") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 12:32:42.300633 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:42.300253 6841 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 31 12:32:42.300633 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:42.300255 6841 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 31 12:32:42.300633 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:42.300184 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-oauth-config\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:42.300633 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:42.300297 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-oauth-config podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:32:50.300283484 +0000 UTC m=+16.783406377 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-oauth-config") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : object "openshift-console"/"console-oauth-config" not registered Oct 31 12:32:42.300633 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:42.300260 6841 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 31 12:32:42.300633 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:42.300327 6841 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 31 12:32:42.300633 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:42.300343 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-oauth-serving-cert podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:32:50.300326585 +0000 UTC m=+16.783449463 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-oauth-serving-cert") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : object "openshift-console"/"oauth-serving-cert" not registered Oct 31 12:32:42.300633 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:42.300366 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-config\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:42.300633 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:42.300393 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-serving-cert podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:32:50.300377531 +0000 UTC m=+16.783500412 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-serving-cert") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : object "openshift-console"/"console-serving-cert" not registered Oct 31 12:32:42.300633 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:42.300401 6841 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 31 12:32:42.300633 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:42.300427 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-service-ca podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:32:50.30041568 +0000 UTC m=+16.783538560 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-service-ca") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : object "openshift-console"/"service-ca" not registered Oct 31 12:32:42.300633 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:42.300446 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-config podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:32:50.300438103 +0000 UTC m=+16.783561021 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-config") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : object "openshift-console"/"console-config" not registered Oct 31 12:32:42.401379 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:42.401329 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8gsv\" (UniqueName: \"kubernetes.io/projected/f84bcfff-6360-4722-965f-8b3f152f2bdf-kube-api-access-b8gsv\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:42.401594 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:42.401536 6841 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 31 12:32:42.401594 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:42.401564 6841 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 31 12:32:42.401594 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:42.401577 6841 projected.go:196] Error preparing data for projected volume kube-api-access-b8gsv for pod openshift-console/console-668c5b5566-4scvp: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 12:32:42.401744 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:42.401645 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f84bcfff-6360-4722-965f-8b3f152f2bdf-kube-api-access-b8gsv podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:32:50.401627127 +0000 UTC m=+16.884750016 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-b8gsv" (UniqueName: "kubernetes.io/projected/f84bcfff-6360-4722-965f-8b3f152f2bdf-kube-api-access-b8gsv") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 12:32:43.298178 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:43.298151 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h9pd6" Oct 31 12:32:43.931103 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:43.931060 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:44.125468 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:44.125433 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:44.862905 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:44.862867 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 12:32:44.863338 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:44.863312 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ea2d5c42-8213-489b-ba92-369c0435c6e5" containerName="alertmanager" containerID="cri-o://2eb17c149b4df36e04b064b83e92f0b1039815a5e4beae36f18b21f05945cadc" gracePeriod=120 Oct 31 12:32:44.863398 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:44.863369 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ea2d5c42-8213-489b-ba92-369c0435c6e5" containerName="kube-rbac-proxy-metric" containerID="cri-o://f1713d3fec45df734eb6e5d1509fbd0a322e6c4ea54725424d8b66202b0b5663" gracePeriod=120 Oct 31 12:32:44.863449 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:44.863398 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ea2d5c42-8213-489b-ba92-369c0435c6e5" containerName="config-reloader" containerID="cri-o://05fac39194a8d616b3f0aa4814b10e65876efe3f857383aa52021d53f57d42dd" gracePeriod=120 Oct 31 12:32:44.863449 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:44.863406 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ea2d5c42-8213-489b-ba92-369c0435c6e5" containerName="kube-rbac-proxy" containerID="cri-o://0951cedc3fd3ba47485810d62d387718fdb8a63f28c8f9f37bc7009e60631e37" gracePeriod=120 Oct 31 12:32:44.863549 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:44.863447 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ea2d5c42-8213-489b-ba92-369c0435c6e5" containerName="prom-label-proxy" containerID="cri-o://fd75328d99325ec80255557e92d33d54426512a43db674e68ed9b2cee254ffc0" gracePeriod=120 Oct 31 12:32:44.863549 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:44.863377 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ea2d5c42-8213-489b-ba92-369c0435c6e5" containerName="kube-rbac-proxy-web" containerID="cri-o://61980da9f603f489ea868d5fb29135ab2e58edd1cfa6653b1b7ceb3866063430" gracePeriod=120 Oct 31 12:32:45.745339 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:45.745296 6841 generic.go:356] "Generic (PLEG): container finished" podID="ea2d5c42-8213-489b-ba92-369c0435c6e5" containerID="fd75328d99325ec80255557e92d33d54426512a43db674e68ed9b2cee254ffc0" exitCode=0 Oct 31 12:32:45.745339 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:45.745328 6841 generic.go:356] "Generic (PLEG): container finished" podID="ea2d5c42-8213-489b-ba92-369c0435c6e5" containerID="0951cedc3fd3ba47485810d62d387718fdb8a63f28c8f9f37bc7009e60631e37" exitCode=0 Oct 31 12:32:45.745339 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:45.745335 6841 generic.go:356] "Generic (PLEG): container finished" podID="ea2d5c42-8213-489b-ba92-369c0435c6e5" containerID="61980da9f603f489ea868d5fb29135ab2e58edd1cfa6653b1b7ceb3866063430" exitCode=0 Oct 31 12:32:45.745339 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:45.745343 6841 generic.go:356] "Generic (PLEG): container finished" podID="ea2d5c42-8213-489b-ba92-369c0435c6e5" containerID="05fac39194a8d616b3f0aa4814b10e65876efe3f857383aa52021d53f57d42dd" exitCode=0 Oct 31 12:32:45.745339 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:45.745348 6841 generic.go:356] "Generic (PLEG): container finished" podID="ea2d5c42-8213-489b-ba92-369c0435c6e5" containerID="2eb17c149b4df36e04b064b83e92f0b1039815a5e4beae36f18b21f05945cadc" exitCode=0 Oct 31 12:32:45.745833 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:45.745370 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea2d5c42-8213-489b-ba92-369c0435c6e5","Type":"ContainerDied","Data":"fd75328d99325ec80255557e92d33d54426512a43db674e68ed9b2cee254ffc0"} Oct 31 12:32:45.745833 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:45.745434 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea2d5c42-8213-489b-ba92-369c0435c6e5","Type":"ContainerDied","Data":"0951cedc3fd3ba47485810d62d387718fdb8a63f28c8f9f37bc7009e60631e37"} Oct 31 12:32:45.745833 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:45.745459 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea2d5c42-8213-489b-ba92-369c0435c6e5","Type":"ContainerDied","Data":"61980da9f603f489ea868d5fb29135ab2e58edd1cfa6653b1b7ceb3866063430"} Oct 31 12:32:45.745833 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:45.745484 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea2d5c42-8213-489b-ba92-369c0435c6e5","Type":"ContainerDied","Data":"05fac39194a8d616b3f0aa4814b10e65876efe3f857383aa52021d53f57d42dd"} Oct 31 12:32:45.745833 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:45.745508 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea2d5c42-8213-489b-ba92-369c0435c6e5","Type":"ContainerDied","Data":"2eb17c149b4df36e04b064b83e92f0b1039815a5e4beae36f18b21f05945cadc"} Oct 31 12:32:46.103674 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.103649 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:46.239038 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.239006 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea2d5c42-8213-489b-ba92-369c0435c6e5-config-out\" (UniqueName: \"kubernetes.io/empty-dir/ea2d5c42-8213-489b-ba92-369c0435c6e5-config-out\") pod \"ea2d5c42-8213-489b-ba92-369c0435c6e5\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " Oct 31 12:32:46.239204 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.239096 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea2d5c42-8213-489b-ba92-369c0435c6e5-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea2d5c42-8213-489b-ba92-369c0435c6e5-alertmanager-trusted-ca-bundle\") pod \"ea2d5c42-8213-489b-ba92-369c0435c6e5\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " Oct 31 12:32:46.239204 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.239118 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-kube-rbac-proxy\") pod \"ea2d5c42-8213-489b-ba92-369c0435c6e5\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " Oct 31 12:32:46.239204 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.239141 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea2d5c42-8213-489b-ba92-369c0435c6e5-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea2d5c42-8213-489b-ba92-369c0435c6e5-metrics-client-ca\") pod \"ea2d5c42-8213-489b-ba92-369c0435c6e5\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " Oct 31 12:32:46.239204 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.239167 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"ea2d5c42-8213-489b-ba92-369c0435c6e5\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " Oct 31 12:32:46.239390 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.239329 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-main-tls\") pod \"ea2d5c42-8213-489b-ba92-369c0435c6e5\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " Oct 31 12:32:46.239444 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.239415 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea2d5c42-8213-489b-ba92-369c0435c6e5-kube-api-access-8wkrd\" (UniqueName: \"kubernetes.io/projected/ea2d5c42-8213-489b-ba92-369c0435c6e5-kube-api-access-8wkrd\") pod \"ea2d5c42-8213-489b-ba92-369c0435c6e5\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " Oct 31 12:32:46.239497 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.239453 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-cluster-tls-config\") pod \"ea2d5c42-8213-489b-ba92-369c0435c6e5\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " Oct 31 12:32:46.239556 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.239495 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-kube-rbac-proxy-web\") pod \"ea2d5c42-8213-489b-ba92-369c0435c6e5\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " Oct 31 12:32:46.239556 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.239531 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-config-volume\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-config-volume\") pod \"ea2d5c42-8213-489b-ba92-369c0435c6e5\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " Oct 31 12:32:46.239657 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.239563 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea2d5c42-8213-489b-ba92-369c0435c6e5-tls-assets\" (UniqueName: \"kubernetes.io/projected/ea2d5c42-8213-489b-ba92-369c0435c6e5-tls-assets\") pod \"ea2d5c42-8213-489b-ba92-369c0435c6e5\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " Oct 31 12:32:46.239657 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.239573 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea2d5c42-8213-489b-ba92-369c0435c6e5-metrics-client-ca" pod "ea2d5c42-8213-489b-ba92-369c0435c6e5" (UID: "ea2d5c42-8213-489b-ba92-369c0435c6e5"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:32:46.239657 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.239591 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea2d5c42-8213-489b-ba92-369c0435c6e5-alertmanager-trusted-ca-bundle" pod "ea2d5c42-8213-489b-ba92-369c0435c6e5" (UID: "ea2d5c42-8213-489b-ba92-369c0435c6e5"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:32:46.239657 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.239602 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea2d5c42-8213-489b-ba92-369c0435c6e5-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ea2d5c42-8213-489b-ba92-369c0435c6e5-alertmanager-main-db\") pod \"ea2d5c42-8213-489b-ba92-369c0435c6e5\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " Oct 31 12:32:46.239859 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.239679 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-web-config\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-web-config\") pod \"ea2d5c42-8213-489b-ba92-369c0435c6e5\" (UID: \"ea2d5c42-8213-489b-ba92-369c0435c6e5\") " Oct 31 12:32:46.239916 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.239898 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea2d5c42-8213-489b-ba92-369c0435c6e5-alertmanager-main-db" pod "ea2d5c42-8213-489b-ba92-369c0435c6e5" (UID: "ea2d5c42-8213-489b-ba92-369c0435c6e5"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:32:46.240476 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.240182 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea2d5c42-8213-489b-ba92-369c0435c6e5-metrics-client-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:46.240476 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.240215 6841 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ea2d5c42-8213-489b-ba92-369c0435c6e5-alertmanager-main-db\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:46.240476 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.240235 6841 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea2d5c42-8213-489b-ba92-369c0435c6e5-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:46.242664 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.242494 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-kube-rbac-proxy" pod "ea2d5c42-8213-489b-ba92-369c0435c6e5" (UID: "ea2d5c42-8213-489b-ba92-369c0435c6e5"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:32:46.242900 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.242876 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-config-volume" pod "ea2d5c42-8213-489b-ba92-369c0435c6e5" (UID: "ea2d5c42-8213-489b-ba92-369c0435c6e5"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:32:46.243014 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.242949 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea2d5c42-8213-489b-ba92-369c0435c6e5-config-out" pod "ea2d5c42-8213-489b-ba92-369c0435c6e5" (UID: "ea2d5c42-8213-489b-ba92-369c0435c6e5"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:32:46.243086 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.243020 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-kube-rbac-proxy-metric" pod "ea2d5c42-8213-489b-ba92-369c0435c6e5" (UID: "ea2d5c42-8213-489b-ba92-369c0435c6e5"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:32:46.243269 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.243244 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea2d5c42-8213-489b-ba92-369c0435c6e5-kube-api-access-8wkrd" pod "ea2d5c42-8213-489b-ba92-369c0435c6e5" (UID: "ea2d5c42-8213-489b-ba92-369c0435c6e5"). InnerVolumeSpecName "kube-api-access-8wkrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:32:46.243369 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.243331 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-main-tls" pod "ea2d5c42-8213-489b-ba92-369c0435c6e5" (UID: "ea2d5c42-8213-489b-ba92-369c0435c6e5"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:32:46.243791 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.243761 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea2d5c42-8213-489b-ba92-369c0435c6e5-tls-assets" pod "ea2d5c42-8213-489b-ba92-369c0435c6e5" (UID: "ea2d5c42-8213-489b-ba92-369c0435c6e5"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:32:46.243963 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.243938 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-kube-rbac-proxy-web" pod "ea2d5c42-8213-489b-ba92-369c0435c6e5" (UID: "ea2d5c42-8213-489b-ba92-369c0435c6e5"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:32:46.247100 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.247086 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-cluster-tls-config" pod "ea2d5c42-8213-489b-ba92-369c0435c6e5" (UID: "ea2d5c42-8213-489b-ba92-369c0435c6e5"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:32:46.253163 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.253140 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-web-config" pod "ea2d5c42-8213-489b-ba92-369c0435c6e5" (UID: "ea2d5c42-8213-489b-ba92-369c0435c6e5"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:32:46.341793 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.341709 6841 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:46.341793 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.341739 6841 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:46.341793 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.341756 6841 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-main-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:46.341793 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.341770 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wkrd\" (UniqueName: \"kubernetes.io/projected/ea2d5c42-8213-489b-ba92-369c0435c6e5-kube-api-access-8wkrd\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:46.341793 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.341783 6841 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-cluster-tls-config\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:46.342066 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.341797 6841 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:46.342066 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.341823 6841 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:46.342066 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.341835 6841 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ea2d5c42-8213-489b-ba92-369c0435c6e5-tls-assets\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:46.342066 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.341847 6841 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ea2d5c42-8213-489b-ba92-369c0435c6e5-web-config\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:46.342066 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.341860 6841 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ea2d5c42-8213-489b-ba92-369c0435c6e5-config-out\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:46.540639 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.540608 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2wtkj" Oct 31 12:32:46.752103 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.752065 6841 generic.go:356] "Generic (PLEG): container finished" podID="ea2d5c42-8213-489b-ba92-369c0435c6e5" containerID="f1713d3fec45df734eb6e5d1509fbd0a322e6c4ea54725424d8b66202b0b5663" exitCode=0 Oct 31 12:32:46.752553 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.752158 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea2d5c42-8213-489b-ba92-369c0435c6e5","Type":"ContainerDied","Data":"f1713d3fec45df734eb6e5d1509fbd0a322e6c4ea54725424d8b66202b0b5663"} Oct 31 12:32:46.752553 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.752494 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:46.752669 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.752573 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea2d5c42-8213-489b-ba92-369c0435c6e5","Type":"ContainerDied","Data":"a63c715a7cc2b7b1d0a46462feb927341b0329e4ab774c81fee0450104ad97ac"} Oct 31 12:32:46.752669 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.752636 6841 scope.go:122] "RemoveContainer" containerID="fd75328d99325ec80255557e92d33d54426512a43db674e68ed9b2cee254ffc0" Oct 31 12:32:46.763451 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.763436 6841 scope.go:122] "RemoveContainer" containerID="f1713d3fec45df734eb6e5d1509fbd0a322e6c4ea54725424d8b66202b0b5663" Oct 31 12:32:46.770280 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.770263 6841 scope.go:122] "RemoveContainer" containerID="0951cedc3fd3ba47485810d62d387718fdb8a63f28c8f9f37bc7009e60631e37" Oct 31 12:32:46.776683 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.776662 6841 scope.go:122] "RemoveContainer" containerID="61980da9f603f489ea868d5fb29135ab2e58edd1cfa6653b1b7ceb3866063430" Oct 31 12:32:46.778137 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.778116 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 12:32:46.783182 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.783160 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 12:32:46.785465 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.785450 6841 scope.go:122] "RemoveContainer" containerID="05fac39194a8d616b3f0aa4814b10e65876efe3f857383aa52021d53f57d42dd" Oct 31 12:32:46.792164 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.792149 6841 scope.go:122] "RemoveContainer" containerID="2eb17c149b4df36e04b064b83e92f0b1039815a5e4beae36f18b21f05945cadc" Oct 31 12:32:46.799172 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.799134 6841 scope.go:122] "RemoveContainer" containerID="475e94bf7c5e862bc04c010bf619e58932a88ded1c15f91acf581c4058959760" Oct 31 12:32:46.805845 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.805827 6841 scope.go:122] "RemoveContainer" containerID="fd75328d99325ec80255557e92d33d54426512a43db674e68ed9b2cee254ffc0" Oct 31 12:32:46.806119 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:46.806099 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd75328d99325ec80255557e92d33d54426512a43db674e68ed9b2cee254ffc0\": container with ID starting with fd75328d99325ec80255557e92d33d54426512a43db674e68ed9b2cee254ffc0 not found: ID does not exist" containerID="fd75328d99325ec80255557e92d33d54426512a43db674e68ed9b2cee254ffc0" Oct 31 12:32:46.806233 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.806126 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd75328d99325ec80255557e92d33d54426512a43db674e68ed9b2cee254ffc0"} err="failed to get container status \"fd75328d99325ec80255557e92d33d54426512a43db674e68ed9b2cee254ffc0\": rpc error: code = NotFound desc = could not find container \"fd75328d99325ec80255557e92d33d54426512a43db674e68ed9b2cee254ffc0\": container with ID starting with fd75328d99325ec80255557e92d33d54426512a43db674e68ed9b2cee254ffc0 not found: ID does not exist" Oct 31 12:32:46.806233 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.806169 6841 scope.go:122] "RemoveContainer" containerID="f1713d3fec45df734eb6e5d1509fbd0a322e6c4ea54725424d8b66202b0b5663" Oct 31 12:32:46.806396 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:46.806376 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1713d3fec45df734eb6e5d1509fbd0a322e6c4ea54725424d8b66202b0b5663\": container with ID starting with f1713d3fec45df734eb6e5d1509fbd0a322e6c4ea54725424d8b66202b0b5663 not found: ID does not exist" containerID="f1713d3fec45df734eb6e5d1509fbd0a322e6c4ea54725424d8b66202b0b5663" Oct 31 12:32:46.806456 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.806399 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1713d3fec45df734eb6e5d1509fbd0a322e6c4ea54725424d8b66202b0b5663"} err="failed to get container status \"f1713d3fec45df734eb6e5d1509fbd0a322e6c4ea54725424d8b66202b0b5663\": rpc error: code = NotFound desc = could not find container \"f1713d3fec45df734eb6e5d1509fbd0a322e6c4ea54725424d8b66202b0b5663\": container with ID starting with f1713d3fec45df734eb6e5d1509fbd0a322e6c4ea54725424d8b66202b0b5663 not found: ID does not exist" Oct 31 12:32:46.806456 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.806411 6841 scope.go:122] "RemoveContainer" containerID="0951cedc3fd3ba47485810d62d387718fdb8a63f28c8f9f37bc7009e60631e37" Oct 31 12:32:46.806604 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:46.806587 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0951cedc3fd3ba47485810d62d387718fdb8a63f28c8f9f37bc7009e60631e37\": container with ID starting with 0951cedc3fd3ba47485810d62d387718fdb8a63f28c8f9f37bc7009e60631e37 not found: ID does not exist" containerID="0951cedc3fd3ba47485810d62d387718fdb8a63f28c8f9f37bc7009e60631e37" Oct 31 12:32:46.806670 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.806605 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0951cedc3fd3ba47485810d62d387718fdb8a63f28c8f9f37bc7009e60631e37"} err="failed to get container status \"0951cedc3fd3ba47485810d62d387718fdb8a63f28c8f9f37bc7009e60631e37\": rpc error: code = NotFound desc = could not find container \"0951cedc3fd3ba47485810d62d387718fdb8a63f28c8f9f37bc7009e60631e37\": container with ID starting with 0951cedc3fd3ba47485810d62d387718fdb8a63f28c8f9f37bc7009e60631e37 not found: ID does not exist" Oct 31 12:32:46.806670 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.806622 6841 scope.go:122] "RemoveContainer" containerID="61980da9f603f489ea868d5fb29135ab2e58edd1cfa6653b1b7ceb3866063430" Oct 31 12:32:46.806844 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:46.806826 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61980da9f603f489ea868d5fb29135ab2e58edd1cfa6653b1b7ceb3866063430\": container with ID starting with 61980da9f603f489ea868d5fb29135ab2e58edd1cfa6653b1b7ceb3866063430 not found: ID does not exist" containerID="61980da9f603f489ea868d5fb29135ab2e58edd1cfa6653b1b7ceb3866063430" Oct 31 12:32:46.806882 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.806850 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61980da9f603f489ea868d5fb29135ab2e58edd1cfa6653b1b7ceb3866063430"} err="failed to get container status \"61980da9f603f489ea868d5fb29135ab2e58edd1cfa6653b1b7ceb3866063430\": rpc error: code = NotFound desc = could not find container \"61980da9f603f489ea868d5fb29135ab2e58edd1cfa6653b1b7ceb3866063430\": container with ID starting with 61980da9f603f489ea868d5fb29135ab2e58edd1cfa6653b1b7ceb3866063430 not found: ID does not exist" Oct 31 12:32:46.806882 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.806864 6841 scope.go:122] "RemoveContainer" containerID="05fac39194a8d616b3f0aa4814b10e65876efe3f857383aa52021d53f57d42dd" Oct 31 12:32:46.807082 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:46.807067 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05fac39194a8d616b3f0aa4814b10e65876efe3f857383aa52021d53f57d42dd\": container with ID starting with 05fac39194a8d616b3f0aa4814b10e65876efe3f857383aa52021d53f57d42dd not found: ID does not exist" containerID="05fac39194a8d616b3f0aa4814b10e65876efe3f857383aa52021d53f57d42dd" Oct 31 12:32:46.807130 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.807084 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05fac39194a8d616b3f0aa4814b10e65876efe3f857383aa52021d53f57d42dd"} err="failed to get container status \"05fac39194a8d616b3f0aa4814b10e65876efe3f857383aa52021d53f57d42dd\": rpc error: code = NotFound desc = could not find container \"05fac39194a8d616b3f0aa4814b10e65876efe3f857383aa52021d53f57d42dd\": container with ID starting with 05fac39194a8d616b3f0aa4814b10e65876efe3f857383aa52021d53f57d42dd not found: ID does not exist" Oct 31 12:32:46.807130 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.807094 6841 scope.go:122] "RemoveContainer" containerID="2eb17c149b4df36e04b064b83e92f0b1039815a5e4beae36f18b21f05945cadc" Oct 31 12:32:46.807300 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:46.807286 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eb17c149b4df36e04b064b83e92f0b1039815a5e4beae36f18b21f05945cadc\": container with ID starting with 2eb17c149b4df36e04b064b83e92f0b1039815a5e4beae36f18b21f05945cadc not found: ID does not exist" containerID="2eb17c149b4df36e04b064b83e92f0b1039815a5e4beae36f18b21f05945cadc" Oct 31 12:32:46.807335 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.807303 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eb17c149b4df36e04b064b83e92f0b1039815a5e4beae36f18b21f05945cadc"} err="failed to get container status \"2eb17c149b4df36e04b064b83e92f0b1039815a5e4beae36f18b21f05945cadc\": rpc error: code = NotFound desc = could not find container \"2eb17c149b4df36e04b064b83e92f0b1039815a5e4beae36f18b21f05945cadc\": container with ID starting with 2eb17c149b4df36e04b064b83e92f0b1039815a5e4beae36f18b21f05945cadc not found: ID does not exist" Oct 31 12:32:46.807335 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.807315 6841 scope.go:122] "RemoveContainer" containerID="475e94bf7c5e862bc04c010bf619e58932a88ded1c15f91acf581c4058959760" Oct 31 12:32:46.807555 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:46.807540 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"475e94bf7c5e862bc04c010bf619e58932a88ded1c15f91acf581c4058959760\": container with ID starting with 475e94bf7c5e862bc04c010bf619e58932a88ded1c15f91acf581c4058959760 not found: ID does not exist" containerID="475e94bf7c5e862bc04c010bf619e58932a88ded1c15f91acf581c4058959760" Oct 31 12:32:46.807606 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.807557 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"475e94bf7c5e862bc04c010bf619e58932a88ded1c15f91acf581c4058959760"} err="failed to get container status \"475e94bf7c5e862bc04c010bf619e58932a88ded1c15f91acf581c4058959760\": rpc error: code = NotFound desc = could not find container \"475e94bf7c5e862bc04c010bf619e58932a88ded1c15f91acf581c4058959760\": container with ID starting with 475e94bf7c5e862bc04c010bf619e58932a88ded1c15f91acf581c4058959760 not found: ID does not exist" Oct 31 12:32:46.812289 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.812268 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 12:32:46.812561 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.812548 6841 cpu_manager.go:414] "containerMap: removing container" podUID="ea2d5c42-8213-489b-ba92-369c0435c6e5" containerName="kube-rbac-proxy" Oct 31 12:32:46.812612 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.812562 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea2d5c42-8213-489b-ba92-369c0435c6e5" containerName="kube-rbac-proxy" Oct 31 12:32:46.812612 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.812573 6841 cpu_manager.go:414] "containerMap: removing container" podUID="ea2d5c42-8213-489b-ba92-369c0435c6e5" containerName="alertmanager" Oct 31 12:32:46.812612 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.812578 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea2d5c42-8213-489b-ba92-369c0435c6e5" containerName="alertmanager" Oct 31 12:32:46.812612 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.812588 6841 cpu_manager.go:414] "containerMap: removing container" podUID="ea2d5c42-8213-489b-ba92-369c0435c6e5" containerName="init-config-reloader" Oct 31 12:32:46.812612 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.812593 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea2d5c42-8213-489b-ba92-369c0435c6e5" containerName="init-config-reloader" Oct 31 12:32:46.812612 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.812604 6841 cpu_manager.go:414] "containerMap: removing container" podUID="ea2d5c42-8213-489b-ba92-369c0435c6e5" containerName="prom-label-proxy" Oct 31 12:32:46.812612 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.812611 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea2d5c42-8213-489b-ba92-369c0435c6e5" containerName="prom-label-proxy" Oct 31 12:32:46.812811 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.812623 6841 cpu_manager.go:414] "containerMap: removing container" podUID="ea2d5c42-8213-489b-ba92-369c0435c6e5" containerName="kube-rbac-proxy-metric" Oct 31 12:32:46.812811 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.812628 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea2d5c42-8213-489b-ba92-369c0435c6e5" containerName="kube-rbac-proxy-metric" Oct 31 12:32:46.812811 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.812637 6841 cpu_manager.go:414] "containerMap: removing container" podUID="ea2d5c42-8213-489b-ba92-369c0435c6e5" containerName="config-reloader" Oct 31 12:32:46.812811 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.812642 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea2d5c42-8213-489b-ba92-369c0435c6e5" containerName="config-reloader" Oct 31 12:32:46.812811 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.812647 6841 cpu_manager.go:414] "containerMap: removing container" podUID="ea2d5c42-8213-489b-ba92-369c0435c6e5" containerName="kube-rbac-proxy-web" Oct 31 12:32:46.812811 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.812651 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea2d5c42-8213-489b-ba92-369c0435c6e5" containerName="kube-rbac-proxy-web" Oct 31 12:32:46.812811 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.812706 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea2d5c42-8213-489b-ba92-369c0435c6e5" containerName="kube-rbac-proxy" Oct 31 12:32:46.812811 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.812714 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea2d5c42-8213-489b-ba92-369c0435c6e5" containerName="alertmanager" Oct 31 12:32:46.812811 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.812721 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea2d5c42-8213-489b-ba92-369c0435c6e5" containerName="init-config-reloader" Oct 31 12:32:46.812811 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.812728 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea2d5c42-8213-489b-ba92-369c0435c6e5" containerName="prom-label-proxy" Oct 31 12:32:46.812811 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.812734 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea2d5c42-8213-489b-ba92-369c0435c6e5" containerName="kube-rbac-proxy-metric" Oct 31 12:32:46.812811 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.812739 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea2d5c42-8213-489b-ba92-369c0435c6e5" containerName="config-reloader" Oct 31 12:32:46.812811 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.812744 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea2d5c42-8213-489b-ba92-369c0435c6e5" containerName="kube-rbac-proxy-web" Oct 31 12:32:46.817225 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.817207 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:46.822901 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.822874 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 12:32:46.823036 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.822899 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 12:32:46.823036 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.822889 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 12:32:46.823036 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.822934 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 12:32:46.823036 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.822940 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 12:32:46.824497 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.824480 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 12:32:46.824581 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.824503 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 12:32:46.825931 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.825913 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 12:32:46.826189 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.826175 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jklxq\"" Oct 31 12:32:46.826346 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.826332 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 12:32:46.833590 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.833568 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 12:32:46.947673 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.947626 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2409df3c-db05-4445-9598-30c275de48af-web-config\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:46.947673 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.947677 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2409df3c-db05-4445-9598-30c275de48af-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:46.947910 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.947707 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2409df3c-db05-4445-9598-30c275de48af-config-volume\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:46.947910 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.947749 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2409df3c-db05-4445-9598-30c275de48af-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:46.947910 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.947791 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2409df3c-db05-4445-9598-30c275de48af-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:46.948063 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.947933 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2409df3c-db05-4445-9598-30c275de48af-config-out\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:46.948063 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.947967 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2409df3c-db05-4445-9598-30c275de48af-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:46.948163 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.948078 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2409df3c-db05-4445-9598-30c275de48af-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:46.948163 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.948113 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2409df3c-db05-4445-9598-30c275de48af-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:46.948233 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.948219 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2409df3c-db05-4445-9598-30c275de48af-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:46.948266 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.948241 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4vmf\" (UniqueName: \"kubernetes.io/projected/2409df3c-db05-4445-9598-30c275de48af-kube-api-access-t4vmf\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:46.948305 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.948267 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2409df3c-db05-4445-9598-30c275de48af-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:46.948361 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:46.948310 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2409df3c-db05-4445-9598-30c275de48af-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:47.049038 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:47.048936 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2409df3c-db05-4445-9598-30c275de48af-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:47.049038 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:47.048977 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2409df3c-db05-4445-9598-30c275de48af-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:47.049258 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:47.049079 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2409df3c-db05-4445-9598-30c275de48af-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:47.049258 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:47.049108 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4vmf\" (UniqueName: \"kubernetes.io/projected/2409df3c-db05-4445-9598-30c275de48af-kube-api-access-t4vmf\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:47.049258 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:47.049133 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2409df3c-db05-4445-9598-30c275de48af-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:47.049258 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:47.049166 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2409df3c-db05-4445-9598-30c275de48af-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:47.049258 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:47.049214 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2409df3c-db05-4445-9598-30c275de48af-web-config\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:47.049258 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:47.049245 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2409df3c-db05-4445-9598-30c275de48af-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:47.049550 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:47.049274 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2409df3c-db05-4445-9598-30c275de48af-config-volume\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:47.049550 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:47.049305 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2409df3c-db05-4445-9598-30c275de48af-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:47.049550 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:47.049346 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2409df3c-db05-4445-9598-30c275de48af-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:47.049550 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:47.049387 6841 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 12:32:47.049550 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:47.049461 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2409df3c-db05-4445-9598-30c275de48af-config-out\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:47.049550 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:47.049495 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2409df3c-db05-4445-9598-30c275de48af-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:47.050005 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:47.049962 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2409df3c-db05-4445-9598-30c275de48af-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:47.050178 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:47.050113 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2409df3c-db05-4445-9598-30c275de48af-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:47.050561 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:47.050538 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2409df3c-db05-4445-9598-30c275de48af-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:47.052658 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:47.052633 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2409df3c-db05-4445-9598-30c275de48af-web-config\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:47.052772 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:47.052637 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2409df3c-db05-4445-9598-30c275de48af-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:47.052818 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:47.052793 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2409df3c-db05-4445-9598-30c275de48af-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:47.053356 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:47.052794 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2409df3c-db05-4445-9598-30c275de48af-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:47.053356 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:47.053064 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2409df3c-db05-4445-9598-30c275de48af-config-volume\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:47.053356 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:47.053207 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2409df3c-db05-4445-9598-30c275de48af-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:47.053356 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:47.053249 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2409df3c-db05-4445-9598-30c275de48af-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:47.053859 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:47.053841 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2409df3c-db05-4445-9598-30c275de48af-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:47.053909 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:47.053878 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2409df3c-db05-4445-9598-30c275de48af-config-out\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:47.070489 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:47.070468 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4vmf\" (UniqueName: \"kubernetes.io/projected/2409df3c-db05-4445-9598-30c275de48af-kube-api-access-t4vmf\") pod \"alertmanager-main-0\" (UID: \"2409df3c-db05-4445-9598-30c275de48af\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:47.140107 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:47.140069 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 12:32:47.425937 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:47.425844 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 12:32:47.430379 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:47.430332 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2409df3c_db05_4445_9598_30c275de48af.slice/crio-3d71d0f64755e9fcdfee5f84a479d62855354f9e9aea339321e1440d58d5b06a WatchSource:0}: Error finding container 3d71d0f64755e9fcdfee5f84a479d62855354f9e9aea339321e1440d58d5b06a: Status 404 returned error can't find the container with id 3d71d0f64755e9fcdfee5f84a479d62855354f9e9aea339321e1440d58d5b06a Oct 31 12:32:47.639208 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:47.639173 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea2d5c42-8213-489b-ba92-369c0435c6e5" path="/var/lib/kubelet/pods/ea2d5c42-8213-489b-ba92-369c0435c6e5/volumes" Oct 31 12:32:47.756960 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:47.756922 6841 generic.go:356] "Generic (PLEG): container finished" podID="2409df3c-db05-4445-9598-30c275de48af" containerID="df84b8843d937dc9102cb47e352c0c7475d8d627dbc81a4ed3a1048495c3eaba" exitCode=0 Oct 31 12:32:47.757428 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:47.757023 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2409df3c-db05-4445-9598-30c275de48af","Type":"ContainerDied","Data":"df84b8843d937dc9102cb47e352c0c7475d8d627dbc81a4ed3a1048495c3eaba"} Oct 31 12:32:47.757428 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:47.757106 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2409df3c-db05-4445-9598-30c275de48af","Type":"ContainerStarted","Data":"3d71d0f64755e9fcdfee5f84a479d62855354f9e9aea339321e1440d58d5b06a"} Oct 31 12:32:48.764793 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:48.764757 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2409df3c-db05-4445-9598-30c275de48af","Type":"ContainerStarted","Data":"b1070b9ab480baeff619465b12bca45527580c1cf46065a4445e75967841fb58"} Oct 31 12:32:48.765202 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:48.764859 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2409df3c-db05-4445-9598-30c275de48af","Type":"ContainerStarted","Data":"8e152c7771933534a44a5e10f3181fbc84c8b8eadc202c40213ef154a06055e1"} Oct 31 12:32:48.765202 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:48.764898 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2409df3c-db05-4445-9598-30c275de48af","Type":"ContainerStarted","Data":"ea2fc22c364daeb17b934b0c4d8b21158e96768378877ff5c870330e2b57042a"} Oct 31 12:32:48.765202 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:48.764922 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2409df3c-db05-4445-9598-30c275de48af","Type":"ContainerStarted","Data":"833638f2d41ef9ee008c2da42aacfa22a85d2969e996b2981a029693adc2eba5"} Oct 31 12:32:48.765202 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:48.764951 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2409df3c-db05-4445-9598-30c275de48af","Type":"ContainerStarted","Data":"c365a515d0a97320104ef91042a60fdef7e4d0d2e5249c1db5a1351a8d34807b"} Oct 31 12:32:48.765202 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:48.764993 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2409df3c-db05-4445-9598-30c275de48af","Type":"ContainerStarted","Data":"473cd0436fe7839943d921cfc646c79d575589397c64e6a1442a291c73d09cc0"} Oct 31 12:32:48.802368 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:48.802307 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.786659009 podStartE2EDuration="2.802289626s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.630617ms" isStatefulPod=false podCreationTimestamp="2026-10-31 12:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:48.800680808 +0000 UTC m=+15.283803703" watchObservedRunningTime="2026-10-31 12:32:48.802289626 +0000 UTC m=+15.285412529" Oct 31 12:32:49.295731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.295695 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 12:32:49.296037 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.295999 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="db5feef6-b3fc-4263-bfc1-66b070df97b4" containerName="prometheus" containerID="cri-o://adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f" gracePeriod=600 Oct 31 12:32:49.296037 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.296028 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="db5feef6-b3fc-4263-bfc1-66b070df97b4" containerName="kube-rbac-proxy-web" containerID="cri-o://a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a" gracePeriod=600 Oct 31 12:32:49.296243 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.296035 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="db5feef6-b3fc-4263-bfc1-66b070df97b4" containerName="kube-rbac-proxy-thanos" containerID="cri-o://241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef" gracePeriod=600 Oct 31 12:32:49.296243 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.296045 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="db5feef6-b3fc-4263-bfc1-66b070df97b4" containerName="config-reloader" containerID="cri-o://ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906" gracePeriod=600 Oct 31 12:32:49.296243 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.296007 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="db5feef6-b3fc-4263-bfc1-66b070df97b4" containerName="kube-rbac-proxy" containerID="cri-o://f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d" gracePeriod=600 Oct 31 12:32:49.296243 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.296052 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="db5feef6-b3fc-4263-bfc1-66b070df97b4" containerName="thanos-sidecar" containerID="cri-o://3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413" gracePeriod=600 Oct 31 12:32:49.499135 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.499099 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:49.534312 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.534288 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:49.675749 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.675663 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db5feef6-b3fc-4263-bfc1-66b070df97b4-kube-api-access-r52xd\" (UniqueName: \"kubernetes.io/projected/db5feef6-b3fc-4263-bfc1-66b070df97b4-kube-api-access-r52xd\") pod \"db5feef6-b3fc-4263-bfc1-66b070df97b4\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " Oct 31 12:32:49.675913 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.675716 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-kube-rbac-proxy\") pod \"db5feef6-b3fc-4263-bfc1-66b070df97b4\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " Oct 31 12:32:49.675913 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.675791 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db5feef6-b3fc-4263-bfc1-66b070df97b4-tls-assets\" (UniqueName: \"kubernetes.io/projected/db5feef6-b3fc-4263-bfc1-66b070df97b4-tls-assets\") pod \"db5feef6-b3fc-4263-bfc1-66b070df97b4\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " Oct 31 12:32:49.675913 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.675822 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"db5feef6-b3fc-4263-bfc1-66b070df97b4\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " Oct 31 12:32:49.675913 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.675860 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-thanos-prometheus-http-client-file\") pod \"db5feef6-b3fc-4263-bfc1-66b070df97b4\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " Oct 31 12:32:49.675913 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.675895 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-web-config\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-web-config\") pod \"db5feef6-b3fc-4263-bfc1-66b070df97b4\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " Oct 31 12:32:49.676220 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.675976 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-configmap-kubelet-serving-ca-bundle\") pod \"db5feef6-b3fc-4263-bfc1-66b070df97b4\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " Oct 31 12:32:49.676220 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.676097 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db5feef6-b3fc-4263-bfc1-66b070df97b4-config-out\" (UniqueName: \"kubernetes.io/empty-dir/db5feef6-b3fc-4263-bfc1-66b070df97b4-config-out\") pod \"db5feef6-b3fc-4263-bfc1-66b070df97b4\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " Oct 31 12:32:49.676220 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.676135 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-configmap-metrics-client-ca\") pod \"db5feef6-b3fc-4263-bfc1-66b070df97b4\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " Oct 31 12:32:49.676220 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.676179 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-trusted-ca-bundle\") pod \"db5feef6-b3fc-4263-bfc1-66b070df97b4\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " Oct 31 12:32:49.676220 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.676209 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-prometheus-k8s-tls\") pod \"db5feef6-b3fc-4263-bfc1-66b070df97b4\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " Oct 31 12:32:49.676458 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.676250 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-db\") pod \"db5feef6-b3fc-4263-bfc1-66b070df97b4\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " Oct 31 12:32:49.676458 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.676290 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-metrics-client-certs\") pod \"db5feef6-b3fc-4263-bfc1-66b070df97b4\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " Oct 31 12:32:49.676458 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.676336 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-rulefiles-0\") pod \"db5feef6-b3fc-4263-bfc1-66b070df97b4\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " Oct 31 12:32:49.676458 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.676370 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-rulefiles-1\") pod \"db5feef6-b3fc-4263-bfc1-66b070df97b4\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " Oct 31 12:32:49.676458 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.676413 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"db5feef6-b3fc-4263-bfc1-66b070df97b4\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " Oct 31 12:32:49.676701 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.676458 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-grpc-tls\") pod \"db5feef6-b3fc-4263-bfc1-66b070df97b4\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " Oct 31 12:32:49.676701 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.676490 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-configmap-serving-certs-ca-bundle\") pod \"db5feef6-b3fc-4263-bfc1-66b070df97b4\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " Oct 31 12:32:49.676701 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.676519 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-config\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-config\") pod \"db5feef6-b3fc-4263-bfc1-66b070df97b4\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " Oct 31 12:32:49.676701 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.676580 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-rulefiles-2\") pod \"db5feef6-b3fc-4263-bfc1-66b070df97b4\" (UID: \"db5feef6-b3fc-4263-bfc1-66b070df97b4\") " Oct 31 12:32:49.677872 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.677379 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-configmap-kubelet-serving-ca-bundle" pod "db5feef6-b3fc-4263-bfc1-66b070df97b4" (UID: "db5feef6-b3fc-4263-bfc1-66b070df97b4"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:32:49.677872 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.677588 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-configmap-metrics-client-ca" pod "db5feef6-b3fc-4263-bfc1-66b070df97b4" (UID: "db5feef6-b3fc-4263-bfc1-66b070df97b4"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:32:49.678469 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.678394 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-rulefiles-2" pod "db5feef6-b3fc-4263-bfc1-66b070df97b4" (UID: "db5feef6-b3fc-4263-bfc1-66b070df97b4"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:32:49.679110 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.678963 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-rulefiles-0" pod "db5feef6-b3fc-4263-bfc1-66b070df97b4" (UID: "db5feef6-b3fc-4263-bfc1-66b070df97b4"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:32:49.679201 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.679178 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-rulefiles-1" pod "db5feef6-b3fc-4263-bfc1-66b070df97b4" (UID: "db5feef6-b3fc-4263-bfc1-66b070df97b4"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:32:49.679291 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.679269 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-db" pod "db5feef6-b3fc-4263-bfc1-66b070df97b4" (UID: "db5feef6-b3fc-4263-bfc1-66b070df97b4"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:32:49.679618 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.679431 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-trusted-ca-bundle" pod "db5feef6-b3fc-4263-bfc1-66b070df97b4" (UID: "db5feef6-b3fc-4263-bfc1-66b070df97b4"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:32:49.679618 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.679455 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-kube-rbac-proxy" pod "db5feef6-b3fc-4263-bfc1-66b070df97b4" (UID: "db5feef6-b3fc-4263-bfc1-66b070df97b4"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:32:49.679618 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.679588 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-configmap-serving-certs-ca-bundle" pod "db5feef6-b3fc-4263-bfc1-66b070df97b4" (UID: "db5feef6-b3fc-4263-bfc1-66b070df97b4"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:32:49.679913 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.679835 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db5feef6-b3fc-4263-bfc1-66b070df97b4-kube-api-access-r52xd" pod "db5feef6-b3fc-4263-bfc1-66b070df97b4" (UID: "db5feef6-b3fc-4263-bfc1-66b070df97b4"). InnerVolumeSpecName "kube-api-access-r52xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:32:49.679913 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.679860 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-prometheus-k8s-thanos-sidecar-tls" pod "db5feef6-b3fc-4263-bfc1-66b070df97b4" (UID: "db5feef6-b3fc-4263-bfc1-66b070df97b4"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:32:49.680059 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.680014 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db5feef6-b3fc-4263-bfc1-66b070df97b4-tls-assets" pod "db5feef6-b3fc-4263-bfc1-66b070df97b4" (UID: "db5feef6-b3fc-4263-bfc1-66b070df97b4"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:32:49.680353 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.680310 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-config" pod "db5feef6-b3fc-4263-bfc1-66b070df97b4" (UID: "db5feef6-b3fc-4263-bfc1-66b070df97b4"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:32:49.680607 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.680582 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-grpc-tls" pod "db5feef6-b3fc-4263-bfc1-66b070df97b4" (UID: "db5feef6-b3fc-4263-bfc1-66b070df97b4"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:32:49.680946 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.680920 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-metrics-client-certs" pod "db5feef6-b3fc-4263-bfc1-66b070df97b4" (UID: "db5feef6-b3fc-4263-bfc1-66b070df97b4"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:32:49.681069 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.681052 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-prometheus-k8s-kube-rbac-proxy-web" pod "db5feef6-b3fc-4263-bfc1-66b070df97b4" (UID: "db5feef6-b3fc-4263-bfc1-66b070df97b4"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:32:49.681322 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.681307 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-thanos-prometheus-http-client-file" pod "db5feef6-b3fc-4263-bfc1-66b070df97b4" (UID: "db5feef6-b3fc-4263-bfc1-66b070df97b4"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:32:49.681434 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.681415 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-prometheus-k8s-tls" pod "db5feef6-b3fc-4263-bfc1-66b070df97b4" (UID: "db5feef6-b3fc-4263-bfc1-66b070df97b4"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:32:49.681660 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.681642 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db5feef6-b3fc-4263-bfc1-66b070df97b4-config-out" pod "db5feef6-b3fc-4263-bfc1-66b070df97b4" (UID: "db5feef6-b3fc-4263-bfc1-66b070df97b4"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:32:49.689459 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.689433 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-web-config" pod "db5feef6-b3fc-4263-bfc1-66b070df97b4" (UID: "db5feef6-b3fc-4263-bfc1-66b070df97b4"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:32:49.771152 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.771122 6841 generic.go:356] "Generic (PLEG): container finished" podID="db5feef6-b3fc-4263-bfc1-66b070df97b4" containerID="241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef" exitCode=0 Oct 31 12:32:49.771152 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.771146 6841 generic.go:356] "Generic (PLEG): container finished" podID="db5feef6-b3fc-4263-bfc1-66b070df97b4" containerID="f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d" exitCode=0 Oct 31 12:32:49.771152 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.771156 6841 generic.go:356] "Generic (PLEG): container finished" podID="db5feef6-b3fc-4263-bfc1-66b070df97b4" containerID="a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a" exitCode=0 Oct 31 12:32:49.771611 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.771163 6841 generic.go:356] "Generic (PLEG): container finished" podID="db5feef6-b3fc-4263-bfc1-66b070df97b4" containerID="3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413" exitCode=0 Oct 31 12:32:49.771611 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.771171 6841 generic.go:356] "Generic (PLEG): container finished" podID="db5feef6-b3fc-4263-bfc1-66b070df97b4" containerID="ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906" exitCode=0 Oct 31 12:32:49.771611 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.771176 6841 generic.go:356] "Generic (PLEG): container finished" podID="db5feef6-b3fc-4263-bfc1-66b070df97b4" containerID="adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f" exitCode=0 Oct 31 12:32:49.771611 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.771203 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:49.771611 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.771207 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db5feef6-b3fc-4263-bfc1-66b070df97b4","Type":"ContainerDied","Data":"241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef"} Oct 31 12:32:49.771611 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.771271 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db5feef6-b3fc-4263-bfc1-66b070df97b4","Type":"ContainerDied","Data":"f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d"} Oct 31 12:32:49.771611 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.771301 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db5feef6-b3fc-4263-bfc1-66b070df97b4","Type":"ContainerDied","Data":"a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a"} Oct 31 12:32:49.771611 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.771326 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db5feef6-b3fc-4263-bfc1-66b070df97b4","Type":"ContainerDied","Data":"3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413"} Oct 31 12:32:49.771611 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.771335 6841 scope.go:122] "RemoveContainer" containerID="241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef" Oct 31 12:32:49.771611 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.771353 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db5feef6-b3fc-4263-bfc1-66b070df97b4","Type":"ContainerDied","Data":"ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906"} Oct 31 12:32:49.771611 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.771378 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db5feef6-b3fc-4263-bfc1-66b070df97b4","Type":"ContainerDied","Data":"adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f"} Oct 31 12:32:49.771611 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.771402 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db5feef6-b3fc-4263-bfc1-66b070df97b4","Type":"ContainerDied","Data":"8d00465d2d3bd7107ed192550f20c47c88dd6e2d52bff4bd6d44a90e4036a013"} Oct 31 12:32:49.778840 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.778816 6841 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:49.778840 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.778844 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r52xd\" (UniqueName: \"kubernetes.io/projected/db5feef6-b3fc-4263-bfc1-66b070df97b4-kube-api-access-r52xd\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:49.779038 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.778860 6841 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-kube-rbac-proxy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:49.779038 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.778881 6841 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/db5feef6-b3fc-4263-bfc1-66b070df97b4-tls-assets\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:49.779038 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.778895 6841 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:49.779038 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.778910 6841 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:49.779038 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.778926 6841 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-web-config\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:49.779038 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.778947 6841 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:49.779038 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.778963 6841 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/db5feef6-b3fc-4263-bfc1-66b070df97b4-config-out\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:49.779038 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.778996 6841 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-configmap-metrics-client-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:49.779038 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.779014 6841 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:49.779038 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.779034 6841 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:49.779480 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.779049 6841 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-db\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:49.779480 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.779066 6841 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-metrics-client-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:49.779480 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.779084 6841 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:49.779480 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.779104 6841 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:49.779480 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.779121 6841 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:49.779480 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.779137 6841 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-secret-grpc-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:49.779480 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.779153 6841 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db5feef6-b3fc-4263-bfc1-66b070df97b4-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:49.779480 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.779176 6841 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/db5feef6-b3fc-4263-bfc1-66b070df97b4-config\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:49.780359 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.780291 6841 scope.go:122] "RemoveContainer" containerID="f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d" Oct 31 12:32:49.787700 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.787679 6841 scope.go:122] "RemoveContainer" containerID="a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a" Oct 31 12:32:49.794078 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.794058 6841 scope.go:122] "RemoveContainer" containerID="3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413" Oct 31 12:32:49.799386 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.799365 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 12:32:49.801099 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.801083 6841 scope.go:122] "RemoveContainer" containerID="ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906" Oct 31 12:32:49.804349 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.804330 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 12:32:49.808003 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.807970 6841 scope.go:122] "RemoveContainer" containerID="adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f" Oct 31 12:32:49.814845 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.814830 6841 scope.go:122] "RemoveContainer" containerID="4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc" Oct 31 12:32:49.821141 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.821125 6841 scope.go:122] "RemoveContainer" containerID="241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef" Oct 31 12:32:49.821408 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:49.821390 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef\": container with ID starting with 241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef not found: ID does not exist" containerID="241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef" Oct 31 12:32:49.821448 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.821419 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef"} err="failed to get container status \"241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef\": rpc error: code = NotFound desc = could not find container \"241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef\": container with ID starting with 241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef not found: ID does not exist" Oct 31 12:32:49.821448 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.821437 6841 scope.go:122] "RemoveContainer" containerID="f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d" Oct 31 12:32:49.821676 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:49.821659 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d\": container with ID starting with f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d not found: ID does not exist" containerID="f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d" Oct 31 12:32:49.821723 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.821682 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d"} err="failed to get container status \"f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d\": rpc error: code = NotFound desc = could not find container \"f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d\": container with ID starting with f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d not found: ID does not exist" Oct 31 12:32:49.821723 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.821696 6841 scope.go:122] "RemoveContainer" containerID="a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a" Oct 31 12:32:49.821922 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:49.821907 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a\": container with ID starting with a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a not found: ID does not exist" containerID="a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a" Oct 31 12:32:49.821961 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.821923 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a"} err="failed to get container status \"a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a\": rpc error: code = NotFound desc = could not find container \"a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a\": container with ID starting with a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a not found: ID does not exist" Oct 31 12:32:49.821961 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.821935 6841 scope.go:122] "RemoveContainer" containerID="3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413" Oct 31 12:32:49.822163 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:49.822148 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413\": container with ID starting with 3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413 not found: ID does not exist" containerID="3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413" Oct 31 12:32:49.822220 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.822168 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413"} err="failed to get container status \"3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413\": rpc error: code = NotFound desc = could not find container \"3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413\": container with ID starting with 3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413 not found: ID does not exist" Oct 31 12:32:49.822220 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.822182 6841 scope.go:122] "RemoveContainer" containerID="ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906" Oct 31 12:32:49.822430 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:49.822413 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906\": container with ID starting with ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906 not found: ID does not exist" containerID="ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906" Oct 31 12:32:49.822465 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.822435 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906"} err="failed to get container status \"ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906\": rpc error: code = NotFound desc = could not find container \"ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906\": container with ID starting with ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906 not found: ID does not exist" Oct 31 12:32:49.822465 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.822449 6841 scope.go:122] "RemoveContainer" containerID="adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f" Oct 31 12:32:49.822665 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:49.822650 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f\": container with ID starting with adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f not found: ID does not exist" containerID="adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f" Oct 31 12:32:49.822697 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.822670 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f"} err="failed to get container status \"adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f\": rpc error: code = NotFound desc = could not find container \"adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f\": container with ID starting with adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f not found: ID does not exist" Oct 31 12:32:49.822697 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.822689 6841 scope.go:122] "RemoveContainer" containerID="4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc" Oct 31 12:32:49.822933 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:49.822917 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc\": container with ID starting with 4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc not found: ID does not exist" containerID="4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc" Oct 31 12:32:49.822964 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.822937 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc"} err="failed to get container status \"4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc\": rpc error: code = NotFound desc = could not find container \"4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc\": container with ID starting with 4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc not found: ID does not exist" Oct 31 12:32:49.822964 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.822953 6841 scope.go:122] "RemoveContainer" containerID="241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef" Oct 31 12:32:49.823168 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.823142 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef"} err="failed to get container status \"241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef\": rpc error: code = NotFound desc = could not find container \"241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef\": container with ID starting with 241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef not found: ID does not exist" Oct 31 12:32:49.823206 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.823168 6841 scope.go:122] "RemoveContainer" containerID="f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d" Oct 31 12:32:49.823353 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.823336 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d"} err="failed to get container status \"f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d\": rpc error: code = NotFound desc = could not find container \"f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d\": container with ID starting with f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d not found: ID does not exist" Oct 31 12:32:49.823353 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.823352 6841 scope.go:122] "RemoveContainer" containerID="a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a" Oct 31 12:32:49.823549 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.823530 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a"} err="failed to get container status \"a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a\": rpc error: code = NotFound desc = could not find container \"a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a\": container with ID starting with a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a not found: ID does not exist" Oct 31 12:32:49.823592 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.823550 6841 scope.go:122] "RemoveContainer" containerID="3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413" Oct 31 12:32:49.823768 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.823750 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413"} err="failed to get container status \"3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413\": rpc error: code = NotFound desc = could not find container \"3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413\": container with ID starting with 3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413 not found: ID does not exist" Oct 31 12:32:49.823805 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.823769 6841 scope.go:122] "RemoveContainer" containerID="ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906" Oct 31 12:32:49.824004 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.823975 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906"} err="failed to get container status \"ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906\": rpc error: code = NotFound desc = could not find container \"ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906\": container with ID starting with ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906 not found: ID does not exist" Oct 31 12:32:49.824054 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.824005 6841 scope.go:122] "RemoveContainer" containerID="adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f" Oct 31 12:32:49.824255 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.824237 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f"} err="failed to get container status \"adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f\": rpc error: code = NotFound desc = could not find container \"adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f\": container with ID starting with adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f not found: ID does not exist" Oct 31 12:32:49.824312 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.824255 6841 scope.go:122] "RemoveContainer" containerID="4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc" Oct 31 12:32:49.824522 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.824501 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc"} err="failed to get container status \"4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc\": rpc error: code = NotFound desc = could not find container \"4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc\": container with ID starting with 4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc not found: ID does not exist" Oct 31 12:32:49.824522 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.824522 6841 scope.go:122] "RemoveContainer" containerID="241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef" Oct 31 12:32:49.824943 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.824911 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef"} err="failed to get container status \"241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef\": rpc error: code = NotFound desc = could not find container \"241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef\": container with ID starting with 241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef not found: ID does not exist" Oct 31 12:32:49.824943 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.824937 6841 scope.go:122] "RemoveContainer" containerID="f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d" Oct 31 12:32:49.825204 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.825184 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d"} err="failed to get container status \"f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d\": rpc error: code = NotFound desc = could not find container \"f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d\": container with ID starting with f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d not found: ID does not exist" Oct 31 12:32:49.825276 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.825206 6841 scope.go:122] "RemoveContainer" containerID="a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a" Oct 31 12:32:49.825474 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.825455 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a"} err="failed to get container status \"a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a\": rpc error: code = NotFound desc = could not find container \"a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a\": container with ID starting with a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a not found: ID does not exist" Oct 31 12:32:49.825516 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.825475 6841 scope.go:122] "RemoveContainer" containerID="3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413" Oct 31 12:32:49.825676 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.825658 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413"} err="failed to get container status \"3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413\": rpc error: code = NotFound desc = could not find container \"3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413\": container with ID starting with 3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413 not found: ID does not exist" Oct 31 12:32:49.825747 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.825678 6841 scope.go:122] "RemoveContainer" containerID="ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906" Oct 31 12:32:49.825900 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.825882 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906"} err="failed to get container status \"ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906\": rpc error: code = NotFound desc = could not find container \"ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906\": container with ID starting with ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906 not found: ID does not exist" Oct 31 12:32:49.825940 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.825901 6841 scope.go:122] "RemoveContainer" containerID="adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f" Oct 31 12:32:49.826199 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.826180 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f"} err="failed to get container status \"adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f\": rpc error: code = NotFound desc = could not find container \"adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f\": container with ID starting with adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f not found: ID does not exist" Oct 31 12:32:49.826242 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.826199 6841 scope.go:122] "RemoveContainer" containerID="4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc" Oct 31 12:32:49.826436 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.826418 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc"} err="failed to get container status \"4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc\": rpc error: code = NotFound desc = could not find container \"4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc\": container with ID starting with 4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc not found: ID does not exist" Oct 31 12:32:49.826508 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.826438 6841 scope.go:122] "RemoveContainer" containerID="241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef" Oct 31 12:32:49.826667 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.826651 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef"} err="failed to get container status \"241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef\": rpc error: code = NotFound desc = could not find container \"241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef\": container with ID starting with 241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef not found: ID does not exist" Oct 31 12:32:49.826708 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.826667 6841 scope.go:122] "RemoveContainer" containerID="f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d" Oct 31 12:32:49.826863 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.826845 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d"} err="failed to get container status \"f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d\": rpc error: code = NotFound desc = could not find container \"f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d\": container with ID starting with f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d not found: ID does not exist" Oct 31 12:32:49.826933 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.826864 6841 scope.go:122] "RemoveContainer" containerID="a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a" Oct 31 12:32:49.827082 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.827065 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a"} err="failed to get container status \"a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a\": rpc error: code = NotFound desc = could not find container \"a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a\": container with ID starting with a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a not found: ID does not exist" Oct 31 12:32:49.827140 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.827082 6841 scope.go:122] "RemoveContainer" containerID="3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413" Oct 31 12:32:49.827263 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.827246 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413"} err="failed to get container status \"3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413\": rpc error: code = NotFound desc = could not find container \"3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413\": container with ID starting with 3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413 not found: ID does not exist" Oct 31 12:32:49.827321 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.827264 6841 scope.go:122] "RemoveContainer" containerID="ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906" Oct 31 12:32:49.827464 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.827447 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906"} err="failed to get container status \"ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906\": rpc error: code = NotFound desc = could not find container \"ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906\": container with ID starting with ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906 not found: ID does not exist" Oct 31 12:32:49.827528 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.827466 6841 scope.go:122] "RemoveContainer" containerID="adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f" Oct 31 12:32:49.827684 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.827668 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f"} err="failed to get container status \"adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f\": rpc error: code = NotFound desc = could not find container \"adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f\": container with ID starting with adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f not found: ID does not exist" Oct 31 12:32:49.827723 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.827685 6841 scope.go:122] "RemoveContainer" containerID="4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc" Oct 31 12:32:49.827872 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.827855 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc"} err="failed to get container status \"4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc\": rpc error: code = NotFound desc = could not find container \"4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc\": container with ID starting with 4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc not found: ID does not exist" Oct 31 12:32:49.827932 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.827874 6841 scope.go:122] "RemoveContainer" containerID="241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef" Oct 31 12:32:49.828128 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.828113 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef"} err="failed to get container status \"241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef\": rpc error: code = NotFound desc = could not find container \"241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef\": container with ID starting with 241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef not found: ID does not exist" Oct 31 12:32:49.828181 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.828129 6841 scope.go:122] "RemoveContainer" containerID="f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d" Oct 31 12:32:49.828323 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.828305 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d"} err="failed to get container status \"f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d\": rpc error: code = NotFound desc = could not find container \"f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d\": container with ID starting with f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d not found: ID does not exist" Oct 31 12:32:49.828359 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.828323 6841 scope.go:122] "RemoveContainer" containerID="a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a" Oct 31 12:32:49.828508 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.828492 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a"} err="failed to get container status \"a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a\": rpc error: code = NotFound desc = could not find container \"a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a\": container with ID starting with a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a not found: ID does not exist" Oct 31 12:32:49.828544 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.828508 6841 scope.go:122] "RemoveContainer" containerID="3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413" Oct 31 12:32:49.828700 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.828680 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413"} err="failed to get container status \"3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413\": rpc error: code = NotFound desc = could not find container \"3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413\": container with ID starting with 3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413 not found: ID does not exist" Oct 31 12:32:49.828767 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.828701 6841 scope.go:122] "RemoveContainer" containerID="ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906" Oct 31 12:32:49.828898 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.828882 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906"} err="failed to get container status \"ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906\": rpc error: code = NotFound desc = could not find container \"ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906\": container with ID starting with ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906 not found: ID does not exist" Oct 31 12:32:49.828934 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.828898 6841 scope.go:122] "RemoveContainer" containerID="adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f" Oct 31 12:32:49.829094 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.829074 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f"} err="failed to get container status \"adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f\": rpc error: code = NotFound desc = could not find container \"adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f\": container with ID starting with adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f not found: ID does not exist" Oct 31 12:32:49.829165 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.829095 6841 scope.go:122] "RemoveContainer" containerID="4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc" Oct 31 12:32:49.829286 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.829271 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc"} err="failed to get container status \"4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc\": rpc error: code = NotFound desc = could not find container \"4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc\": container with ID starting with 4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc not found: ID does not exist" Oct 31 12:32:49.829333 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.829286 6841 scope.go:122] "RemoveContainer" containerID="241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef" Oct 31 12:32:49.829486 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.829469 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef"} err="failed to get container status \"241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef\": rpc error: code = NotFound desc = could not find container \"241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef\": container with ID starting with 241414025feb6a636a2944441640f9c081dee2d0e8c7f346e246193f10eb10ef not found: ID does not exist" Oct 31 12:32:49.829546 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.829486 6841 scope.go:122] "RemoveContainer" containerID="f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d" Oct 31 12:32:49.829695 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.829680 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d"} err="failed to get container status \"f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d\": rpc error: code = NotFound desc = could not find container \"f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d\": container with ID starting with f9f3161bd4de40f1aa9b8ac2b33fb9210849ba31d3b73554024325f2493d994d not found: ID does not exist" Oct 31 12:32:49.829735 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.829695 6841 scope.go:122] "RemoveContainer" containerID="a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a" Oct 31 12:32:49.829874 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.829859 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a"} err="failed to get container status \"a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a\": rpc error: code = NotFound desc = could not find container \"a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a\": container with ID starting with a687e9c6e1da89f5fcd8e2482050d46e4f1a87c1d71da585b9da92877afcca7a not found: ID does not exist" Oct 31 12:32:49.829915 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.829874 6841 scope.go:122] "RemoveContainer" containerID="3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413" Oct 31 12:32:49.830071 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.830054 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413"} err="failed to get container status \"3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413\": rpc error: code = NotFound desc = could not find container \"3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413\": container with ID starting with 3f21bde64311c13c02e87aa76c6db754ec4ab7bfac11e300f6edaaad8eff4413 not found: ID does not exist" Oct 31 12:32:49.830071 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.830071 6841 scope.go:122] "RemoveContainer" containerID="ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906" Oct 31 12:32:49.830247 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.830232 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906"} err="failed to get container status \"ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906\": rpc error: code = NotFound desc = could not find container \"ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906\": container with ID starting with ff3004496dc132992bdd5dc324deb3e21a38e6ef898da2188ccec65b430b9906 not found: ID does not exist" Oct 31 12:32:49.830283 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.830247 6841 scope.go:122] "RemoveContainer" containerID="adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f" Oct 31 12:32:49.830423 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.830408 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f"} err="failed to get container status \"adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f\": rpc error: code = NotFound desc = could not find container \"adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f\": container with ID starting with adae324e3caab03e5e5ad79cdb7bc4aaf8bc65d193b73f3b6e51d940934e784f not found: ID does not exist" Oct 31 12:32:49.830459 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.830423 6841 scope.go:122] "RemoveContainer" containerID="4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc" Oct 31 12:32:49.830620 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.830604 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc"} err="failed to get container status \"4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc\": rpc error: code = NotFound desc = could not find container \"4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc\": container with ID starting with 4beb6a9f15d46955d32a74be812f182a3279ce0b2831eba2f49b89ed1bbec6dc not found: ID does not exist" Oct 31 12:32:49.833234 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.833214 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 12:32:49.833545 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.833530 6841 cpu_manager.go:414] "containerMap: removing container" podUID="db5feef6-b3fc-4263-bfc1-66b070df97b4" containerName="kube-rbac-proxy-thanos" Oct 31 12:32:49.833611 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.833549 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db5feef6-b3fc-4263-bfc1-66b070df97b4" containerName="kube-rbac-proxy-thanos" Oct 31 12:32:49.833611 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.833568 6841 cpu_manager.go:414] "containerMap: removing container" podUID="db5feef6-b3fc-4263-bfc1-66b070df97b4" containerName="kube-rbac-proxy-web" Oct 31 12:32:49.833611 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.833578 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db5feef6-b3fc-4263-bfc1-66b070df97b4" containerName="kube-rbac-proxy-web" Oct 31 12:32:49.833611 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.833589 6841 cpu_manager.go:414] "containerMap: removing container" podUID="db5feef6-b3fc-4263-bfc1-66b070df97b4" containerName="thanos-sidecar" Oct 31 12:32:49.833611 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.833596 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db5feef6-b3fc-4263-bfc1-66b070df97b4" containerName="thanos-sidecar" Oct 31 12:32:49.833854 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.833619 6841 cpu_manager.go:414] "containerMap: removing container" podUID="db5feef6-b3fc-4263-bfc1-66b070df97b4" containerName="kube-rbac-proxy" Oct 31 12:32:49.833854 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.833626 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db5feef6-b3fc-4263-bfc1-66b070df97b4" containerName="kube-rbac-proxy" Oct 31 12:32:49.833854 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.833637 6841 cpu_manager.go:414] "containerMap: removing container" podUID="db5feef6-b3fc-4263-bfc1-66b070df97b4" containerName="prometheus" Oct 31 12:32:49.833854 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.833644 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db5feef6-b3fc-4263-bfc1-66b070df97b4" containerName="prometheus" Oct 31 12:32:49.833854 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.833659 6841 cpu_manager.go:414] "containerMap: removing container" podUID="db5feef6-b3fc-4263-bfc1-66b070df97b4" containerName="config-reloader" Oct 31 12:32:49.833854 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.833666 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db5feef6-b3fc-4263-bfc1-66b070df97b4" containerName="config-reloader" Oct 31 12:32:49.833854 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.833678 6841 cpu_manager.go:414] "containerMap: removing container" podUID="db5feef6-b3fc-4263-bfc1-66b070df97b4" containerName="init-config-reloader" Oct 31 12:32:49.833854 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.833685 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db5feef6-b3fc-4263-bfc1-66b070df97b4" containerName="init-config-reloader" Oct 31 12:32:49.833854 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.833752 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="db5feef6-b3fc-4263-bfc1-66b070df97b4" containerName="kube-rbac-proxy" Oct 31 12:32:49.833854 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.833763 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="db5feef6-b3fc-4263-bfc1-66b070df97b4" containerName="prometheus" Oct 31 12:32:49.833854 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.833772 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="db5feef6-b3fc-4263-bfc1-66b070df97b4" containerName="config-reloader" Oct 31 12:32:49.833854 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.833784 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="db5feef6-b3fc-4263-bfc1-66b070df97b4" containerName="init-config-reloader" Oct 31 12:32:49.833854 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.833796 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="db5feef6-b3fc-4263-bfc1-66b070df97b4" containerName="kube-rbac-proxy-thanos" Oct 31 12:32:49.833854 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.833804 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="db5feef6-b3fc-4263-bfc1-66b070df97b4" containerName="kube-rbac-proxy-web" Oct 31 12:32:49.833854 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.833811 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="db5feef6-b3fc-4263-bfc1-66b070df97b4" containerName="thanos-sidecar" Oct 31 12:32:49.839156 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.839138 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:49.847234 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.847215 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 12:32:49.847378 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.847363 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 12:32:49.847432 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.847422 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 12:32:49.848412 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.848395 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 12:32:49.857371 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.857346 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 12:32:49.857459 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.857411 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 12:32:49.857459 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.857412 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 12:32:49.858207 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.858188 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 12:32:49.858458 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.858442 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 12:32:49.858532 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.858455 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 12:32:49.858532 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.858474 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 12:32:49.858532 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.858479 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 12:32:49.858532 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.858487 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-bgz64\"" Oct 31 12:32:49.858532 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.858497 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 12:32:49.858532 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.858518 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 12:32:49.858725 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.858551 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-crggt40j4ekbj\"" Oct 31 12:32:49.858725 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.858606 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 12:32:49.859871 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.859853 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 12:32:49.981652 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.981620 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d4e20b11-4a0f-484f-930e-32f25592e75d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:49.981652 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.981657 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d4e20b11-4a0f-484f-930e-32f25592e75d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:49.981839 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.981674 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d4e20b11-4a0f-484f-930e-32f25592e75d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:49.981839 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.981731 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hgm5\" (UniqueName: \"kubernetes.io/projected/d4e20b11-4a0f-484f-930e-32f25592e75d-kube-api-access-7hgm5\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:49.981903 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.981848 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d4e20b11-4a0f-484f-930e-32f25592e75d-config\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:49.981903 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.981892 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4e20b11-4a0f-484f-930e-32f25592e75d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:49.981960 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.981916 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d4e20b11-4a0f-484f-930e-32f25592e75d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:49.981960 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.981937 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d4e20b11-4a0f-484f-930e-32f25592e75d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:49.982113 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.982030 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4e20b11-4a0f-484f-930e-32f25592e75d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:49.982113 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.982082 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d4e20b11-4a0f-484f-930e-32f25592e75d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:49.982177 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.982121 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d4e20b11-4a0f-484f-930e-32f25592e75d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:49.982177 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.982136 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d4e20b11-4a0f-484f-930e-32f25592e75d-web-config\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:49.982177 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.982150 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4e20b11-4a0f-484f-930e-32f25592e75d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:49.982268 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.982179 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d4e20b11-4a0f-484f-930e-32f25592e75d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:49.982297 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.982277 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d4e20b11-4a0f-484f-930e-32f25592e75d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:49.982329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.982305 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d4e20b11-4a0f-484f-930e-32f25592e75d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:49.982329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.982320 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d4e20b11-4a0f-484f-930e-32f25592e75d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:49.982503 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.982392 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d4e20b11-4a0f-484f-930e-32f25592e75d-config-out\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:49.982503 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.982411 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d4e20b11-4a0f-484f-930e-32f25592e75d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:49.982503 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:49.982429 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d4e20b11-4a0f-484f-930e-32f25592e75d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.032466 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:50.032431 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:32:50.083122 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.083090 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d4e20b11-4a0f-484f-930e-32f25592e75d-config\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.083236 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.083152 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4e20b11-4a0f-484f-930e-32f25592e75d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.083236 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.083191 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d4e20b11-4a0f-484f-930e-32f25592e75d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.083236 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.083221 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d4e20b11-4a0f-484f-930e-32f25592e75d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.083364 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.083261 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4e20b11-4a0f-484f-930e-32f25592e75d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.083364 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.083288 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d4e20b11-4a0f-484f-930e-32f25592e75d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.083364 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.083318 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d4e20b11-4a0f-484f-930e-32f25592e75d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.083364 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.083342 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d4e20b11-4a0f-484f-930e-32f25592e75d-web-config\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.083559 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.083362 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4e20b11-4a0f-484f-930e-32f25592e75d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.083559 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.083401 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d4e20b11-4a0f-484f-930e-32f25592e75d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.083559 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.083458 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d4e20b11-4a0f-484f-930e-32f25592e75d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.083559 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.083485 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d4e20b11-4a0f-484f-930e-32f25592e75d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.083559 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.083507 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d4e20b11-4a0f-484f-930e-32f25592e75d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.083815 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.083566 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d4e20b11-4a0f-484f-930e-32f25592e75d-config-out\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.083815 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.083589 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d4e20b11-4a0f-484f-930e-32f25592e75d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.083815 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.083618 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d4e20b11-4a0f-484f-930e-32f25592e75d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.083815 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.083673 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d4e20b11-4a0f-484f-930e-32f25592e75d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.083815 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.083702 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d4e20b11-4a0f-484f-930e-32f25592e75d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.083815 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.083732 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d4e20b11-4a0f-484f-930e-32f25592e75d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.083815 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.083762 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hgm5\" (UniqueName: \"kubernetes.io/projected/d4e20b11-4a0f-484f-930e-32f25592e75d-kube-api-access-7hgm5\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.084448 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.083904 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d4e20b11-4a0f-484f-930e-32f25592e75d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.084448 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.083927 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4e20b11-4a0f-484f-930e-32f25592e75d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.084448 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.083977 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4e20b11-4a0f-484f-930e-32f25592e75d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.085202 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.085175 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4e20b11-4a0f-484f-930e-32f25592e75d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.086473 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.086451 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d4e20b11-4a0f-484f-930e-32f25592e75d-config\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.086574 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.086534 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d4e20b11-4a0f-484f-930e-32f25592e75d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.087099 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.086766 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d4e20b11-4a0f-484f-930e-32f25592e75d-web-config\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.087099 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.086882 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d4e20b11-4a0f-484f-930e-32f25592e75d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.087249 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.087164 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d4e20b11-4a0f-484f-930e-32f25592e75d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.087249 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.087185 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d4e20b11-4a0f-484f-930e-32f25592e75d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.087379 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.087350 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d4e20b11-4a0f-484f-930e-32f25592e75d-config-out\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.087585 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.087554 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d4e20b11-4a0f-484f-930e-32f25592e75d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.087698 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.087646 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d4e20b11-4a0f-484f-930e-32f25592e75d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.088140 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.088119 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d4e20b11-4a0f-484f-930e-32f25592e75d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.088433 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.088414 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d4e20b11-4a0f-484f-930e-32f25592e75d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.088632 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.088609 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d4e20b11-4a0f-484f-930e-32f25592e75d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.088856 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.088832 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d4e20b11-4a0f-484f-930e-32f25592e75d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.089061 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.089042 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d4e20b11-4a0f-484f-930e-32f25592e75d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.089298 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.089282 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d4e20b11-4a0f-484f-930e-32f25592e75d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.095978 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.095961 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hgm5\" (UniqueName: \"kubernetes.io/projected/d4e20b11-4a0f-484f-930e-32f25592e75d-kube-api-access-7hgm5\") pod \"prometheus-k8s-0\" (UID: \"d4e20b11-4a0f-484f-930e-32f25592e75d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.164847 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.164812 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:50.386263 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.386217 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-serving-cert\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:50.386263 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.386274 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-oauth-serving-cert\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:50.386506 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.386306 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-oauth-config\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:50.386506 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.386323 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-config\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:50.386506 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.386374 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-trusted-ca-bundle\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:50.386506 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:50.386384 6841 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 31 12:32:50.386506 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.386404 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-service-ca\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:50.386506 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:50.386422 6841 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 31 12:32:50.386506 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:50.386448 6841 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 31 12:32:50.386506 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:50.386454 6841 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 12:32:50.386506 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:50.386466 6841 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 31 12:32:50.386506 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:50.386454 6841 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 31 12:32:50.386807 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:50.386465 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-serving-cert podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:33:06.386445259 +0000 UTC m=+32.869568143 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-serving-cert") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : object "openshift-console"/"console-serving-cert" not registered Oct 31 12:32:50.386807 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:50.386536 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-oauth-serving-cert podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:33:06.386523307 +0000 UTC m=+32.869646200 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-oauth-serving-cert") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : object "openshift-console"/"oauth-serving-cert" not registered Oct 31 12:32:50.386807 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:50.386550 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-oauth-config podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:33:06.386541576 +0000 UTC m=+32.869664464 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-oauth-config") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : object "openshift-console"/"console-oauth-config" not registered Oct 31 12:32:50.386807 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:50.386570 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-trusted-ca-bundle podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:33:06.386561454 +0000 UTC m=+32.869684349 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-trusted-ca-bundle") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 12:32:50.386807 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:50.386589 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-service-ca podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:33:06.386581091 +0000 UTC m=+32.869703985 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-service-ca") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : object "openshift-console"/"service-ca" not registered Oct 31 12:32:50.386807 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:50.386608 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-config podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:33:06.386599068 +0000 UTC m=+32.869721962 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-config") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : object "openshift-console"/"console-config" not registered Oct 31 12:32:50.460137 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.460113 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 12:32:50.461451 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:32:50.461407 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e20b11_4a0f_484f_930e_32f25592e75d.slice/crio-cf2707b63225b599c7c142dae3a8f5b4a22d9584c2e015c06a92418c8e82f242 WatchSource:0}: Error finding container cf2707b63225b599c7c142dae3a8f5b4a22d9584c2e015c06a92418c8e82f242: Status 404 returned error can't find the container with id cf2707b63225b599c7c142dae3a8f5b4a22d9584c2e015c06a92418c8e82f242 Oct 31 12:32:50.487603 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.487578 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8gsv\" (UniqueName: \"kubernetes.io/projected/f84bcfff-6360-4722-965f-8b3f152f2bdf-kube-api-access-b8gsv\") pod \"console-668c5b5566-4scvp\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:50.487712 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:50.487700 6841 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 31 12:32:50.487750 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:50.487715 6841 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 31 12:32:50.487750 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:50.487725 6841 projected.go:196] Error preparing data for projected volume kube-api-access-b8gsv for pod openshift-console/console-668c5b5566-4scvp: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 12:32:50.487809 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:32:50.487771 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f84bcfff-6360-4722-965f-8b3f152f2bdf-kube-api-access-b8gsv podName:f84bcfff-6360-4722-965f-8b3f152f2bdf nodeName:}" failed. No retries permitted until 2026-10-31 12:33:06.487759824 +0000 UTC m=+32.970882719 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-b8gsv" (UniqueName: "kubernetes.io/projected/f84bcfff-6360-4722-965f-8b3f152f2bdf-kube-api-access-b8gsv") pod "console-668c5b5566-4scvp" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 12:32:50.776689 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.776658 6841 generic.go:356] "Generic (PLEG): container finished" podID="d4e20b11-4a0f-484f-930e-32f25592e75d" containerID="85aad35cb6887a36219bc13483e2554aef9582fce4b1f32b1158613f867fae52" exitCode=0 Oct 31 12:32:50.777139 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.776746 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d4e20b11-4a0f-484f-930e-32f25592e75d","Type":"ContainerDied","Data":"85aad35cb6887a36219bc13483e2554aef9582fce4b1f32b1158613f867fae52"} Oct 31 12:32:50.777139 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:50.776830 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d4e20b11-4a0f-484f-930e-32f25592e75d","Type":"ContainerStarted","Data":"cf2707b63225b599c7c142dae3a8f5b4a22d9584c2e015c06a92418c8e82f242"} Oct 31 12:32:51.640784 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:51.640709 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db5feef6-b3fc-4263-bfc1-66b070df97b4" path="/var/lib/kubelet/pods/db5feef6-b3fc-4263-bfc1-66b070df97b4/volumes" Oct 31 12:32:51.785285 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:51.785254 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d4e20b11-4a0f-484f-930e-32f25592e75d","Type":"ContainerStarted","Data":"2e4f52bf83c97a50ed1caf94dfa3df171dc73737a490d019a00b39b5011262ec"} Oct 31 12:32:51.785642 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:51.785315 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d4e20b11-4a0f-484f-930e-32f25592e75d","Type":"ContainerStarted","Data":"8b88086c5accc51d6b6ebb637e1f67801b80efb3b8e8c0d552da7fb6baf1d272"} Oct 31 12:32:51.785642 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:51.785341 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d4e20b11-4a0f-484f-930e-32f25592e75d","Type":"ContainerStarted","Data":"236ab70733c669980285174108a0885dfa07ba61181bfb8a6d3559c173a2afe5"} Oct 31 12:32:51.785642 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:51.785366 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d4e20b11-4a0f-484f-930e-32f25592e75d","Type":"ContainerStarted","Data":"b5d3f86488bc70b0121c292a433e2de62cb0a6f94d2f4d0dbef871002518da69"} Oct 31 12:32:51.785642 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:51.785389 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d4e20b11-4a0f-484f-930e-32f25592e75d","Type":"ContainerStarted","Data":"73d6ed959ed9228e14c54e88ad2ae26117228d04bdef103688948d32087ebcba"} Oct 31 12:32:51.785642 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:51.785410 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d4e20b11-4a0f-484f-930e-32f25592e75d","Type":"ContainerStarted","Data":"a8434da2556c64081408a4bf1dcbc20f797ccabf86b727b44e0301a2a3078110"} Oct 31 12:32:51.823274 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:51.822785 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.806204968 podStartE2EDuration="2.822766198s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.56123ms" isStatefulPod=false podCreationTimestamp="2026-10-31 12:32:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:51.819048993 +0000 UTC m=+18.302171894" watchObservedRunningTime="2026-10-31 12:32:51.822766198 +0000 UTC m=+18.305889099" Oct 31 12:32:52.849190 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:52.849152 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:53.050428 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:53.050391 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:53.252410 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:53.252373 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:55.095788 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:55.095758 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:32:55.165889 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:55.165844 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:32:59.638118 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:59.638073 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-668c5b5566-4scvp" podUID="f84bcfff-6360-4722-965f-8b3f152f2bdf" containerName="console" containerID="cri-o://a7619af8ff161fc7f9fbdabb4b43ead005820e717cb2100fd925ca8d6de5d2a2" gracePeriod=15 Oct 31 12:32:59.816308 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:59.816273 6841 generic.go:356] "Generic (PLEG): container finished" podID="f84bcfff-6360-4722-965f-8b3f152f2bdf" containerID="a7619af8ff161fc7f9fbdabb4b43ead005820e717cb2100fd925ca8d6de5d2a2" exitCode=0 Oct 31 12:32:59.816452 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:59.816344 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-668c5b5566-4scvp" event={"ID":"f84bcfff-6360-4722-965f-8b3f152f2bdf","Type":"ContainerDied","Data":"a7619af8ff161fc7f9fbdabb4b43ead005820e717cb2100fd925ca8d6de5d2a2"} Oct 31 12:32:59.881211 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:59.881189 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:32:59.979854 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:59.979827 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-oauth-config\") pod \"f84bcfff-6360-4722-965f-8b3f152f2bdf\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " Oct 31 12:32:59.980022 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:59.979905 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-oauth-serving-cert\") pod \"f84bcfff-6360-4722-965f-8b3f152f2bdf\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " Oct 31 12:32:59.980022 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:59.979925 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f84bcfff-6360-4722-965f-8b3f152f2bdf-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f84bcfff-6360-4722-965f-8b3f152f2bdf-tmp\") pod \"f84bcfff-6360-4722-965f-8b3f152f2bdf\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " Oct 31 12:32:59.980022 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:59.979974 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-service-ca\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-service-ca\") pod \"f84bcfff-6360-4722-965f-8b3f152f2bdf\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " Oct 31 12:32:59.980190 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:59.980033 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-config\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-config\") pod \"f84bcfff-6360-4722-965f-8b3f152f2bdf\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " Oct 31 12:32:59.980190 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:59.980061 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f84bcfff-6360-4722-965f-8b3f152f2bdf-kube-api-access-b8gsv\" (UniqueName: \"kubernetes.io/projected/f84bcfff-6360-4722-965f-8b3f152f2bdf-kube-api-access-b8gsv\") pod \"f84bcfff-6360-4722-965f-8b3f152f2bdf\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " Oct 31 12:32:59.980190 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:59.980098 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-trusted-ca-bundle\") pod \"f84bcfff-6360-4722-965f-8b3f152f2bdf\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " Oct 31 12:32:59.980190 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:59.980133 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-serving-cert\") pod \"f84bcfff-6360-4722-965f-8b3f152f2bdf\" (UID: \"f84bcfff-6360-4722-965f-8b3f152f2bdf\") " Oct 31 12:32:59.980384 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:59.980331 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f84bcfff-6360-4722-965f-8b3f152f2bdf-tmp" pod "f84bcfff-6360-4722-965f-8b3f152f2bdf" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:32:59.980467 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:59.980441 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-oauth-serving-cert" pod "f84bcfff-6360-4722-965f-8b3f152f2bdf" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:32:59.980515 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:59.980492 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-config" pod "f84bcfff-6360-4722-965f-8b3f152f2bdf" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:32:59.980609 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:59.980579 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-trusted-ca-bundle" pod "f84bcfff-6360-4722-965f-8b3f152f2bdf" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:32:59.980743 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:59.980712 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-service-ca" pod "f84bcfff-6360-4722-965f-8b3f152f2bdf" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:32:59.982269 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:59.982241 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-oauth-config" pod "f84bcfff-6360-4722-965f-8b3f152f2bdf" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:32:59.982424 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:59.982404 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-serving-cert" pod "f84bcfff-6360-4722-965f-8b3f152f2bdf" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:32:59.982485 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:32:59.982453 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f84bcfff-6360-4722-965f-8b3f152f2bdf-kube-api-access-b8gsv" pod "f84bcfff-6360-4722-965f-8b3f152f2bdf" (UID: "f84bcfff-6360-4722-965f-8b3f152f2bdf"). InnerVolumeSpecName "kube-api-access-b8gsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:33:00.081690 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:33:00.081658 6841 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-oauth-config\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:33:00.081690 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:33:00.081686 6841 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-oauth-serving-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:33:00.081690 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:33:00.081696 6841 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f84bcfff-6360-4722-965f-8b3f152f2bdf-tmp\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:33:00.081911 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:33:00.081708 6841 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:33:00.081911 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:33:00.081716 6841 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-config\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:33:00.081911 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:33:00.081725 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8gsv\" (UniqueName: \"kubernetes.io/projected/f84bcfff-6360-4722-965f-8b3f152f2bdf-kube-api-access-b8gsv\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:33:00.081911 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:33:00.081735 6841 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f84bcfff-6360-4722-965f-8b3f152f2bdf-trusted-ca-bundle\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:33:00.081911 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:33:00.081743 6841 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f84bcfff-6360-4722-965f-8b3f152f2bdf-console-serving-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:33:00.821293 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:33:00.821261 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-668c5b5566-4scvp" event={"ID":"f84bcfff-6360-4722-965f-8b3f152f2bdf","Type":"ContainerDied","Data":"2fad3db0afe9655284a9a370727d43c125b97c26b3561f4a17f974fc4544524e"} Oct 31 12:33:00.821719 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:33:00.821344 6841 scope.go:122] "RemoveContainer" containerID="a7619af8ff161fc7f9fbdabb4b43ead005820e717cb2100fd925ca8d6de5d2a2" Oct 31 12:33:00.821719 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:33:00.821276 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-668c5b5566-4scvp" Oct 31 12:33:00.848258 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:33:00.848232 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-668c5b5566-4scvp"] Oct 31 12:33:00.852619 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:33:00.852593 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-668c5b5566-4scvp"] Oct 31 12:33:01.639466 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:33:01.639434 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f84bcfff-6360-4722-965f-8b3f152f2bdf" path="/var/lib/kubelet/pods/f84bcfff-6360-4722-965f-8b3f152f2bdf/volumes" Oct 31 12:33:05.191217 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:33:05.191182 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:05.392803 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:33:05.392764 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:05.994112 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:33:05.994079 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7sf98" Oct 31 12:33:20.032320 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:33:20.032282 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:33:23.827601 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:33:23.827565 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:33:23.842715 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:33:23.842689 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:33:23.920061 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:33:23.920037 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 12:33:29.161621 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:33:29.161584 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:29.361656 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:33:29.361626 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:34.777101 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:33:34.777060 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:37.781481 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:33:37.781450 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:41.822375 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:33:41.822336 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:42.020655 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:33:42.020616 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:48.980004 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:33:48.979949 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:49.177377 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:33:49.177340 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:49.389416 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:33:49.389314 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:50.033163 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:33:50.033127 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:33:50.186879 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:33:50.186847 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:33:50.386095 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:33:50.385978 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:34:20.032451 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:34:20.032414 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:34:34.836477 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:34:34.836438 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:34:50.033292 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:34:50.033249 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:35:15.508261 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:35:15.508217 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:35:20.032776 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:35:20.032730 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:35:34.894552 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:35:34.894507 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:35:50.033167 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:35:50.033120 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:36:07.136145 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:07.136106 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:09.318286 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:09.318235 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:10.488121 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:10.488090 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:11.020826 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:11.020799 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckhr9r"] Oct 31 12:36:11.021146 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:11.021133 6841 cpu_manager.go:414] "containerMap: removing container" podUID="f84bcfff-6360-4722-965f-8b3f152f2bdf" containerName="console" Oct 31 12:36:11.021197 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:11.021149 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f84bcfff-6360-4722-965f-8b3f152f2bdf" containerName="console" Oct 31 12:36:11.021231 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:11.021223 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="f84bcfff-6360-4722-965f-8b3f152f2bdf" containerName="console" Oct 31 12:36:11.024311 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:11.024296 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckhr9r" Oct 31 12:36:11.029947 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:11.029922 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 31 12:36:11.029947 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:11.029937 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 31 12:36:11.030130 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:11.030037 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-9n4xz\"" Oct 31 12:36:11.034330 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:11.034311 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckhr9r"] Oct 31 12:36:11.120191 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:11.120152 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ec07c3a-7876-467b-8581-97538b73b62e-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckhr9r\" (UID: \"4ec07c3a-7876-467b-8581-97538b73b62e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckhr9r" Oct 31 12:36:11.120191 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:11.120193 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpbc7\" (UniqueName: \"kubernetes.io/projected/4ec07c3a-7876-467b-8581-97538b73b62e-kube-api-access-kpbc7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckhr9r\" (UID: \"4ec07c3a-7876-467b-8581-97538b73b62e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckhr9r" Oct 31 12:36:11.120393 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:11.120284 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ec07c3a-7876-467b-8581-97538b73b62e-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckhr9r\" (UID: \"4ec07c3a-7876-467b-8581-97538b73b62e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckhr9r" Oct 31 12:36:11.221184 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:11.221150 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ec07c3a-7876-467b-8581-97538b73b62e-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckhr9r\" (UID: \"4ec07c3a-7876-467b-8581-97538b73b62e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckhr9r" Oct 31 12:36:11.221184 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:11.221183 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpbc7\" (UniqueName: \"kubernetes.io/projected/4ec07c3a-7876-467b-8581-97538b73b62e-kube-api-access-kpbc7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckhr9r\" (UID: \"4ec07c3a-7876-467b-8581-97538b73b62e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckhr9r" Oct 31 12:36:11.221410 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:11.221207 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ec07c3a-7876-467b-8581-97538b73b62e-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckhr9r\" (UID: \"4ec07c3a-7876-467b-8581-97538b73b62e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckhr9r" Oct 31 12:36:11.221628 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:11.221599 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ec07c3a-7876-467b-8581-97538b73b62e-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckhr9r\" (UID: \"4ec07c3a-7876-467b-8581-97538b73b62e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckhr9r" Oct 31 12:36:11.221702 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:11.221649 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ec07c3a-7876-467b-8581-97538b73b62e-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckhr9r\" (UID: \"4ec07c3a-7876-467b-8581-97538b73b62e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckhr9r" Oct 31 12:36:11.233853 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:11.233828 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpbc7\" (UniqueName: \"kubernetes.io/projected/4ec07c3a-7876-467b-8581-97538b73b62e-kube-api-access-kpbc7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckhr9r\" (UID: \"4ec07c3a-7876-467b-8581-97538b73b62e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckhr9r" Oct 31 12:36:11.340046 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:11.339958 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckhr9r" Oct 31 12:36:11.613372 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:11.613304 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckhr9r"] Oct 31 12:36:11.616369 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:36:11.616337 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ec07c3a_7876_467b_8581_97538b73b62e.slice/crio-837b1fe5d6570232e8f09cbc9a99f004a42d947ab26ff70bfa938550bcf3c161 WatchSource:0}: Error finding container 837b1fe5d6570232e8f09cbc9a99f004a42d947ab26ff70bfa938550bcf3c161: Status 404 returned error can't find the container with id 837b1fe5d6570232e8f09cbc9a99f004a42d947ab26ff70bfa938550bcf3c161 Oct 31 12:36:11.618380 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:11.618364 6841 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 12:36:12.479968 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:12.479931 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckhr9r" event={"ID":"4ec07c3a-7876-467b-8581-97538b73b62e","Type":"ContainerStarted","Data":"837b1fe5d6570232e8f09cbc9a99f004a42d947ab26ff70bfa938550bcf3c161"} Oct 31 12:36:13.097478 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.097441 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n5cxt"] Oct 31 12:36:13.101065 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.101039 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n5cxt" Oct 31 12:36:13.116625 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.116603 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-d5dzq\"" Oct 31 12:36:13.135133 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.135105 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n5cxt"] Oct 31 12:36:13.225889 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.225854 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9qq2"] Oct 31 12:36:13.229261 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.229243 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9qq2" Oct 31 12:36:13.237062 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.237039 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7dc8d73a-2166-4536-82eb-2be71518770b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n5cxt\" (UID: \"7dc8d73a-2166-4536-82eb-2be71518770b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n5cxt" Oct 31 12:36:13.237174 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.237149 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jxnv\" (UniqueName: \"kubernetes.io/projected/7dc8d73a-2166-4536-82eb-2be71518770b-kube-api-access-9jxnv\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n5cxt\" (UID: \"7dc8d73a-2166-4536-82eb-2be71518770b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n5cxt" Oct 31 12:36:13.237269 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.237251 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7dc8d73a-2166-4536-82eb-2be71518770b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n5cxt\" (UID: \"7dc8d73a-2166-4536-82eb-2be71518770b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n5cxt" Oct 31 12:36:13.246897 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.246876 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9qq2"] Oct 31 12:36:13.247483 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.247461 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-728pp\"" Oct 31 12:36:13.307674 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.307645 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5tzkl"] Oct 31 12:36:13.311334 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.311309 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5tzkl" Oct 31 12:36:13.322436 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.322410 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5tzkl"] Oct 31 12:36:13.338687 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.338663 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jxnv\" (UniqueName: \"kubernetes.io/projected/7dc8d73a-2166-4536-82eb-2be71518770b-kube-api-access-9jxnv\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n5cxt\" (UID: \"7dc8d73a-2166-4536-82eb-2be71518770b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n5cxt" Oct 31 12:36:13.338828 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.338711 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsndj\" (UniqueName: \"kubernetes.io/projected/bf0d7afa-9a99-4e74-a103-8d68ac678193-kube-api-access-tsndj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9qq2\" (UID: \"bf0d7afa-9a99-4e74-a103-8d68ac678193\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9qq2" Oct 31 12:36:13.338892 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.338847 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7dc8d73a-2166-4536-82eb-2be71518770b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n5cxt\" (UID: \"7dc8d73a-2166-4536-82eb-2be71518770b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n5cxt" Oct 31 12:36:13.338892 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.338868 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf0d7afa-9a99-4e74-a103-8d68ac678193-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9qq2\" (UID: \"bf0d7afa-9a99-4e74-a103-8d68ac678193\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9qq2" Oct 31 12:36:13.338991 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.338897 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7dc8d73a-2166-4536-82eb-2be71518770b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n5cxt\" (UID: \"7dc8d73a-2166-4536-82eb-2be71518770b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n5cxt" Oct 31 12:36:13.338991 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.338958 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf0d7afa-9a99-4e74-a103-8d68ac678193-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9qq2\" (UID: \"bf0d7afa-9a99-4e74-a103-8d68ac678193\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9qq2" Oct 31 12:36:13.339216 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.339197 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7dc8d73a-2166-4536-82eb-2be71518770b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n5cxt\" (UID: \"7dc8d73a-2166-4536-82eb-2be71518770b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n5cxt" Oct 31 12:36:13.339259 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.339248 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7dc8d73a-2166-4536-82eb-2be71518770b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n5cxt\" (UID: \"7dc8d73a-2166-4536-82eb-2be71518770b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n5cxt" Oct 31 12:36:13.345505 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.345483 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-sj2hw\"" Oct 31 12:36:13.360341 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.360286 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jxnv\" (UniqueName: \"kubernetes.io/projected/7dc8d73a-2166-4536-82eb-2be71518770b-kube-api-access-9jxnv\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n5cxt\" (UID: \"7dc8d73a-2166-4536-82eb-2be71518770b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n5cxt" Oct 31 12:36:13.403674 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.403643 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c54vs9"] Oct 31 12:36:13.407218 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.407201 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c54vs9" Oct 31 12:36:13.412220 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.412201 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-t2k4q\"" Oct 31 12:36:13.416823 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.416800 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n5cxt" Oct 31 12:36:13.420098 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.419946 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c54vs9"] Oct 31 12:36:13.440101 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.440078 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97b09674-cbef-40a6-a291-3a03a2355042-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5tzkl\" (UID: \"97b09674-cbef-40a6-a291-3a03a2355042\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5tzkl" Oct 31 12:36:13.440194 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.440120 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97b09674-cbef-40a6-a291-3a03a2355042-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5tzkl\" (UID: \"97b09674-cbef-40a6-a291-3a03a2355042\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5tzkl" Oct 31 12:36:13.440194 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.440163 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsndj\" (UniqueName: \"kubernetes.io/projected/bf0d7afa-9a99-4e74-a103-8d68ac678193-kube-api-access-tsndj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9qq2\" (UID: \"bf0d7afa-9a99-4e74-a103-8d68ac678193\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9qq2" Oct 31 12:36:13.440303 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.440197 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf0d7afa-9a99-4e74-a103-8d68ac678193-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9qq2\" (UID: \"bf0d7afa-9a99-4e74-a103-8d68ac678193\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9qq2" Oct 31 12:36:13.440303 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.440282 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66mjv\" (UniqueName: \"kubernetes.io/projected/97b09674-cbef-40a6-a291-3a03a2355042-kube-api-access-66mjv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5tzkl\" (UID: \"97b09674-cbef-40a6-a291-3a03a2355042\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5tzkl" Oct 31 12:36:13.440416 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.440329 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf0d7afa-9a99-4e74-a103-8d68ac678193-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9qq2\" (UID: \"bf0d7afa-9a99-4e74-a103-8d68ac678193\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9qq2" Oct 31 12:36:13.440577 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.440557 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf0d7afa-9a99-4e74-a103-8d68ac678193-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9qq2\" (UID: \"bf0d7afa-9a99-4e74-a103-8d68ac678193\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9qq2" Oct 31 12:36:13.440697 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.440672 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf0d7afa-9a99-4e74-a103-8d68ac678193-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9qq2\" (UID: \"bf0d7afa-9a99-4e74-a103-8d68ac678193\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9qq2" Oct 31 12:36:13.455961 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.455920 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsndj\" (UniqueName: \"kubernetes.io/projected/bf0d7afa-9a99-4e74-a103-8d68ac678193-kube-api-access-tsndj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9qq2\" (UID: \"bf0d7afa-9a99-4e74-a103-8d68ac678193\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9qq2" Oct 31 12:36:13.541020 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.540965 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97b09674-cbef-40a6-a291-3a03a2355042-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5tzkl\" (UID: \"97b09674-cbef-40a6-a291-3a03a2355042\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5tzkl" Oct 31 12:36:13.541179 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.541033 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8be6efa-e396-43c7-a277-1f0cb8de7d83-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c54vs9\" (UID: \"e8be6efa-e396-43c7-a277-1f0cb8de7d83\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c54vs9" Oct 31 12:36:13.541179 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.541061 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97b09674-cbef-40a6-a291-3a03a2355042-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5tzkl\" (UID: \"97b09674-cbef-40a6-a291-3a03a2355042\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5tzkl" Oct 31 12:36:13.541179 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.541083 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8be6efa-e396-43c7-a277-1f0cb8de7d83-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c54vs9\" (UID: \"e8be6efa-e396-43c7-a277-1f0cb8de7d83\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c54vs9" Oct 31 12:36:13.541344 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.541282 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkvhx\" (UniqueName: \"kubernetes.io/projected/e8be6efa-e396-43c7-a277-1f0cb8de7d83-kube-api-access-jkvhx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c54vs9\" (UID: \"e8be6efa-e396-43c7-a277-1f0cb8de7d83\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c54vs9" Oct 31 12:36:13.541344 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.541333 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66mjv\" (UniqueName: \"kubernetes.io/projected/97b09674-cbef-40a6-a291-3a03a2355042-kube-api-access-66mjv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5tzkl\" (UID: \"97b09674-cbef-40a6-a291-3a03a2355042\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5tzkl" Oct 31 12:36:13.541443 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.541382 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97b09674-cbef-40a6-a291-3a03a2355042-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5tzkl\" (UID: \"97b09674-cbef-40a6-a291-3a03a2355042\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5tzkl" Oct 31 12:36:13.541443 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.541417 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97b09674-cbef-40a6-a291-3a03a2355042-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5tzkl\" (UID: \"97b09674-cbef-40a6-a291-3a03a2355042\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5tzkl" Oct 31 12:36:13.547298 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.547283 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9qq2" Oct 31 12:36:13.553997 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.553964 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66mjv\" (UniqueName: \"kubernetes.io/projected/97b09674-cbef-40a6-a291-3a03a2355042-kube-api-access-66mjv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5tzkl\" (UID: \"97b09674-cbef-40a6-a291-3a03a2355042\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5tzkl" Oct 31 12:36:13.629333 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.629273 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5tzkl" Oct 31 12:36:13.642400 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.642380 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8be6efa-e396-43c7-a277-1f0cb8de7d83-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c54vs9\" (UID: \"e8be6efa-e396-43c7-a277-1f0cb8de7d83\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c54vs9" Oct 31 12:36:13.642476 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.642417 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8be6efa-e396-43c7-a277-1f0cb8de7d83-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c54vs9\" (UID: \"e8be6efa-e396-43c7-a277-1f0cb8de7d83\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c54vs9" Oct 31 12:36:13.642476 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.642453 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkvhx\" (UniqueName: \"kubernetes.io/projected/e8be6efa-e396-43c7-a277-1f0cb8de7d83-kube-api-access-jkvhx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c54vs9\" (UID: \"e8be6efa-e396-43c7-a277-1f0cb8de7d83\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c54vs9" Oct 31 12:36:13.642731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.642713 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8be6efa-e396-43c7-a277-1f0cb8de7d83-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c54vs9\" (UID: \"e8be6efa-e396-43c7-a277-1f0cb8de7d83\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c54vs9" Oct 31 12:36:13.642791 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.642771 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8be6efa-e396-43c7-a277-1f0cb8de7d83-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c54vs9\" (UID: \"e8be6efa-e396-43c7-a277-1f0cb8de7d83\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c54vs9" Oct 31 12:36:13.655333 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.655310 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkvhx\" (UniqueName: \"kubernetes.io/projected/e8be6efa-e396-43c7-a277-1f0cb8de7d83-kube-api-access-jkvhx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c54vs9\" (UID: \"e8be6efa-e396-43c7-a277-1f0cb8de7d83\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c54vs9" Oct 31 12:36:13.707008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.706936 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n5cxt"] Oct 31 12:36:13.711118 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:36:13.711091 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc8d73a_2166_4536_82eb_2be71518770b.slice/crio-aed66d64a3345136e1c93207abe8f8dbe73b6b34681308c647a0d7afaab52ef1 WatchSource:0}: Error finding container aed66d64a3345136e1c93207abe8f8dbe73b6b34681308c647a0d7afaab52ef1: Status 404 returned error can't find the container with id aed66d64a3345136e1c93207abe8f8dbe73b6b34681308c647a0d7afaab52ef1 Oct 31 12:36:13.723933 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.723909 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c54vs9" Oct 31 12:36:13.837254 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.837224 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9qq2"] Oct 31 12:36:13.839921 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:36:13.839889 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf0d7afa_9a99_4e74_a103_8d68ac678193.slice/crio-ff1b61c38579c1dc02036d07c80ca87af7a3c83af06b29b8f027befbc52d1518 WatchSource:0}: Error finding container ff1b61c38579c1dc02036d07c80ca87af7a3c83af06b29b8f027befbc52d1518: Status 404 returned error can't find the container with id ff1b61c38579c1dc02036d07c80ca87af7a3c83af06b29b8f027befbc52d1518 Oct 31 12:36:13.907969 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.907896 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5tzkl"] Oct 31 12:36:13.911285 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:36:13.911259 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b09674_cbef_40a6_a291_3a03a2355042.slice/crio-8bde697d90c360fd7d6914d660789463a538611efdfa0f4cc7675d57c995e91e WatchSource:0}: Error finding container 8bde697d90c360fd7d6914d660789463a538611efdfa0f4cc7675d57c995e91e: Status 404 returned error can't find the container with id 8bde697d90c360fd7d6914d660789463a538611efdfa0f4cc7675d57c995e91e Oct 31 12:36:13.997235 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:13.997197 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c54vs9"] Oct 31 12:36:14.000573 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:36:14.000549 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8be6efa_e396_43c7_a277_1f0cb8de7d83.slice/crio-e88b270d8c95bdff00e5f5bc43080fa05264843bd73924dcc4f7bc64a430c056 WatchSource:0}: Error finding container e88b270d8c95bdff00e5f5bc43080fa05264843bd73924dcc4f7bc64a430c056: Status 404 returned error can't find the container with id e88b270d8c95bdff00e5f5bc43080fa05264843bd73924dcc4f7bc64a430c056 Oct 31 12:36:14.498510 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:14.498472 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c54vs9" event={"ID":"e8be6efa-e396-43c7-a277-1f0cb8de7d83","Type":"ContainerStarted","Data":"e88b270d8c95bdff00e5f5bc43080fa05264843bd73924dcc4f7bc64a430c056"} Oct 31 12:36:14.501038 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:14.501004 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9qq2" event={"ID":"bf0d7afa-9a99-4e74-a103-8d68ac678193","Type":"ContainerStarted","Data":"ff1b61c38579c1dc02036d07c80ca87af7a3c83af06b29b8f027befbc52d1518"} Oct 31 12:36:14.504568 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:14.504537 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n5cxt" event={"ID":"7dc8d73a-2166-4536-82eb-2be71518770b","Type":"ContainerStarted","Data":"aed66d64a3345136e1c93207abe8f8dbe73b6b34681308c647a0d7afaab52ef1"} Oct 31 12:36:14.507467 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:14.507439 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5tzkl" event={"ID":"97b09674-cbef-40a6-a291-3a03a2355042","Type":"ContainerStarted","Data":"8bde697d90c360fd7d6914d660789463a538611efdfa0f4cc7675d57c995e91e"} Oct 31 12:36:14.551487 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:14.551412 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:14.774315 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:14.774191 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:14.814014 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:14.812446 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfbdw"] Oct 31 12:36:14.816184 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:14.816159 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfbdw" Oct 31 12:36:14.823857 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:14.823833 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-brjnq\"" Oct 31 12:36:14.829481 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:14.829455 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfbdw"] Oct 31 12:36:14.954555 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:14.954506 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa93e723-8c4e-4530-999f-54b461e2f308-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfbdw\" (UID: \"aa93e723-8c4e-4530-999f-54b461e2f308\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfbdw" Oct 31 12:36:14.954734 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:14.954581 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa93e723-8c4e-4530-999f-54b461e2f308-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfbdw\" (UID: \"aa93e723-8c4e-4530-999f-54b461e2f308\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfbdw" Oct 31 12:36:14.954734 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:14.954693 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24x5g\" (UniqueName: \"kubernetes.io/projected/aa93e723-8c4e-4530-999f-54b461e2f308-kube-api-access-24x5g\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfbdw\" (UID: \"aa93e723-8c4e-4530-999f-54b461e2f308\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfbdw" Oct 31 12:36:15.056538 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:15.055736 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24x5g\" (UniqueName: \"kubernetes.io/projected/aa93e723-8c4e-4530-999f-54b461e2f308-kube-api-access-24x5g\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfbdw\" (UID: \"aa93e723-8c4e-4530-999f-54b461e2f308\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfbdw" Oct 31 12:36:15.056538 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:15.055823 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa93e723-8c4e-4530-999f-54b461e2f308-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfbdw\" (UID: \"aa93e723-8c4e-4530-999f-54b461e2f308\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfbdw" Oct 31 12:36:15.056538 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:15.055870 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa93e723-8c4e-4530-999f-54b461e2f308-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfbdw\" (UID: \"aa93e723-8c4e-4530-999f-54b461e2f308\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfbdw" Oct 31 12:36:15.056538 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:15.056440 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa93e723-8c4e-4530-999f-54b461e2f308-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfbdw\" (UID: \"aa93e723-8c4e-4530-999f-54b461e2f308\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfbdw" Oct 31 12:36:15.056931 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:15.056622 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa93e723-8c4e-4530-999f-54b461e2f308-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfbdw\" (UID: \"aa93e723-8c4e-4530-999f-54b461e2f308\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfbdw" Oct 31 12:36:15.080572 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:15.080530 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24x5g\" (UniqueName: \"kubernetes.io/projected/aa93e723-8c4e-4530-999f-54b461e2f308-kube-api-access-24x5g\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfbdw\" (UID: \"aa93e723-8c4e-4530-999f-54b461e2f308\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfbdw" Oct 31 12:36:15.141067 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:15.141023 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfbdw" Oct 31 12:36:15.441215 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:15.441132 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfbdw"] Oct 31 12:36:15.446615 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:36:15.446585 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa93e723_8c4e_4530_999f_54b461e2f308.slice/crio-ce6381b401abd5eefe186597c9d318c981eae4b97071e4e11ae63d9a8872eeea WatchSource:0}: Error finding container ce6381b401abd5eefe186597c9d318c981eae4b97071e4e11ae63d9a8872eeea: Status 404 returned error can't find the container with id ce6381b401abd5eefe186597c9d318c981eae4b97071e4e11ae63d9a8872eeea Oct 31 12:36:15.520062 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:15.520023 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfbdw" event={"ID":"aa93e723-8c4e-4530-999f-54b461e2f308","Type":"ContainerStarted","Data":"ce6381b401abd5eefe186597c9d318c981eae4b97071e4e11ae63d9a8872eeea"} Oct 31 12:36:16.001662 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:16.001568 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:16.239732 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:16.239646 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:18.957660 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:18.957580 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:20.050235 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:36:20.050165 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:36:25.583409 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:25.583380 6841 generic.go:356] "Generic (PLEG): container finished" podID="bf0d7afa-9a99-4e74-a103-8d68ac678193" containerID="ed1c1a8c97063b419bb69b46767b02bc1777f9e326b689732ae57d5182cdc308" exitCode=0 Oct 31 12:36:25.583819 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:25.583450 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9qq2" event={"ID":"bf0d7afa-9a99-4e74-a103-8d68ac678193","Type":"ContainerDied","Data":"ed1c1a8c97063b419bb69b46767b02bc1777f9e326b689732ae57d5182cdc308"} Oct 31 12:36:25.585325 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:25.585299 6841 generic.go:356] "Generic (PLEG): container finished" podID="7dc8d73a-2166-4536-82eb-2be71518770b" containerID="9f38b9579a203dac4c6158fdf1f0b4438515b1fcce1aac9963edc6192fcfe7b4" exitCode=0 Oct 31 12:36:25.585416 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:25.585325 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n5cxt" event={"ID":"7dc8d73a-2166-4536-82eb-2be71518770b","Type":"ContainerDied","Data":"9f38b9579a203dac4c6158fdf1f0b4438515b1fcce1aac9963edc6192fcfe7b4"} Oct 31 12:36:26.594097 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:26.594069 6841 generic.go:356] "Generic (PLEG): container finished" podID="e8be6efa-e396-43c7-a277-1f0cb8de7d83" containerID="8219761b273f49a39399f938a3a016e12842119dd362552de3a57ac9128c8df1" exitCode=0 Oct 31 12:36:26.594478 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:26.594106 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c54vs9" event={"ID":"e8be6efa-e396-43c7-a277-1f0cb8de7d83","Type":"ContainerDied","Data":"8219761b273f49a39399f938a3a016e12842119dd362552de3a57ac9128c8df1"} Oct 31 12:36:26.595896 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:26.595872 6841 generic.go:356] "Generic (PLEG): container finished" podID="aa93e723-8c4e-4530-999f-54b461e2f308" containerID="3763eaa1f7979c79dbcd9fbcc920540b0564ab18f355605bc68da6532747e43a" exitCode=0 Oct 31 12:36:26.596012 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:26.595946 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfbdw" event={"ID":"aa93e723-8c4e-4530-999f-54b461e2f308","Type":"ContainerDied","Data":"3763eaa1f7979c79dbcd9fbcc920540b0564ab18f355605bc68da6532747e43a"} Oct 31 12:36:26.597750 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:26.597726 6841 generic.go:356] "Generic (PLEG): container finished" podID="97b09674-cbef-40a6-a291-3a03a2355042" containerID="66990918d5437225d849dcb3df4c8fee5f7d48860547b30e38bbcdbfc20474c8" exitCode=0 Oct 31 12:36:26.597840 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:26.597794 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5tzkl" event={"ID":"97b09674-cbef-40a6-a291-3a03a2355042","Type":"ContainerDied","Data":"66990918d5437225d849dcb3df4c8fee5f7d48860547b30e38bbcdbfc20474c8"} Oct 31 12:36:27.603591 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:27.603562 6841 generic.go:356] "Generic (PLEG): container finished" podID="4ec07c3a-7876-467b-8581-97538b73b62e" containerID="06e367410b991ba70a17451f9abd0564f3e46c86c51a9e1f7e2afbab0dcb9b74" exitCode=0 Oct 31 12:36:27.604083 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:27.603603 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckhr9r" event={"ID":"4ec07c3a-7876-467b-8581-97538b73b62e","Type":"ContainerDied","Data":"06e367410b991ba70a17451f9abd0564f3e46c86c51a9e1f7e2afbab0dcb9b74"} Oct 31 12:36:27.605714 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:27.605691 6841 generic.go:356] "Generic (PLEG): container finished" podID="bf0d7afa-9a99-4e74-a103-8d68ac678193" containerID="27d927d3acaa6877e636baea87cd721df35804a180e5de121f78ed60b661ef01" exitCode=0 Oct 31 12:36:27.605797 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:27.605759 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9qq2" event={"ID":"bf0d7afa-9a99-4e74-a103-8d68ac678193","Type":"ContainerDied","Data":"27d927d3acaa6877e636baea87cd721df35804a180e5de121f78ed60b661ef01"} Oct 31 12:36:27.608108 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:27.608090 6841 generic.go:356] "Generic (PLEG): container finished" podID="7dc8d73a-2166-4536-82eb-2be71518770b" containerID="05fa6ada1f7da2bbb07e70de2e4feefd8c663ef418ac889c17555c9407e44aa9" exitCode=0 Oct 31 12:36:27.608215 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:27.608112 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n5cxt" event={"ID":"7dc8d73a-2166-4536-82eb-2be71518770b","Type":"ContainerDied","Data":"05fa6ada1f7da2bbb07e70de2e4feefd8c663ef418ac889c17555c9407e44aa9"} Oct 31 12:36:28.615285 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:28.615246 6841 generic.go:356] "Generic (PLEG): container finished" podID="e8be6efa-e396-43c7-a277-1f0cb8de7d83" containerID="219da1e7283213591adba6283ec8bb358db922a34c14e7cdc70e23753c4a66b1" exitCode=0 Oct 31 12:36:28.615756 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:28.615373 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c54vs9" event={"ID":"e8be6efa-e396-43c7-a277-1f0cb8de7d83","Type":"ContainerDied","Data":"219da1e7283213591adba6283ec8bb358db922a34c14e7cdc70e23753c4a66b1"} Oct 31 12:36:29.623056 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:29.623015 6841 generic.go:356] "Generic (PLEG): container finished" podID="97b09674-cbef-40a6-a291-3a03a2355042" containerID="e5c4bd280420057a02aa1aa4830bf1ab519cbf42b23819aa14ba7afd3adbf2bf" exitCode=0 Oct 31 12:36:29.623489 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:29.623070 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5tzkl" event={"ID":"97b09674-cbef-40a6-a291-3a03a2355042","Type":"ContainerDied","Data":"e5c4bd280420057a02aa1aa4830bf1ab519cbf42b23819aa14ba7afd3adbf2bf"} Oct 31 12:36:32.649127 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:32.648261 6841 generic.go:356] "Generic (PLEG): container finished" podID="4ec07c3a-7876-467b-8581-97538b73b62e" containerID="bfbd66df74e3d92469d4f45fd136e5221902498dd468a3cc0a023d1172426e88" exitCode=0 Oct 31 12:36:32.649127 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:32.648375 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckhr9r" event={"ID":"4ec07c3a-7876-467b-8581-97538b73b62e","Type":"ContainerDied","Data":"bfbd66df74e3d92469d4f45fd136e5221902498dd468a3cc0a023d1172426e88"} Oct 31 12:36:33.658311 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:33.658273 6841 generic.go:356] "Generic (PLEG): container finished" podID="aa93e723-8c4e-4530-999f-54b461e2f308" containerID="0bf66b37fff9760caee6bc0eee6e6d1d46310603ef8f3218e697c41dd3c9f098" exitCode=0 Oct 31 12:36:33.658786 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:33.658356 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfbdw" event={"ID":"aa93e723-8c4e-4530-999f-54b461e2f308","Type":"ContainerDied","Data":"0bf66b37fff9760caee6bc0eee6e6d1d46310603ef8f3218e697c41dd3c9f098"} Oct 31 12:36:34.953242 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:34.953200 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:42.360166 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:42.360125 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:42.570945 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:42.570902 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:42.716075 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:42.715968 6841 generic.go:356] "Generic (PLEG): container finished" podID="7dc8d73a-2166-4536-82eb-2be71518770b" containerID="1ae09aee89be130724b51554096351777f06cc5701016b34fad013e41a98110a" exitCode=0 Oct 31 12:36:42.716075 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:42.716028 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n5cxt" event={"ID":"7dc8d73a-2166-4536-82eb-2be71518770b","Type":"ContainerDied","Data":"1ae09aee89be130724b51554096351777f06cc5701016b34fad013e41a98110a"} Oct 31 12:36:42.718739 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:42.718620 6841 generic.go:356] "Generic (PLEG): container finished" podID="97b09674-cbef-40a6-a291-3a03a2355042" containerID="2c0454c0dbcf12bf09bda488a269bf6487f72f5e44b7ea0305d7cc052d92852f" exitCode=0 Oct 31 12:36:42.718739 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:42.718660 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5tzkl" event={"ID":"97b09674-cbef-40a6-a291-3a03a2355042","Type":"ContainerDied","Data":"2c0454c0dbcf12bf09bda488a269bf6487f72f5e44b7ea0305d7cc052d92852f"} Oct 31 12:36:42.721501 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:42.721480 6841 generic.go:356] "Generic (PLEG): container finished" podID="e8be6efa-e396-43c7-a277-1f0cb8de7d83" containerID="f27fdf897dccb619f44ad11d975cc44248ed3191fa210f9d2085216505428c8f" exitCode=0 Oct 31 12:36:42.721609 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:42.721539 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c54vs9" event={"ID":"e8be6efa-e396-43c7-a277-1f0cb8de7d83","Type":"ContainerDied","Data":"f27fdf897dccb619f44ad11d975cc44248ed3191fa210f9d2085216505428c8f"} Oct 31 12:36:42.724011 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:42.723974 6841 generic.go:356] "Generic (PLEG): container finished" podID="bf0d7afa-9a99-4e74-a103-8d68ac678193" containerID="290c74f56291c111a188e15813828c4de9dfd68c92d0c3cf80134d11080808d2" exitCode=0 Oct 31 12:36:42.724100 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:42.724025 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9qq2" event={"ID":"bf0d7afa-9a99-4e74-a103-8d68ac678193","Type":"ContainerDied","Data":"290c74f56291c111a188e15813828c4de9dfd68c92d0c3cf80134d11080808d2"} Oct 31 12:36:43.134561 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.134486 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:43.147525 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.147492 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-c97kk"] Oct 31 12:36:43.151572 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.151549 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-c97kk" Oct 31 12:36:43.166375 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.166330 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 12:36:43.171174 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.171140 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 12:36:43.176492 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.176468 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 12:36:43.176671 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.176654 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 12:36:43.176794 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.176764 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-96llj\"" Oct 31 12:36:43.181743 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.181726 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 12:36:43.229147 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.229069 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8df744a4-e981-4e9f-a400-b6b7ebf91aed-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c44b469d4-c97kk\" (UID: \"8df744a4-e981-4e9f-a400-b6b7ebf91aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-c97kk" Oct 31 12:36:43.229147 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.229119 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8df744a4-e981-4e9f-a400-b6b7ebf91aed-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c44b469d4-c97kk\" (UID: \"8df744a4-e981-4e9f-a400-b6b7ebf91aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-c97kk" Oct 31 12:36:43.229354 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.229217 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8df744a4-e981-4e9f-a400-b6b7ebf91aed-manifests\") pod \"opendatahub-operator-controller-manager-5c44b469d4-c97kk\" (UID: \"8df744a4-e981-4e9f-a400-b6b7ebf91aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-c97kk" Oct 31 12:36:43.229354 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.229314 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97dvb\" (UniqueName: \"kubernetes.io/projected/8df744a4-e981-4e9f-a400-b6b7ebf91aed-kube-api-access-97dvb\") pod \"opendatahub-operator-controller-manager-5c44b469d4-c97kk\" (UID: \"8df744a4-e981-4e9f-a400-b6b7ebf91aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-c97kk" Oct 31 12:36:43.229480 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.229454 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8df744a4-e981-4e9f-a400-b6b7ebf91aed-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c44b469d4-c97kk\" (UID: \"8df744a4-e981-4e9f-a400-b6b7ebf91aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-c97kk" Oct 31 12:36:43.229538 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.229508 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8df744a4-e981-4e9f-a400-b6b7ebf91aed-tmp\") pod \"opendatahub-operator-controller-manager-5c44b469d4-c97kk\" (UID: \"8df744a4-e981-4e9f-a400-b6b7ebf91aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-c97kk" Oct 31 12:36:43.330752 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.330720 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97dvb\" (UniqueName: \"kubernetes.io/projected/8df744a4-e981-4e9f-a400-b6b7ebf91aed-kube-api-access-97dvb\") pod \"opendatahub-operator-controller-manager-5c44b469d4-c97kk\" (UID: \"8df744a4-e981-4e9f-a400-b6b7ebf91aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-c97kk" Oct 31 12:36:43.330939 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.330765 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8df744a4-e981-4e9f-a400-b6b7ebf91aed-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c44b469d4-c97kk\" (UID: \"8df744a4-e981-4e9f-a400-b6b7ebf91aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-c97kk" Oct 31 12:36:43.330939 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.330801 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8df744a4-e981-4e9f-a400-b6b7ebf91aed-tmp\") pod \"opendatahub-operator-controller-manager-5c44b469d4-c97kk\" (UID: \"8df744a4-e981-4e9f-a400-b6b7ebf91aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-c97kk" Oct 31 12:36:43.330939 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.330848 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8df744a4-e981-4e9f-a400-b6b7ebf91aed-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c44b469d4-c97kk\" (UID: \"8df744a4-e981-4e9f-a400-b6b7ebf91aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-c97kk" Oct 31 12:36:43.330939 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.330928 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8df744a4-e981-4e9f-a400-b6b7ebf91aed-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c44b469d4-c97kk\" (UID: \"8df744a4-e981-4e9f-a400-b6b7ebf91aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-c97kk" Oct 31 12:36:43.331195 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.330978 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8df744a4-e981-4e9f-a400-b6b7ebf91aed-manifests\") pod \"opendatahub-operator-controller-manager-5c44b469d4-c97kk\" (UID: \"8df744a4-e981-4e9f-a400-b6b7ebf91aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-c97kk" Oct 31 12:36:43.331587 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.331546 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8df744a4-e981-4e9f-a400-b6b7ebf91aed-manifests\") pod \"opendatahub-operator-controller-manager-5c44b469d4-c97kk\" (UID: \"8df744a4-e981-4e9f-a400-b6b7ebf91aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-c97kk" Oct 31 12:36:43.333536 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.333508 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8df744a4-e981-4e9f-a400-b6b7ebf91aed-tmp\") pod \"opendatahub-operator-controller-manager-5c44b469d4-c97kk\" (UID: \"8df744a4-e981-4e9f-a400-b6b7ebf91aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-c97kk" Oct 31 12:36:43.333712 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.333694 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8df744a4-e981-4e9f-a400-b6b7ebf91aed-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c44b469d4-c97kk\" (UID: \"8df744a4-e981-4e9f-a400-b6b7ebf91aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-c97kk" Oct 31 12:36:43.333893 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.333863 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8df744a4-e981-4e9f-a400-b6b7ebf91aed-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c44b469d4-c97kk\" (UID: \"8df744a4-e981-4e9f-a400-b6b7ebf91aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-c97kk" Oct 31 12:36:43.334275 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.334251 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8df744a4-e981-4e9f-a400-b6b7ebf91aed-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c44b469d4-c97kk\" (UID: \"8df744a4-e981-4e9f-a400-b6b7ebf91aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-c97kk" Oct 31 12:36:43.437712 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.437648 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97dvb\" (UniqueName: \"kubernetes.io/projected/8df744a4-e981-4e9f-a400-b6b7ebf91aed-kube-api-access-97dvb\") pod \"opendatahub-operator-controller-manager-5c44b469d4-c97kk\" (UID: \"8df744a4-e981-4e9f-a400-b6b7ebf91aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-c97kk" Oct 31 12:36:43.470544 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.470506 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-c97kk" Oct 31 12:36:43.540118 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.540059 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-c97kk"] Oct 31 12:36:43.715522 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.715460 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:43.883413 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.883383 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c54vs9" Oct 31 12:36:43.941362 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.939782 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8be6efa-e396-43c7-a277-1f0cb8de7d83-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8be6efa-e396-43c7-a277-1f0cb8de7d83-bundle\") pod \"e8be6efa-e396-43c7-a277-1f0cb8de7d83\" (UID: \"e8be6efa-e396-43c7-a277-1f0cb8de7d83\") " Oct 31 12:36:43.941362 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.939870 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8be6efa-e396-43c7-a277-1f0cb8de7d83-kube-api-access-jkvhx\" (UniqueName: \"kubernetes.io/projected/e8be6efa-e396-43c7-a277-1f0cb8de7d83-kube-api-access-jkvhx\") pod \"e8be6efa-e396-43c7-a277-1f0cb8de7d83\" (UID: \"e8be6efa-e396-43c7-a277-1f0cb8de7d83\") " Oct 31 12:36:43.941362 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.939945 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8be6efa-e396-43c7-a277-1f0cb8de7d83-util\" (UniqueName: \"kubernetes.io/empty-dir/e8be6efa-e396-43c7-a277-1f0cb8de7d83-util\") pod \"e8be6efa-e396-43c7-a277-1f0cb8de7d83\" (UID: \"e8be6efa-e396-43c7-a277-1f0cb8de7d83\") " Oct 31 12:36:43.941362 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.940914 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8be6efa-e396-43c7-a277-1f0cb8de7d83-bundle" pod "e8be6efa-e396-43c7-a277-1f0cb8de7d83" (UID: "e8be6efa-e396-43c7-a277-1f0cb8de7d83"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:36:43.941801 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.941597 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:43.944335 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.944300 6841 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8be6efa-e396-43c7-a277-1f0cb8de7d83-bundle\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:36:43.948385 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.948337 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8be6efa-e396-43c7-a277-1f0cb8de7d83-util" pod "e8be6efa-e396-43c7-a277-1f0cb8de7d83" (UID: "e8be6efa-e396-43c7-a277-1f0cb8de7d83"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:36:43.955682 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.955651 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9qq2" Oct 31 12:36:43.955842 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.955816 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8be6efa-e396-43c7-a277-1f0cb8de7d83-kube-api-access-jkvhx" pod "e8be6efa-e396-43c7-a277-1f0cb8de7d83" (UID: "e8be6efa-e396-43c7-a277-1f0cb8de7d83"). InnerVolumeSpecName "kube-api-access-jkvhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:36:43.991254 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.991220 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5tzkl" Oct 31 12:36:43.992217 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.992133 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-c97kk"] Oct 31 12:36:43.993752 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:43.993680 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n5cxt" Oct 31 12:36:44.001302 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:36:44.001259 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8df744a4_e981_4e9f_a400_b6b7ebf91aed.slice/crio-4552b8a1aee732b7100edb9a0f82ed6ca815b897086e0f91f9a3f06dbabaefc5 WatchSource:0}: Error finding container 4552b8a1aee732b7100edb9a0f82ed6ca815b897086e0f91f9a3f06dbabaefc5: Status 404 returned error can't find the container with id 4552b8a1aee732b7100edb9a0f82ed6ca815b897086e0f91f9a3f06dbabaefc5 Oct 31 12:36:44.045253 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.045220 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc8d73a-2166-4536-82eb-2be71518770b-util\" (UniqueName: \"kubernetes.io/empty-dir/7dc8d73a-2166-4536-82eb-2be71518770b-util\") pod \"7dc8d73a-2166-4536-82eb-2be71518770b\" (UID: \"7dc8d73a-2166-4536-82eb-2be71518770b\") " Oct 31 12:36:44.045429 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.045261 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf0d7afa-9a99-4e74-a103-8d68ac678193-util\" (UniqueName: \"kubernetes.io/empty-dir/bf0d7afa-9a99-4e74-a103-8d68ac678193-util\") pod \"bf0d7afa-9a99-4e74-a103-8d68ac678193\" (UID: \"bf0d7afa-9a99-4e74-a103-8d68ac678193\") " Oct 31 12:36:44.045429 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.045286 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97b09674-cbef-40a6-a291-3a03a2355042-kube-api-access-66mjv\" (UniqueName: \"kubernetes.io/projected/97b09674-cbef-40a6-a291-3a03a2355042-kube-api-access-66mjv\") pod \"97b09674-cbef-40a6-a291-3a03a2355042\" (UID: \"97b09674-cbef-40a6-a291-3a03a2355042\") " Oct 31 12:36:44.045429 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.045315 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97b09674-cbef-40a6-a291-3a03a2355042-bundle\" (UniqueName: \"kubernetes.io/empty-dir/97b09674-cbef-40a6-a291-3a03a2355042-bundle\") pod \"97b09674-cbef-40a6-a291-3a03a2355042\" (UID: \"97b09674-cbef-40a6-a291-3a03a2355042\") " Oct 31 12:36:44.045429 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.045410 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf0d7afa-9a99-4e74-a103-8d68ac678193-bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf0d7afa-9a99-4e74-a103-8d68ac678193-bundle\") pod \"bf0d7afa-9a99-4e74-a103-8d68ac678193\" (UID: \"bf0d7afa-9a99-4e74-a103-8d68ac678193\") " Oct 31 12:36:44.045652 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.045439 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf0d7afa-9a99-4e74-a103-8d68ac678193-kube-api-access-tsndj\" (UniqueName: \"kubernetes.io/projected/bf0d7afa-9a99-4e74-a103-8d68ac678193-kube-api-access-tsndj\") pod \"bf0d7afa-9a99-4e74-a103-8d68ac678193\" (UID: \"bf0d7afa-9a99-4e74-a103-8d68ac678193\") " Oct 31 12:36:44.045652 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.045475 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dc8d73a-2166-4536-82eb-2be71518770b-kube-api-access-9jxnv\" (UniqueName: \"kubernetes.io/projected/7dc8d73a-2166-4536-82eb-2be71518770b-kube-api-access-9jxnv\") pod \"7dc8d73a-2166-4536-82eb-2be71518770b\" (UID: \"7dc8d73a-2166-4536-82eb-2be71518770b\") " Oct 31 12:36:44.045652 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.045517 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97b09674-cbef-40a6-a291-3a03a2355042-util\" (UniqueName: \"kubernetes.io/empty-dir/97b09674-cbef-40a6-a291-3a03a2355042-util\") pod \"97b09674-cbef-40a6-a291-3a03a2355042\" (UID: \"97b09674-cbef-40a6-a291-3a03a2355042\") " Oct 31 12:36:44.045652 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.045600 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc8d73a-2166-4536-82eb-2be71518770b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7dc8d73a-2166-4536-82eb-2be71518770b-bundle\") pod \"7dc8d73a-2166-4536-82eb-2be71518770b\" (UID: \"7dc8d73a-2166-4536-82eb-2be71518770b\") " Oct 31 12:36:44.045853 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.045832 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkvhx\" (UniqueName: \"kubernetes.io/projected/e8be6efa-e396-43c7-a277-1f0cb8de7d83-kube-api-access-jkvhx\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:36:44.045853 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.045849 6841 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8be6efa-e396-43c7-a277-1f0cb8de7d83-util\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:36:44.046675 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.046214 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf0d7afa-9a99-4e74-a103-8d68ac678193-bundle" pod "bf0d7afa-9a99-4e74-a103-8d68ac678193" (UID: "bf0d7afa-9a99-4e74-a103-8d68ac678193"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:36:44.046675 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.046578 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc8d73a-2166-4536-82eb-2be71518770b-bundle" pod "7dc8d73a-2166-4536-82eb-2be71518770b" (UID: "7dc8d73a-2166-4536-82eb-2be71518770b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:36:44.046955 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.046924 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97b09674-cbef-40a6-a291-3a03a2355042-bundle" pod "97b09674-cbef-40a6-a291-3a03a2355042" (UID: "97b09674-cbef-40a6-a291-3a03a2355042"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:36:44.048335 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.048306 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97b09674-cbef-40a6-a291-3a03a2355042-kube-api-access-66mjv" pod "97b09674-cbef-40a6-a291-3a03a2355042" (UID: "97b09674-cbef-40a6-a291-3a03a2355042"). InnerVolumeSpecName "kube-api-access-66mjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:36:44.048533 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.048508 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf0d7afa-9a99-4e74-a103-8d68ac678193-kube-api-access-tsndj" pod "bf0d7afa-9a99-4e74-a103-8d68ac678193" (UID: "bf0d7afa-9a99-4e74-a103-8d68ac678193"). InnerVolumeSpecName "kube-api-access-tsndj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:36:44.049556 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.049520 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc8d73a-2166-4536-82eb-2be71518770b-kube-api-access-9jxnv" pod "7dc8d73a-2166-4536-82eb-2be71518770b" (UID: "7dc8d73a-2166-4536-82eb-2be71518770b"). InnerVolumeSpecName "kube-api-access-9jxnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:36:44.052015 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.051957 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf0d7afa-9a99-4e74-a103-8d68ac678193-util" pod "bf0d7afa-9a99-4e74-a103-8d68ac678193" (UID: "bf0d7afa-9a99-4e74-a103-8d68ac678193"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:36:44.052015 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.052003 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc8d73a-2166-4536-82eb-2be71518770b-util" pod "7dc8d73a-2166-4536-82eb-2be71518770b" (UID: "7dc8d73a-2166-4536-82eb-2be71518770b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:36:44.053187 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.053165 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97b09674-cbef-40a6-a291-3a03a2355042-util" pod "97b09674-cbef-40a6-a291-3a03a2355042" (UID: "97b09674-cbef-40a6-a291-3a03a2355042"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:36:44.146323 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.146286 6841 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf0d7afa-9a99-4e74-a103-8d68ac678193-bundle\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:36:44.146323 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.146321 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsndj\" (UniqueName: \"kubernetes.io/projected/bf0d7afa-9a99-4e74-a103-8d68ac678193-kube-api-access-tsndj\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:36:44.146552 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.146333 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jxnv\" (UniqueName: \"kubernetes.io/projected/7dc8d73a-2166-4536-82eb-2be71518770b-kube-api-access-9jxnv\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:36:44.146552 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.146344 6841 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97b09674-cbef-40a6-a291-3a03a2355042-util\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:36:44.146552 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.146357 6841 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7dc8d73a-2166-4536-82eb-2be71518770b-bundle\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:36:44.146552 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.146369 6841 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7dc8d73a-2166-4536-82eb-2be71518770b-util\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:36:44.146552 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.146381 6841 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf0d7afa-9a99-4e74-a103-8d68ac678193-util\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:36:44.146552 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.146393 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66mjv\" (UniqueName: \"kubernetes.io/projected/97b09674-cbef-40a6-a291-3a03a2355042-kube-api-access-66mjv\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:36:44.146552 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.146401 6841 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97b09674-cbef-40a6-a291-3a03a2355042-bundle\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:36:44.161290 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.161256 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:44.412717 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.412611 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:44.641512 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.641391 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:44.740825 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.740792 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckhr9r" event={"ID":"4ec07c3a-7876-467b-8581-97538b73b62e","Type":"ContainerStarted","Data":"9fb9468014f99290a05db6366b4340bd4e3d9dbad7618097a7b932eaa2d74732"} Oct 31 12:36:44.745525 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.745499 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c54vs9" event={"ID":"e8be6efa-e396-43c7-a277-1f0cb8de7d83","Type":"ContainerDied","Data":"e88b270d8c95bdff00e5f5bc43080fa05264843bd73924dcc4f7bc64a430c056"} Oct 31 12:36:44.745663 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.745543 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e88b270d8c95bdff00e5f5bc43080fa05264843bd73924dcc4f7bc64a430c056" Oct 31 12:36:44.745663 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.745615 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c54vs9" Oct 31 12:36:44.749413 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.749388 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9qq2" Oct 31 12:36:44.749413 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.749403 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9qq2" event={"ID":"bf0d7afa-9a99-4e74-a103-8d68ac678193","Type":"ContainerDied","Data":"ff1b61c38579c1dc02036d07c80ca87af7a3c83af06b29b8f027befbc52d1518"} Oct 31 12:36:44.749598 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.749446 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff1b61c38579c1dc02036d07c80ca87af7a3c83af06b29b8f027befbc52d1518" Oct 31 12:36:44.758009 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.756769 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n5cxt" Oct 31 12:36:44.758009 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.756790 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7n5cxt" event={"ID":"7dc8d73a-2166-4536-82eb-2be71518770b","Type":"ContainerDied","Data":"aed66d64a3345136e1c93207abe8f8dbe73b6b34681308c647a0d7afaab52ef1"} Oct 31 12:36:44.758009 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.756833 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aed66d64a3345136e1c93207abe8f8dbe73b6b34681308c647a0d7afaab52ef1" Oct 31 12:36:44.761522 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.761498 6841 generic.go:356] "Generic (PLEG): container finished" podID="aa93e723-8c4e-4530-999f-54b461e2f308" containerID="feb404a67aac9cae99f5446e29ad3b6552013827a00a59dec946cc5f499e0dbc" exitCode=0 Oct 31 12:36:44.761703 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.761682 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfbdw" event={"ID":"aa93e723-8c4e-4530-999f-54b461e2f308","Type":"ContainerDied","Data":"feb404a67aac9cae99f5446e29ad3b6552013827a00a59dec946cc5f499e0dbc"} Oct 31 12:36:44.765882 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.765839 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckhr9r" podStartSLOduration=3.969006702 podStartE2EDuration="34.765823055s" totalImagesPullingTime="30.796816353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:36:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:36:44.762446272 +0000 UTC m=+251.245569174" watchObservedRunningTime="2026-10-31 12:36:44.765823055 +0000 UTC m=+251.248945954" Oct 31 12:36:44.766222 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.766153 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-c97kk" event={"ID":"8df744a4-e981-4e9f-a400-b6b7ebf91aed","Type":"ContainerStarted","Data":"4552b8a1aee732b7100edb9a0f82ed6ca815b897086e0f91f9a3f06dbabaefc5"} Oct 31 12:36:44.769612 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.769551 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5tzkl" event={"ID":"97b09674-cbef-40a6-a291-3a03a2355042","Type":"ContainerDied","Data":"8bde697d90c360fd7d6914d660789463a538611efdfa0f4cc7675d57c995e91e"} Oct 31 12:36:44.769612 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.769592 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bde697d90c360fd7d6914d660789463a538611efdfa0f4cc7675d57c995e91e" Oct 31 12:36:44.769612 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:44.769596 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5tzkl" Oct 31 12:36:45.777424 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:45.777387 6841 generic.go:356] "Generic (PLEG): container finished" podID="4ec07c3a-7876-467b-8581-97538b73b62e" containerID="9fb9468014f99290a05db6366b4340bd4e3d9dbad7618097a7b932eaa2d74732" exitCode=0 Oct 31 12:36:45.777845 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:45.777474 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckhr9r" event={"ID":"4ec07c3a-7876-467b-8581-97538b73b62e","Type":"ContainerDied","Data":"9fb9468014f99290a05db6366b4340bd4e3d9dbad7618097a7b932eaa2d74732"} Oct 31 12:36:45.909038 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:45.909014 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfbdw" Oct 31 12:36:45.964926 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:45.964889 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa93e723-8c4e-4530-999f-54b461e2f308-bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa93e723-8c4e-4530-999f-54b461e2f308-bundle\") pod \"aa93e723-8c4e-4530-999f-54b461e2f308\" (UID: \"aa93e723-8c4e-4530-999f-54b461e2f308\") " Oct 31 12:36:45.965080 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:45.965066 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa93e723-8c4e-4530-999f-54b461e2f308-kube-api-access-24x5g\" (UniqueName: \"kubernetes.io/projected/aa93e723-8c4e-4530-999f-54b461e2f308-kube-api-access-24x5g\") pod \"aa93e723-8c4e-4530-999f-54b461e2f308\" (UID: \"aa93e723-8c4e-4530-999f-54b461e2f308\") " Oct 31 12:36:45.965140 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:45.965098 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa93e723-8c4e-4530-999f-54b461e2f308-util\" (UniqueName: \"kubernetes.io/empty-dir/aa93e723-8c4e-4530-999f-54b461e2f308-util\") pod \"aa93e723-8c4e-4530-999f-54b461e2f308\" (UID: \"aa93e723-8c4e-4530-999f-54b461e2f308\") " Oct 31 12:36:45.965357 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:45.965329 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa93e723-8c4e-4530-999f-54b461e2f308-bundle" pod "aa93e723-8c4e-4530-999f-54b461e2f308" (UID: "aa93e723-8c4e-4530-999f-54b461e2f308"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:36:45.967420 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:45.967389 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa93e723-8c4e-4530-999f-54b461e2f308-kube-api-access-24x5g" pod "aa93e723-8c4e-4530-999f-54b461e2f308" (UID: "aa93e723-8c4e-4530-999f-54b461e2f308"). InnerVolumeSpecName "kube-api-access-24x5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:36:45.972178 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:45.972151 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa93e723-8c4e-4530-999f-54b461e2f308-util" pod "aa93e723-8c4e-4530-999f-54b461e2f308" (UID: "aa93e723-8c4e-4530-999f-54b461e2f308"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:36:46.066171 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:46.066052 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24x5g\" (UniqueName: \"kubernetes.io/projected/aa93e723-8c4e-4530-999f-54b461e2f308-kube-api-access-24x5g\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:36:46.066171 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:46.066085 6841 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa93e723-8c4e-4530-999f-54b461e2f308-util\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:36:46.066171 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:46.066099 6841 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa93e723-8c4e-4530-999f-54b461e2f308-bundle\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:36:46.614145 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:46.614047 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:46.789636 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:46.789608 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfbdw" Oct 31 12:36:46.791010 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:46.789627 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bfbdw" event={"ID":"aa93e723-8c4e-4530-999f-54b461e2f308","Type":"ContainerDied","Data":"ce6381b401abd5eefe186597c9d318c981eae4b97071e4e11ae63d9a8872eeea"} Oct 31 12:36:46.791010 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:46.790646 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce6381b401abd5eefe186597c9d318c981eae4b97071e4e11ae63d9a8872eeea" Oct 31 12:36:46.849954 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:46.849922 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:46.924695 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:46.924669 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckhr9r" Oct 31 12:36:46.975697 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:46.975672 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ec07c3a-7876-467b-8581-97538b73b62e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ec07c3a-7876-467b-8581-97538b73b62e-bundle\") pod \"4ec07c3a-7876-467b-8581-97538b73b62e\" (UID: \"4ec07c3a-7876-467b-8581-97538b73b62e\") " Oct 31 12:36:46.975846 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:46.975738 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ec07c3a-7876-467b-8581-97538b73b62e-kube-api-access-kpbc7\" (UniqueName: \"kubernetes.io/projected/4ec07c3a-7876-467b-8581-97538b73b62e-kube-api-access-kpbc7\") pod \"4ec07c3a-7876-467b-8581-97538b73b62e\" (UID: \"4ec07c3a-7876-467b-8581-97538b73b62e\") " Oct 31 12:36:46.975846 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:46.975783 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ec07c3a-7876-467b-8581-97538b73b62e-util\" (UniqueName: \"kubernetes.io/empty-dir/4ec07c3a-7876-467b-8581-97538b73b62e-util\") pod \"4ec07c3a-7876-467b-8581-97538b73b62e\" (UID: \"4ec07c3a-7876-467b-8581-97538b73b62e\") " Oct 31 12:36:46.976532 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:46.976505 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ec07c3a-7876-467b-8581-97538b73b62e-bundle" pod "4ec07c3a-7876-467b-8581-97538b73b62e" (UID: "4ec07c3a-7876-467b-8581-97538b73b62e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:36:46.978242 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:46.978212 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ec07c3a-7876-467b-8581-97538b73b62e-kube-api-access-kpbc7" pod "4ec07c3a-7876-467b-8581-97538b73b62e" (UID: "4ec07c3a-7876-467b-8581-97538b73b62e"). InnerVolumeSpecName "kube-api-access-kpbc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:36:46.980577 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:46.980551 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ec07c3a-7876-467b-8581-97538b73b62e-util" pod "4ec07c3a-7876-467b-8581-97538b73b62e" (UID: "4ec07c3a-7876-467b-8581-97538b73b62e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:36:47.077371 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:47.077331 6841 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ec07c3a-7876-467b-8581-97538b73b62e-bundle\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:36:47.077371 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:47.077372 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpbc7\" (UniqueName: \"kubernetes.io/projected/4ec07c3a-7876-467b-8581-97538b73b62e-kube-api-access-kpbc7\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:36:47.077700 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:47.077390 6841 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ec07c3a-7876-467b-8581-97538b73b62e-util\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:36:47.795614 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:47.795594 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckhr9r" Oct 31 12:36:47.795926 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:47.795622 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckhr9r" event={"ID":"4ec07c3a-7876-467b-8581-97538b73b62e","Type":"ContainerDied","Data":"837b1fe5d6570232e8f09cbc9a99f004a42d947ab26ff70bfa938550bcf3c161"} Oct 31 12:36:47.795926 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:47.795666 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="837b1fe5d6570232e8f09cbc9a99f004a42d947ab26ff70bfa938550bcf3c161" Oct 31 12:36:48.800788 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:48.800750 6841 generic.go:356] "Generic (PLEG): container finished" podID="8df744a4-e981-4e9f-a400-b6b7ebf91aed" containerID="684debcf5d74aa4a37ed0ee10e8d00b84671a3ddcf02b53c3220a61626efb3c9" exitCode=0 Oct 31 12:36:48.801194 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:48.800825 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-c97kk" event={"ID":"8df744a4-e981-4e9f-a400-b6b7ebf91aed","Type":"ContainerDied","Data":"684debcf5d74aa4a37ed0ee10e8d00b84671a3ddcf02b53c3220a61626efb3c9"} Oct 31 12:36:49.807196 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:49.807157 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-c97kk" event={"ID":"8df744a4-e981-4e9f-a400-b6b7ebf91aed","Type":"ContainerStarted","Data":"648d396c1d273d8019361ca95733fee5f656039766382e45dba5e751cdc354e1"} Oct 31 12:36:49.807589 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:49.807285 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-c97kk" Oct 31 12:36:49.858866 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:49.858819 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-c97kk" podStartSLOduration=2.039997712 podStartE2EDuration="6.858807018s" totalImagesPullingTime="4.405751981s" totalInitContainerRuntime="413.057325ms" isStatefulPod=false podCreationTimestamp="2026-10-31 12:36:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:36:49.855565107 +0000 UTC m=+256.338688018" watchObservedRunningTime="2026-10-31 12:36:49.858807018 +0000 UTC m=+256.341929919" Oct 31 12:36:50.033084 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:36:50.033048 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:36:50.586446 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:50.586411 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:50.792057 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:50.792022 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:53.800709 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:53.800673 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:55.797685 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:55.797648 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:57.826027 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.825976 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-88qp9"] Oct 31 12:36:57.826407 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826297 6841 cpu_manager.go:414] "containerMap: removing container" podUID="4ec07c3a-7876-467b-8581-97538b73b62e" containerName="extract" Oct 31 12:36:57.826407 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826309 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ec07c3a-7876-467b-8581-97538b73b62e" containerName="extract" Oct 31 12:36:57.826407 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826319 6841 cpu_manager.go:414] "containerMap: removing container" podUID="bf0d7afa-9a99-4e74-a103-8d68ac678193" containerName="util" Oct 31 12:36:57.826407 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826324 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf0d7afa-9a99-4e74-a103-8d68ac678193" containerName="util" Oct 31 12:36:57.826407 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826330 6841 cpu_manager.go:414] "containerMap: removing container" podUID="aa93e723-8c4e-4530-999f-54b461e2f308" containerName="extract" Oct 31 12:36:57.826407 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826335 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa93e723-8c4e-4530-999f-54b461e2f308" containerName="extract" Oct 31 12:36:57.826407 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826343 6841 cpu_manager.go:414] "containerMap: removing container" podUID="e8be6efa-e396-43c7-a277-1f0cb8de7d83" containerName="pull" Oct 31 12:36:57.826407 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826347 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8be6efa-e396-43c7-a277-1f0cb8de7d83" containerName="pull" Oct 31 12:36:57.826407 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826353 6841 cpu_manager.go:414] "containerMap: removing container" podUID="e8be6efa-e396-43c7-a277-1f0cb8de7d83" containerName="extract" Oct 31 12:36:57.826407 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826358 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8be6efa-e396-43c7-a277-1f0cb8de7d83" containerName="extract" Oct 31 12:36:57.826407 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826368 6841 cpu_manager.go:414] "containerMap: removing container" podUID="aa93e723-8c4e-4530-999f-54b461e2f308" containerName="util" Oct 31 12:36:57.826407 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826373 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa93e723-8c4e-4530-999f-54b461e2f308" containerName="util" Oct 31 12:36:57.826407 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826379 6841 cpu_manager.go:414] "containerMap: removing container" podUID="97b09674-cbef-40a6-a291-3a03a2355042" containerName="extract" Oct 31 12:36:57.826407 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826384 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97b09674-cbef-40a6-a291-3a03a2355042" containerName="extract" Oct 31 12:36:57.826407 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826394 6841 cpu_manager.go:414] "containerMap: removing container" podUID="e8be6efa-e396-43c7-a277-1f0cb8de7d83" containerName="util" Oct 31 12:36:57.826407 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826398 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8be6efa-e396-43c7-a277-1f0cb8de7d83" containerName="util" Oct 31 12:36:57.826407 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826404 6841 cpu_manager.go:414] "containerMap: removing container" podUID="97b09674-cbef-40a6-a291-3a03a2355042" containerName="util" Oct 31 12:36:57.826407 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826408 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97b09674-cbef-40a6-a291-3a03a2355042" containerName="util" Oct 31 12:36:57.826407 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826414 6841 cpu_manager.go:414] "containerMap: removing container" podUID="97b09674-cbef-40a6-a291-3a03a2355042" containerName="pull" Oct 31 12:36:57.826943 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826418 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97b09674-cbef-40a6-a291-3a03a2355042" containerName="pull" Oct 31 12:36:57.826943 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826424 6841 cpu_manager.go:414] "containerMap: removing container" podUID="4ec07c3a-7876-467b-8581-97538b73b62e" containerName="pull" Oct 31 12:36:57.826943 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826428 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ec07c3a-7876-467b-8581-97538b73b62e" containerName="pull" Oct 31 12:36:57.826943 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826440 6841 cpu_manager.go:414] "containerMap: removing container" podUID="aa93e723-8c4e-4530-999f-54b461e2f308" containerName="pull" Oct 31 12:36:57.826943 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826444 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa93e723-8c4e-4530-999f-54b461e2f308" containerName="pull" Oct 31 12:36:57.826943 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826456 6841 cpu_manager.go:414] "containerMap: removing container" podUID="4ec07c3a-7876-467b-8581-97538b73b62e" containerName="util" Oct 31 12:36:57.826943 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826460 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ec07c3a-7876-467b-8581-97538b73b62e" containerName="util" Oct 31 12:36:57.826943 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826466 6841 cpu_manager.go:414] "containerMap: removing container" podUID="bf0d7afa-9a99-4e74-a103-8d68ac678193" containerName="extract" Oct 31 12:36:57.826943 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826470 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf0d7afa-9a99-4e74-a103-8d68ac678193" containerName="extract" Oct 31 12:36:57.826943 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826482 6841 cpu_manager.go:414] "containerMap: removing container" podUID="7dc8d73a-2166-4536-82eb-2be71518770b" containerName="pull" Oct 31 12:36:57.826943 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826486 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dc8d73a-2166-4536-82eb-2be71518770b" containerName="pull" Oct 31 12:36:57.826943 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826491 6841 cpu_manager.go:414] "containerMap: removing container" podUID="7dc8d73a-2166-4536-82eb-2be71518770b" containerName="extract" Oct 31 12:36:57.826943 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826495 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dc8d73a-2166-4536-82eb-2be71518770b" containerName="extract" Oct 31 12:36:57.826943 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826502 6841 cpu_manager.go:414] "containerMap: removing container" podUID="7dc8d73a-2166-4536-82eb-2be71518770b" containerName="util" Oct 31 12:36:57.826943 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826506 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dc8d73a-2166-4536-82eb-2be71518770b" containerName="util" Oct 31 12:36:57.826943 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826513 6841 cpu_manager.go:414] "containerMap: removing container" podUID="bf0d7afa-9a99-4e74-a103-8d68ac678193" containerName="pull" Oct 31 12:36:57.826943 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826517 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf0d7afa-9a99-4e74-a103-8d68ac678193" containerName="pull" Oct 31 12:36:57.826943 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826578 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="97b09674-cbef-40a6-a291-3a03a2355042" containerName="extract" Oct 31 12:36:57.826943 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826588 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf0d7afa-9a99-4e74-a103-8d68ac678193" containerName="extract" Oct 31 12:36:57.826943 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826594 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dc8d73a-2166-4536-82eb-2be71518770b" containerName="extract" Oct 31 12:36:57.826943 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826599 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ec07c3a-7876-467b-8581-97538b73b62e" containerName="extract" Oct 31 12:36:57.826943 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826605 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa93e723-8c4e-4530-999f-54b461e2f308" containerName="extract" Oct 31 12:36:57.826943 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.826610 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8be6efa-e396-43c7-a277-1f0cb8de7d83" containerName="extract" Oct 31 12:36:57.832531 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.832511 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-88qp9" Oct 31 12:36:57.861876 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.861846 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-fzgtn\"" Oct 31 12:36:57.863321 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.863297 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-88qp9"] Oct 31 12:36:57.863437 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.863303 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 31 12:36:57.863623 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.863604 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 12:36:57.863721 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.863634 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 12:36:57.969778 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:57.969748 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7srp2\" (UniqueName: \"kubernetes.io/projected/63f4505c-09a4-4903-b6ba-824d5052bb24-kube-api-access-7srp2\") pod \"dns-operator-controller-manager-8557cbdb84-88qp9\" (UID: \"63f4505c-09a4-4903-b6ba-824d5052bb24\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-88qp9" Oct 31 12:36:58.071101 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:58.071069 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7srp2\" (UniqueName: \"kubernetes.io/projected/63f4505c-09a4-4903-b6ba-824d5052bb24-kube-api-access-7srp2\") pod \"dns-operator-controller-manager-8557cbdb84-88qp9\" (UID: \"63f4505c-09a4-4903-b6ba-824d5052bb24\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-88qp9" Oct 31 12:36:58.085282 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:58.085222 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7srp2\" (UniqueName: \"kubernetes.io/projected/63f4505c-09a4-4903-b6ba-824d5052bb24-kube-api-access-7srp2\") pod \"dns-operator-controller-manager-8557cbdb84-88qp9\" (UID: \"63f4505c-09a4-4903-b6ba-824d5052bb24\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-88qp9" Oct 31 12:36:58.145322 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:58.145291 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-88qp9" Oct 31 12:36:58.420372 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:58.420273 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-88qp9"] Oct 31 12:36:58.423893 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:36:58.423855 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f4505c_09a4_4903_b6ba_824d5052bb24.slice/crio-6cf565dfa2321a1df3d28dc594c2a53731ca2477b9ddcb2a9c3c14ecfe6896c9 WatchSource:0}: Error finding container 6cf565dfa2321a1df3d28dc594c2a53731ca2477b9ddcb2a9c3c14ecfe6896c9: Status 404 returned error can't find the container with id 6cf565dfa2321a1df3d28dc594c2a53731ca2477b9ddcb2a9c3c14ecfe6896c9 Oct 31 12:36:58.845812 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:58.845780 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-88qp9" event={"ID":"63f4505c-09a4-4903-b6ba-824d5052bb24","Type":"ContainerStarted","Data":"6cf565dfa2321a1df3d28dc594c2a53731ca2477b9ddcb2a9c3c14ecfe6896c9"} Oct 31 12:36:58.957137 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:58.957101 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:36:59.813305 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:59.813263 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2s98x"] Oct 31 12:36:59.817050 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:59.817025 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2s98x" Oct 31 12:36:59.821797 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:59.821772 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 31 12:36:59.822064 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:59.822036 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 31 12:36:59.823228 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:59.823176 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-8j25l\"" Oct 31 12:36:59.836552 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:59.836527 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2s98x"] Oct 31 12:36:59.989021 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:59.988956 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9rl6\" (UniqueName: \"kubernetes.io/projected/59222f02-c42c-4a96-a582-9131d3af2150-kube-api-access-h9rl6\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-2s98x\" (UID: \"59222f02-c42c-4a96-a582-9131d3af2150\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2s98x" Oct 31 12:36:59.989411 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:36:59.989037 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59222f02-c42c-4a96-a582-9131d3af2150-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-2s98x\" (UID: \"59222f02-c42c-4a96-a582-9131d3af2150\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2s98x" Oct 31 12:37:00.090555 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:00.090470 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9rl6\" (UniqueName: \"kubernetes.io/projected/59222f02-c42c-4a96-a582-9131d3af2150-kube-api-access-h9rl6\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-2s98x\" (UID: \"59222f02-c42c-4a96-a582-9131d3af2150\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2s98x" Oct 31 12:37:00.090555 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:00.090543 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59222f02-c42c-4a96-a582-9131d3af2150-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-2s98x\" (UID: \"59222f02-c42c-4a96-a582-9131d3af2150\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2s98x" Oct 31 12:37:00.091020 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:00.090997 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59222f02-c42c-4a96-a582-9131d3af2150-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-2s98x\" (UID: \"59222f02-c42c-4a96-a582-9131d3af2150\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2s98x" Oct 31 12:37:00.103009 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:00.102960 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9rl6\" (UniqueName: \"kubernetes.io/projected/59222f02-c42c-4a96-a582-9131d3af2150-kube-api-access-h9rl6\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-2s98x\" (UID: \"59222f02-c42c-4a96-a582-9131d3af2150\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2s98x" Oct 31 12:37:00.132884 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:00.132856 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2s98x" Oct 31 12:37:00.434628 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:00.434542 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2s98x"] Oct 31 12:37:00.439385 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:37:00.439356 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59222f02_c42c_4a96_a582_9131d3af2150.slice/crio-ef2471b838a1adc28977054ee1777554fc8fd82c2a9abc0723b479c7082628d0 WatchSource:0}: Error finding container ef2471b838a1adc28977054ee1777554fc8fd82c2a9abc0723b479c7082628d0: Status 404 returned error can't find the container with id ef2471b838a1adc28977054ee1777554fc8fd82c2a9abc0723b479c7082628d0 Oct 31 12:37:00.813814 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:00.813778 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-c97kk" Oct 31 12:37:00.856909 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:00.856874 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2s98x" event={"ID":"59222f02-c42c-4a96-a582-9131d3af2150","Type":"ContainerStarted","Data":"ef2471b838a1adc28977054ee1777554fc8fd82c2a9abc0723b479c7082628d0"} Oct 31 12:37:01.582375 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:01.582325 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-x7xbq"] Oct 31 12:37:01.585947 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:01.585917 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x7xbq" Oct 31 12:37:01.597730 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:01.597708 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-bgvgh\"" Oct 31 12:37:01.605443 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:01.605417 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-x7xbq"] Oct 31 12:37:01.706274 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:01.706238 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4k5b\" (UniqueName: \"kubernetes.io/projected/38fff3bd-4067-4f10-b87f-b9e0fa806e3a-kube-api-access-g4k5b\") pod \"limitador-operator-controller-manager-5778bb4449-x7xbq\" (UID: \"38fff3bd-4067-4f10-b87f-b9e0fa806e3a\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x7xbq" Oct 31 12:37:01.807446 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:01.807401 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4k5b\" (UniqueName: \"kubernetes.io/projected/38fff3bd-4067-4f10-b87f-b9e0fa806e3a-kube-api-access-g4k5b\") pod \"limitador-operator-controller-manager-5778bb4449-x7xbq\" (UID: \"38fff3bd-4067-4f10-b87f-b9e0fa806e3a\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x7xbq" Oct 31 12:37:01.828141 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:01.828105 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4k5b\" (UniqueName: \"kubernetes.io/projected/38fff3bd-4067-4f10-b87f-b9e0fa806e3a-kube-api-access-g4k5b\") pod \"limitador-operator-controller-manager-5778bb4449-x7xbq\" (UID: \"38fff3bd-4067-4f10-b87f-b9e0fa806e3a\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x7xbq" Oct 31 12:37:01.864236 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:01.864145 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-88qp9" event={"ID":"63f4505c-09a4-4903-b6ba-824d5052bb24","Type":"ContainerStarted","Data":"fddeaadcb269b85c23eeb038610d7f4224f13692c3c7afdd1d84b525862a0b95"} Oct 31 12:37:01.864423 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:01.864345 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-88qp9" Oct 31 12:37:01.902250 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:01.902219 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x7xbq" Oct 31 12:37:01.906491 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:01.906431 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-88qp9" podStartSLOduration=2.201724091 podStartE2EDuration="4.906415684s" totalImagesPullingTime="2.704691593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:36:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:01.903905453 +0000 UTC m=+268.387028355" watchObservedRunningTime="2026-10-31 12:37:01.906415684 +0000 UTC m=+268.389538584" Oct 31 12:37:02.035345 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:02.035309 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:02.190013 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:02.188231 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-x7xbq"] Oct 31 12:37:02.191592 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:37:02.191562 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38fff3bd_4067_4f10_b87f_b9e0fa806e3a.slice/crio-ffdbbbacb1c3f2ffd41fd4418e266868050aba62034e4d907603150387f8a19b WatchSource:0}: Error finding container ffdbbbacb1c3f2ffd41fd4418e266868050aba62034e4d907603150387f8a19b: Status 404 returned error can't find the container with id ffdbbbacb1c3f2ffd41fd4418e266868050aba62034e4d907603150387f8a19b Oct 31 12:37:02.236820 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:02.236775 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:02.439733 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:02.439643 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:02.871815 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:02.871778 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2s98x" event={"ID":"59222f02-c42c-4a96-a582-9131d3af2150","Type":"ContainerStarted","Data":"de871981d94b13dcdd1791edee4d810f809d46e9f1386b3515a0da89616fc280"} Oct 31 12:37:02.873281 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:02.873250 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x7xbq" event={"ID":"38fff3bd-4067-4f10-b87f-b9e0fa806e3a","Type":"ContainerStarted","Data":"ffdbbbacb1c3f2ffd41fd4418e266868050aba62034e4d907603150387f8a19b"} Oct 31 12:37:02.898826 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:02.898771 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2s98x" podStartSLOduration=2.03250553 podStartE2EDuration="3.898753535s" totalImagesPullingTime="1.866248005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:36:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:02.896627959 +0000 UTC m=+269.379750859" watchObservedRunningTime="2026-10-31 12:37:02.898753535 +0000 UTC m=+269.381876437" Oct 31 12:37:03.405642 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:03.405594 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:04.259563 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:04.259524 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:04.388736 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:04.388698 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-s6mp9"] Oct 31 12:37:04.393016 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:04.392276 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s6mp9" Oct 31 12:37:04.400901 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:04.400875 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-9n6ng\"" Oct 31 12:37:04.406191 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:04.406172 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 31 12:37:04.406319 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:04.406287 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 31 12:37:04.409896 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:04.409871 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-s6mp9"] Oct 31 12:37:04.510224 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:04.510094 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:04.530412 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:04.530380 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5091b234-cbf6-4835-95be-ecf9ee0440ef-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-s6mp9\" (UID: \"5091b234-cbf6-4835-95be-ecf9ee0440ef\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s6mp9" Oct 31 12:37:04.530593 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:04.530451 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz4gk\" (UniqueName: \"kubernetes.io/projected/5091b234-cbf6-4835-95be-ecf9ee0440ef-kube-api-access-qz4gk\") pod \"cert-manager-webhook-5dbbbf75cc-s6mp9\" (UID: \"5091b234-cbf6-4835-95be-ecf9ee0440ef\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s6mp9" Oct 31 12:37:04.631328 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:04.631291 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5091b234-cbf6-4835-95be-ecf9ee0440ef-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-s6mp9\" (UID: \"5091b234-cbf6-4835-95be-ecf9ee0440ef\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s6mp9" Oct 31 12:37:04.631490 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:04.631361 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz4gk\" (UniqueName: \"kubernetes.io/projected/5091b234-cbf6-4835-95be-ecf9ee0440ef-kube-api-access-qz4gk\") pod \"cert-manager-webhook-5dbbbf75cc-s6mp9\" (UID: \"5091b234-cbf6-4835-95be-ecf9ee0440ef\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s6mp9" Oct 31 12:37:04.671850 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:04.671814 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5091b234-cbf6-4835-95be-ecf9ee0440ef-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-s6mp9\" (UID: \"5091b234-cbf6-4835-95be-ecf9ee0440ef\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s6mp9" Oct 31 12:37:04.678392 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:04.678365 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz4gk\" (UniqueName: \"kubernetes.io/projected/5091b234-cbf6-4835-95be-ecf9ee0440ef-kube-api-access-qz4gk\") pod \"cert-manager-webhook-5dbbbf75cc-s6mp9\" (UID: \"5091b234-cbf6-4835-95be-ecf9ee0440ef\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s6mp9" Oct 31 12:37:04.722795 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:04.722770 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s6mp9" Oct 31 12:37:04.874085 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:04.874045 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:04.884592 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:04.884554 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x7xbq" event={"ID":"38fff3bd-4067-4f10-b87f-b9e0fa806e3a","Type":"ContainerStarted","Data":"c7aa3fe652a6f6d7a2428a6a83aa917f07ffa773a807170886119319efb13562"} Oct 31 12:37:04.884763 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:04.884699 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x7xbq" Oct 31 12:37:05.000612 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:05.000567 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x7xbq" podStartSLOduration=2.349207989 podStartE2EDuration="4.000552672s" totalImagesPullingTime="1.651344683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:04.917838192 +0000 UTC m=+271.400961094" watchObservedRunningTime="2026-10-31 12:37:05.000552672 +0000 UTC m=+271.483675583" Oct 31 12:37:05.001973 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:05.001929 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-s6mp9"] Oct 31 12:37:05.004626 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:37:05.004602 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5091b234_cbf6_4835_95be_ecf9ee0440ef.slice/crio-2ca2f118a4a909428f8bed77cfbb56b42e839ad81020e5bf3faec0daaabd3e07 WatchSource:0}: Error finding container 2ca2f118a4a909428f8bed77cfbb56b42e839ad81020e5bf3faec0daaabd3e07: Status 404 returned error can't find the container with id 2ca2f118a4a909428f8bed77cfbb56b42e839ad81020e5bf3faec0daaabd3e07 Oct 31 12:37:05.144699 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:05.144605 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:05.889949 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:05.889911 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s6mp9" event={"ID":"5091b234-cbf6-4835-95be-ecf9ee0440ef","Type":"ContainerStarted","Data":"2ca2f118a4a909428f8bed77cfbb56b42e839ad81020e5bf3faec0daaabd3e07"} Oct 31 12:37:06.105941 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:06.105872 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:06.330811 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:06.330781 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:06.627530 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:06.627444 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:06.846475 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:06.846443 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:06.949878 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:06.949846 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-t5rbc"] Oct 31 12:37:06.953270 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:06.953247 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t5rbc" Oct 31 12:37:06.964519 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:06.964493 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-r6htm\"" Oct 31 12:37:06.967599 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:06.967579 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-t5rbc"] Oct 31 12:37:07.050747 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:07.050711 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/71fea128-de42-4ec8-b3d7-4ad80de0045e-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-t5rbc\" (UID: \"71fea128-de42-4ec8-b3d7-4ad80de0045e\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t5rbc" Oct 31 12:37:07.050904 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:07.050782 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcn7x\" (UniqueName: \"kubernetes.io/projected/71fea128-de42-4ec8-b3d7-4ad80de0045e-kube-api-access-gcn7x\") pod \"cert-manager-cainjector-5c57bbd4fc-t5rbc\" (UID: \"71fea128-de42-4ec8-b3d7-4ad80de0045e\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t5rbc" Oct 31 12:37:07.151525 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:07.151488 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/71fea128-de42-4ec8-b3d7-4ad80de0045e-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-t5rbc\" (UID: \"71fea128-de42-4ec8-b3d7-4ad80de0045e\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t5rbc" Oct 31 12:37:07.151717 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:07.151554 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcn7x\" (UniqueName: \"kubernetes.io/projected/71fea128-de42-4ec8-b3d7-4ad80de0045e-kube-api-access-gcn7x\") pod \"cert-manager-cainjector-5c57bbd4fc-t5rbc\" (UID: \"71fea128-de42-4ec8-b3d7-4ad80de0045e\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t5rbc" Oct 31 12:37:07.163932 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:07.163906 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/71fea128-de42-4ec8-b3d7-4ad80de0045e-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-t5rbc\" (UID: \"71fea128-de42-4ec8-b3d7-4ad80de0045e\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t5rbc" Oct 31 12:37:07.164095 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:07.164012 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcn7x\" (UniqueName: \"kubernetes.io/projected/71fea128-de42-4ec8-b3d7-4ad80de0045e-kube-api-access-gcn7x\") pod \"cert-manager-cainjector-5c57bbd4fc-t5rbc\" (UID: \"71fea128-de42-4ec8-b3d7-4ad80de0045e\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t5rbc" Oct 31 12:37:07.267891 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:07.267797 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t5rbc" Oct 31 12:37:07.550314 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:07.550286 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-t5rbc"] Oct 31 12:37:07.555169 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:37:07.555116 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71fea128_de42_4ec8_b3d7_4ad80de0045e.slice/crio-f864e7e7f9cd7621c36ef5af19f386622e8577ea4618bdef2af160a294f6815a WatchSource:0}: Error finding container f864e7e7f9cd7621c36ef5af19f386622e8577ea4618bdef2af160a294f6815a: Status 404 returned error can't find the container with id f864e7e7f9cd7621c36ef5af19f386622e8577ea4618bdef2af160a294f6815a Oct 31 12:37:07.901909 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:07.901806 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t5rbc" event={"ID":"71fea128-de42-4ec8-b3d7-4ad80de0045e","Type":"ContainerStarted","Data":"f864e7e7f9cd7621c36ef5af19f386622e8577ea4618bdef2af160a294f6815a"} Oct 31 12:37:08.907700 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:08.907661 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t5rbc" event={"ID":"71fea128-de42-4ec8-b3d7-4ad80de0045e","Type":"ContainerStarted","Data":"4fb108d604a54635fca64c0a45aaf13e76ce3f1320f50e8af10b01f4cbec09c4"} Oct 31 12:37:08.909373 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:08.909340 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s6mp9" event={"ID":"5091b234-cbf6-4835-95be-ecf9ee0440ef","Type":"ContainerStarted","Data":"00d422210c807c729c4f993e6a893b4c60664e39855e19c0000fca51a96b1037"} Oct 31 12:37:08.909497 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:08.909424 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s6mp9" Oct 31 12:37:08.927564 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:08.927514 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t5rbc" podStartSLOduration=2.141013599 podStartE2EDuration="2.927503484s" totalImagesPullingTime="786.489885ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:08.925538127 +0000 UTC m=+275.408661065" watchObservedRunningTime="2026-10-31 12:37:08.927503484 +0000 UTC m=+275.410626385" Oct 31 12:37:08.946757 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:08.946697 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s6mp9" podStartSLOduration=1.966722577 podStartE2EDuration="4.946671961s" totalImagesPullingTime="2.979949384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:08.943049131 +0000 UTC m=+275.426172034" watchObservedRunningTime="2026-10-31 12:37:08.946671961 +0000 UTC m=+275.429794862" Oct 31 12:37:09.569886 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:09.569852 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:10.414082 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:10.414045 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:10.668358 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:10.668289 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:11.836754 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:11.836719 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:12.875773 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:12.875743 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-88qp9" Oct 31 12:37:13.522015 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:13.521965 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-44jbg"] Oct 31 12:37:13.525355 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:13.525334 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-44jbg" Oct 31 12:37:13.532255 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:13.532134 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 31 12:37:13.532255 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:13.532197 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-gcpbr\"" Oct 31 12:37:13.532255 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:13.532201 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 31 12:37:13.537681 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:13.537658 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-44jbg"] Oct 31 12:37:13.710116 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:13.710088 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/282ca2ce-a93b-419e-9da6-3df6251f8f09-tmp\") pod \"jobset-operator-674cd59dc5-44jbg\" (UID: \"282ca2ce-a93b-419e-9da6-3df6251f8f09\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-44jbg" Oct 31 12:37:13.710275 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:13.710127 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4q9z\" (UniqueName: \"kubernetes.io/projected/282ca2ce-a93b-419e-9da6-3df6251f8f09-kube-api-access-v4q9z\") pod \"jobset-operator-674cd59dc5-44jbg\" (UID: \"282ca2ce-a93b-419e-9da6-3df6251f8f09\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-44jbg" Oct 31 12:37:13.810708 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:13.810626 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/282ca2ce-a93b-419e-9da6-3df6251f8f09-tmp\") pod \"jobset-operator-674cd59dc5-44jbg\" (UID: \"282ca2ce-a93b-419e-9da6-3df6251f8f09\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-44jbg" Oct 31 12:37:13.810708 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:13.810689 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4q9z\" (UniqueName: \"kubernetes.io/projected/282ca2ce-a93b-419e-9da6-3df6251f8f09-kube-api-access-v4q9z\") pod \"jobset-operator-674cd59dc5-44jbg\" (UID: \"282ca2ce-a93b-419e-9da6-3df6251f8f09\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-44jbg" Oct 31 12:37:13.811118 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:13.811097 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/282ca2ce-a93b-419e-9da6-3df6251f8f09-tmp\") pod \"jobset-operator-674cd59dc5-44jbg\" (UID: \"282ca2ce-a93b-419e-9da6-3df6251f8f09\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-44jbg" Oct 31 12:37:13.824564 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:13.824534 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4q9z\" (UniqueName: \"kubernetes.io/projected/282ca2ce-a93b-419e-9da6-3df6251f8f09-kube-api-access-v4q9z\") pod \"jobset-operator-674cd59dc5-44jbg\" (UID: \"282ca2ce-a93b-419e-9da6-3df6251f8f09\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-44jbg" Oct 31 12:37:13.836296 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:13.836278 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-44jbg" Oct 31 12:37:14.108024 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:14.107994 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-44jbg"] Oct 31 12:37:14.110630 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:37:14.110595 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod282ca2ce_a93b_419e_9da6_3df6251f8f09.slice/crio-fa5f076f067a6b4405de3bdfbba5d25a805a391b983b44221649ec9f307a85bc WatchSource:0}: Error finding container fa5f076f067a6b4405de3bdfbba5d25a805a391b983b44221649ec9f307a85bc: Status 404 returned error can't find the container with id fa5f076f067a6b4405de3bdfbba5d25a805a391b983b44221649ec9f307a85bc Oct 31 12:37:14.243188 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:14.243156 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:14.916643 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:14.916615 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s6mp9" Oct 31 12:37:14.938616 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:14.938586 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-44jbg" event={"ID":"282ca2ce-a93b-419e-9da6-3df6251f8f09","Type":"ContainerStarted","Data":"fa5f076f067a6b4405de3bdfbba5d25a805a391b983b44221649ec9f307a85bc"} Oct 31 12:37:15.892530 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:15.892453 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x7xbq" Oct 31 12:37:16.950097 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:16.950065 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-44jbg" event={"ID":"282ca2ce-a93b-419e-9da6-3df6251f8f09","Type":"ContainerStarted","Data":"39048db6fe81a7d840eaa474c9f719b579afd4f3bdf6b2b9aca14c2d7a8a1bcf"} Oct 31 12:37:16.969002 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:16.968937 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-44jbg" podStartSLOduration=1.7506449480000001 podStartE2EDuration="3.968925081s" totalImagesPullingTime="2.218280133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:16.967808889 +0000 UTC m=+283.450931793" watchObservedRunningTime="2026-10-31 12:37:16.968925081 +0000 UTC m=+283.452047981" Oct 31 12:37:17.178522 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:17.178482 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:17.384958 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:17.384876 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:19.491483 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:19.491409 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:19.670333 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:19.670303 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-54cf69b775-mvpw2"] Oct 31 12:37:19.729020 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:19.728925 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54cf69b775-mvpw2"] Oct 31 12:37:19.729255 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:19.729034 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54cf69b775-mvpw2" Oct 31 12:37:19.739563 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:19.739539 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 12:37:19.739693 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:19.739572 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 31 12:37:19.740728 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:19.740704 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:19.741442 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:19.741425 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xqbvn\"" Oct 31 12:37:19.745786 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:19.745744 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 31 12:37:19.756265 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:19.756241 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c41ccbe4-e016-4012-b581-6e97c3a46e4c-tmp\") pod \"kube-auth-proxy-54cf69b775-mvpw2\" (UID: \"c41ccbe4-e016-4012-b581-6e97c3a46e4c\") " pod="openshift-ingress/kube-auth-proxy-54cf69b775-mvpw2" Oct 31 12:37:19.756360 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:19.756279 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjx7h\" (UniqueName: \"kubernetes.io/projected/c41ccbe4-e016-4012-b581-6e97c3a46e4c-kube-api-access-mjx7h\") pod \"kube-auth-proxy-54cf69b775-mvpw2\" (UID: \"c41ccbe4-e016-4012-b581-6e97c3a46e4c\") " pod="openshift-ingress/kube-auth-proxy-54cf69b775-mvpw2" Oct 31 12:37:19.756422 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:19.756404 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c41ccbe4-e016-4012-b581-6e97c3a46e4c-tls-certs\") pod \"kube-auth-proxy-54cf69b775-mvpw2\" (UID: \"c41ccbe4-e016-4012-b581-6e97c3a46e4c\") " pod="openshift-ingress/kube-auth-proxy-54cf69b775-mvpw2" Oct 31 12:37:19.781374 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:19.781353 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 12:37:19.857316 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:19.857282 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c41ccbe4-e016-4012-b581-6e97c3a46e4c-tls-certs\") pod \"kube-auth-proxy-54cf69b775-mvpw2\" (UID: \"c41ccbe4-e016-4012-b581-6e97c3a46e4c\") " pod="openshift-ingress/kube-auth-proxy-54cf69b775-mvpw2" Oct 31 12:37:19.857473 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:19.857333 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c41ccbe4-e016-4012-b581-6e97c3a46e4c-tmp\") pod \"kube-auth-proxy-54cf69b775-mvpw2\" (UID: \"c41ccbe4-e016-4012-b581-6e97c3a46e4c\") " pod="openshift-ingress/kube-auth-proxy-54cf69b775-mvpw2" Oct 31 12:37:19.857473 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:19.857383 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjx7h\" (UniqueName: \"kubernetes.io/projected/c41ccbe4-e016-4012-b581-6e97c3a46e4c-kube-api-access-mjx7h\") pod \"kube-auth-proxy-54cf69b775-mvpw2\" (UID: \"c41ccbe4-e016-4012-b581-6e97c3a46e4c\") " pod="openshift-ingress/kube-auth-proxy-54cf69b775-mvpw2" Oct 31 12:37:19.859715 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:19.859689 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c41ccbe4-e016-4012-b581-6e97c3a46e4c-tmp\") pod \"kube-auth-proxy-54cf69b775-mvpw2\" (UID: \"c41ccbe4-e016-4012-b581-6e97c3a46e4c\") " pod="openshift-ingress/kube-auth-proxy-54cf69b775-mvpw2" Oct 31 12:37:19.860089 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:19.860068 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c41ccbe4-e016-4012-b581-6e97c3a46e4c-tls-certs\") pod \"kube-auth-proxy-54cf69b775-mvpw2\" (UID: \"c41ccbe4-e016-4012-b581-6e97c3a46e4c\") " pod="openshift-ingress/kube-auth-proxy-54cf69b775-mvpw2" Oct 31 12:37:19.882495 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:19.882466 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjx7h\" (UniqueName: \"kubernetes.io/projected/c41ccbe4-e016-4012-b581-6e97c3a46e4c-kube-api-access-mjx7h\") pod \"kube-auth-proxy-54cf69b775-mvpw2\" (UID: \"c41ccbe4-e016-4012-b581-6e97c3a46e4c\") " pod="openshift-ingress/kube-auth-proxy-54cf69b775-mvpw2" Oct 31 12:37:19.960475 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:19.960435 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:20.033203 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:37:20.033123 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:37:20.054447 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:20.054421 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54cf69b775-mvpw2" Oct 31 12:37:20.304454 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:20.304376 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:20.330181 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:20.330152 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54cf69b775-mvpw2"] Oct 31 12:37:20.332649 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:37:20.332614 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc41ccbe4_e016_4012_b581_6e97c3a46e4c.slice/crio-a1bf48afc4431664594decb86b15c23ffc4dd70be4929867d5236b19718873c5 WatchSource:0}: Error finding container a1bf48afc4431664594decb86b15c23ffc4dd70be4929867d5236b19718873c5: Status 404 returned error can't find the container with id a1bf48afc4431664594decb86b15c23ffc4dd70be4929867d5236b19718873c5 Oct 31 12:37:20.524847 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:20.524803 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:20.973585 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:20.973542 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54cf69b775-mvpw2" event={"ID":"c41ccbe4-e016-4012-b581-6e97c3a46e4c","Type":"ContainerStarted","Data":"a1bf48afc4431664594decb86b15c23ffc4dd70be4929867d5236b19718873c5"} Oct 31 12:37:21.170744 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:21.170702 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:21.368240 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:21.368133 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:21.945706 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:21.945668 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:22.985381 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:22.985289 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54cf69b775-mvpw2" event={"ID":"c41ccbe4-e016-4012-b581-6e97c3a46e4c","Type":"ContainerStarted","Data":"61dd19481db3cbcfa74a81df5d713c61cf547e24900c1028ea702a915e0e55c9"} Oct 31 12:37:23.017623 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:23.017570 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-54cf69b775-mvpw2" podStartSLOduration=1.651319135 podStartE2EDuration="4.017558026s" totalImagesPullingTime="2.366238891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:23.015723181 +0000 UTC m=+289.498846081" watchObservedRunningTime="2026-10-31 12:37:23.017558026 +0000 UTC m=+289.500680928" Oct 31 12:37:27.155236 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:27.155202 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:27.358858 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:27.358808 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:28.158885 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:28.158848 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:28.359722 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:28.359684 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:28.759312 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:28.759277 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:28.954713 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:28.954673 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:31.900079 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:31.900042 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:32.102948 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:32.102905 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:34.991720 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:34.991685 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:37.354074 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:37.354030 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:37.555003 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:37.554962 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:37.951890 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:37.951852 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:38.151021 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:38.150962 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:43.162412 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:43.162374 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:43.362273 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:43.362218 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:46.389677 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:46.389628 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:46.583947 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:46.583914 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:49.519947 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:49.519903 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:49.722132 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:49.722101 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:50.035668 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:37:50.035632 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:37:50.958041 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:50.957996 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:51.157502 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:51.157461 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:51.353023 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:51.352908 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:51.546838 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:51.546782 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:59.626369 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:59.626318 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:37:59.925409 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:37:59.925324 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:16.999753 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:16.999720 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-j72kb"] Oct 31 12:38:17.003234 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.003212 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-j72kb" Oct 31 12:38:17.015717 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.015693 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 12:38:17.015832 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.015693 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 12:38:17.017257 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.017232 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-j72kb"] Oct 31 12:38:17.018560 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.018535 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:17.057648 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.057620 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96de151e-d180-45de-8c17-a934e66ddc50-cert\") pod \"kserve-controller-manager-776cc66d47-j72kb\" (UID: \"96de151e-d180-45de-8c17-a934e66ddc50\") " pod="opendatahub/kserve-controller-manager-776cc66d47-j72kb" Oct 31 12:38:17.057779 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.057679 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf5tl\" (UniqueName: \"kubernetes.io/projected/96de151e-d180-45de-8c17-a934e66ddc50-kube-api-access-tf5tl\") pod \"kserve-controller-manager-776cc66d47-j72kb\" (UID: \"96de151e-d180-45de-8c17-a934e66ddc50\") " pod="opendatahub/kserve-controller-manager-776cc66d47-j72kb" Oct 31 12:38:17.082727 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.082703 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-jm7lb"] Oct 31 12:38:17.086014 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.085999 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-jm7lb" Oct 31 12:38:17.106938 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.106911 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-rsvxv\"" Oct 31 12:38:17.107088 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.106959 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-m5xqf\"" Oct 31 12:38:17.107376 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.107358 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 31 12:38:17.107464 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.107373 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 31 12:38:17.107963 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.107944 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-jm7lb"] Oct 31 12:38:17.111138 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.111117 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-s2ldm"] Oct 31 12:38:17.114464 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.114447 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-s2ldm" Oct 31 12:38:17.139309 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.139280 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-gdnx9\"" Oct 31 12:38:17.139592 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.139577 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 31 12:38:17.139692 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.139673 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 31 12:38:17.145348 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.145321 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-s2ldm"] Oct 31 12:38:17.158335 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.158311 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrwkr\" (UniqueName: \"kubernetes.io/projected/ecf5ba9f-127b-45de-969f-8128243e0bff-kube-api-access-vrwkr\") pod \"model-serving-api-75bd89fdfc-jm7lb\" (UID: \"ecf5ba9f-127b-45de-969f-8128243e0bff\") " pod="opendatahub/model-serving-api-75bd89fdfc-jm7lb" Oct 31 12:38:17.158486 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.158345 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96de151e-d180-45de-8c17-a934e66ddc50-cert\") pod \"kserve-controller-manager-776cc66d47-j72kb\" (UID: \"96de151e-d180-45de-8c17-a934e66ddc50\") " pod="opendatahub/kserve-controller-manager-776cc66d47-j72kb" Oct 31 12:38:17.158486 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.158381 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecf5ba9f-127b-45de-969f-8128243e0bff-tls-certs\") pod \"model-serving-api-75bd89fdfc-jm7lb\" (UID: \"ecf5ba9f-127b-45de-969f-8128243e0bff\") " pod="opendatahub/model-serving-api-75bd89fdfc-jm7lb" Oct 31 12:38:17.158486 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.158479 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf5tl\" (UniqueName: \"kubernetes.io/projected/96de151e-d180-45de-8c17-a934e66ddc50-kube-api-access-tf5tl\") pod \"kserve-controller-manager-776cc66d47-j72kb\" (UID: \"96de151e-d180-45de-8c17-a934e66ddc50\") " pod="opendatahub/kserve-controller-manager-776cc66d47-j72kb" Oct 31 12:38:17.158660 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.158529 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7c757c0-2ab1-4639-9594-42ab52830a01-metrics-certs\") pod \"odh-model-controller-769875db56-s2ldm\" (UID: \"e7c757c0-2ab1-4639-9594-42ab52830a01\") " pod="opendatahub/odh-model-controller-769875db56-s2ldm" Oct 31 12:38:17.158660 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.158575 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9lkp\" (UniqueName: \"kubernetes.io/projected/e7c757c0-2ab1-4639-9594-42ab52830a01-kube-api-access-h9lkp\") pod \"odh-model-controller-769875db56-s2ldm\" (UID: \"e7c757c0-2ab1-4639-9594-42ab52830a01\") " pod="opendatahub/odh-model-controller-769875db56-s2ldm" Oct 31 12:38:17.158660 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.158622 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e7c757c0-2ab1-4639-9594-42ab52830a01-cert\") pod \"odh-model-controller-769875db56-s2ldm\" (UID: \"e7c757c0-2ab1-4639-9594-42ab52830a01\") " pod="opendatahub/odh-model-controller-769875db56-s2ldm" Oct 31 12:38:17.160913 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.160891 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96de151e-d180-45de-8c17-a934e66ddc50-cert\") pod \"kserve-controller-manager-776cc66d47-j72kb\" (UID: \"96de151e-d180-45de-8c17-a934e66ddc50\") " pod="opendatahub/kserve-controller-manager-776cc66d47-j72kb" Oct 31 12:38:17.192194 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.192168 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf5tl\" (UniqueName: \"kubernetes.io/projected/96de151e-d180-45de-8c17-a934e66ddc50-kube-api-access-tf5tl\") pod \"kserve-controller-manager-776cc66d47-j72kb\" (UID: \"96de151e-d180-45de-8c17-a934e66ddc50\") " pod="opendatahub/kserve-controller-manager-776cc66d47-j72kb" Oct 31 12:38:17.259484 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.259398 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrwkr\" (UniqueName: \"kubernetes.io/projected/ecf5ba9f-127b-45de-969f-8128243e0bff-kube-api-access-vrwkr\") pod \"model-serving-api-75bd89fdfc-jm7lb\" (UID: \"ecf5ba9f-127b-45de-969f-8128243e0bff\") " pod="opendatahub/model-serving-api-75bd89fdfc-jm7lb" Oct 31 12:38:17.259484 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.259448 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecf5ba9f-127b-45de-969f-8128243e0bff-tls-certs\") pod \"model-serving-api-75bd89fdfc-jm7lb\" (UID: \"ecf5ba9f-127b-45de-969f-8128243e0bff\") " pod="opendatahub/model-serving-api-75bd89fdfc-jm7lb" Oct 31 12:38:17.259721 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.259510 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7c757c0-2ab1-4639-9594-42ab52830a01-metrics-certs\") pod \"odh-model-controller-769875db56-s2ldm\" (UID: \"e7c757c0-2ab1-4639-9594-42ab52830a01\") " pod="opendatahub/odh-model-controller-769875db56-s2ldm" Oct 31 12:38:17.259721 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.259537 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9lkp\" (UniqueName: \"kubernetes.io/projected/e7c757c0-2ab1-4639-9594-42ab52830a01-kube-api-access-h9lkp\") pod \"odh-model-controller-769875db56-s2ldm\" (UID: \"e7c757c0-2ab1-4639-9594-42ab52830a01\") " pod="opendatahub/odh-model-controller-769875db56-s2ldm" Oct 31 12:38:17.259721 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.259587 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e7c757c0-2ab1-4639-9594-42ab52830a01-cert\") pod \"odh-model-controller-769875db56-s2ldm\" (UID: \"e7c757c0-2ab1-4639-9594-42ab52830a01\") " pod="opendatahub/odh-model-controller-769875db56-s2ldm" Oct 31 12:38:17.262045 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.262018 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecf5ba9f-127b-45de-969f-8128243e0bff-tls-certs\") pod \"model-serving-api-75bd89fdfc-jm7lb\" (UID: \"ecf5ba9f-127b-45de-969f-8128243e0bff\") " pod="opendatahub/model-serving-api-75bd89fdfc-jm7lb" Oct 31 12:38:17.262162 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.262070 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7c757c0-2ab1-4639-9594-42ab52830a01-metrics-certs\") pod \"odh-model-controller-769875db56-s2ldm\" (UID: \"e7c757c0-2ab1-4639-9594-42ab52830a01\") " pod="opendatahub/odh-model-controller-769875db56-s2ldm" Oct 31 12:38:17.262162 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.262137 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e7c757c0-2ab1-4639-9594-42ab52830a01-cert\") pod \"odh-model-controller-769875db56-s2ldm\" (UID: \"e7c757c0-2ab1-4639-9594-42ab52830a01\") " pod="opendatahub/odh-model-controller-769875db56-s2ldm" Oct 31 12:38:17.278169 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.278142 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrwkr\" (UniqueName: \"kubernetes.io/projected/ecf5ba9f-127b-45de-969f-8128243e0bff-kube-api-access-vrwkr\") pod \"model-serving-api-75bd89fdfc-jm7lb\" (UID: \"ecf5ba9f-127b-45de-969f-8128243e0bff\") " pod="opendatahub/model-serving-api-75bd89fdfc-jm7lb" Oct 31 12:38:17.278613 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.278595 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9lkp\" (UniqueName: \"kubernetes.io/projected/e7c757c0-2ab1-4639-9594-42ab52830a01-kube-api-access-h9lkp\") pod \"odh-model-controller-769875db56-s2ldm\" (UID: \"e7c757c0-2ab1-4639-9594-42ab52830a01\") " pod="opendatahub/odh-model-controller-769875db56-s2ldm" Oct 31 12:38:17.287271 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.287245 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:17.316145 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.316120 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-j72kb" Oct 31 12:38:17.400426 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.399231 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-jm7lb" Oct 31 12:38:17.426836 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.426799 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-s2ldm" Oct 31 12:38:17.485217 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.485184 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl"] Oct 31 12:38:17.496432 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.488929 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl" Oct 31 12:38:17.496432 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.495386 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 12:38:17.499019 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.498149 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xtsw2\"" Oct 31 12:38:17.502067 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.500973 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 12:38:17.502067 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.501077 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 12:38:17.502067 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.501103 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 12:38:17.504371 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.504346 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl"] Oct 31 12:38:17.515047 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.514555 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:17.562522 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.562488 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r5tbl\" (UID: \"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl" Oct 31 12:38:17.562680 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.562528 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r5tbl\" (UID: \"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl" Oct 31 12:38:17.562680 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.562551 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r5tbl\" (UID: \"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl" Oct 31 12:38:17.562680 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.562615 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv7p8\" (UniqueName: \"kubernetes.io/projected/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-kube-api-access-xv7p8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r5tbl\" (UID: \"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl" Oct 31 12:38:17.562780 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.562751 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r5tbl\" (UID: \"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl" Oct 31 12:38:17.562817 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.562799 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r5tbl\" (UID: \"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl" Oct 31 12:38:17.562849 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.562828 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r5tbl\" (UID: \"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl" Oct 31 12:38:17.624933 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.624896 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-j72kb"] Oct 31 12:38:17.654013 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:38:17.653972 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96de151e_d180_45de_8c17_a934e66ddc50.slice/crio-09d65e9b21c41b07470b0ed6e6ccac5d695ed577002d169c681a7e2dfd2dd12a WatchSource:0}: Error finding container 09d65e9b21c41b07470b0ed6e6ccac5d695ed577002d169c681a7e2dfd2dd12a: Status 404 returned error can't find the container with id 09d65e9b21c41b07470b0ed6e6ccac5d695ed577002d169c681a7e2dfd2dd12a Oct 31 12:38:17.663579 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.663559 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r5tbl\" (UID: \"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl" Oct 31 12:38:17.663674 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.663596 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r5tbl\" (UID: \"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl" Oct 31 12:38:17.663674 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.663625 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r5tbl\" (UID: \"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl" Oct 31 12:38:17.663813 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.663792 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r5tbl\" (UID: \"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl" Oct 31 12:38:17.663875 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.663831 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r5tbl\" (UID: \"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl" Oct 31 12:38:17.663875 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.663849 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv7p8\" (UniqueName: \"kubernetes.io/projected/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-kube-api-access-xv7p8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r5tbl\" (UID: \"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl" Oct 31 12:38:17.664001 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.663884 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r5tbl\" (UID: \"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl" Oct 31 12:38:17.664613 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.664589 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r5tbl\" (UID: \"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl" Oct 31 12:38:17.666110 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.666084 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r5tbl\" (UID: \"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl" Oct 31 12:38:17.666204 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.666137 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r5tbl\" (UID: \"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl" Oct 31 12:38:17.666204 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.666166 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r5tbl\" (UID: \"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl" Oct 31 12:38:17.666204 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.666179 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r5tbl\" (UID: \"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl" Oct 31 12:38:17.701299 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.701272 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-jm7lb"] Oct 31 12:38:17.702314 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:38:17.702283 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecf5ba9f_127b_45de_969f_8128243e0bff.slice/crio-d0a036429ee1fa6585cafc6fc60001c1b17c9ef6d9d8cd82c70ee0e18780b9a5 WatchSource:0}: Error finding container d0a036429ee1fa6585cafc6fc60001c1b17c9ef6d9d8cd82c70ee0e18780b9a5: Status 404 returned error can't find the container with id d0a036429ee1fa6585cafc6fc60001c1b17c9ef6d9d8cd82c70ee0e18780b9a5 Oct 31 12:38:17.703019 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.702977 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r5tbl\" (UID: \"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl" Oct 31 12:38:17.705346 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.705326 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv7p8\" (UniqueName: \"kubernetes.io/projected/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-kube-api-access-xv7p8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r5tbl\" (UID: \"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl" Oct 31 12:38:17.729276 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.729237 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-s2ldm"] Oct 31 12:38:17.731441 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:38:17.731406 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7c757c0_2ab1_4639_9594_42ab52830a01.slice/crio-123098598f14d9be2decbbe41bb6f1c689aa8d5d82f208c7f22a0fc025c7f693 WatchSource:0}: Error finding container 123098598f14d9be2decbbe41bb6f1c689aa8d5d82f208c7f22a0fc025c7f693: Status 404 returned error can't find the container with id 123098598f14d9be2decbbe41bb6f1c689aa8d5d82f208c7f22a0fc025c7f693 Oct 31 12:38:17.810043 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:17.809962 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl" Oct 31 12:38:18.091769 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:18.091709 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl"] Oct 31 12:38:18.094449 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:38:18.094424 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc408c29a_b6d8_4b6d_96a3_a6c7c65c729a.slice/crio-ac5658f098330d046a6097237d6cad8d4c9fc69c18760e5a20f79899a8bf7bbf WatchSource:0}: Error finding container ac5658f098330d046a6097237d6cad8d4c9fc69c18760e5a20f79899a8bf7bbf: Status 404 returned error can't find the container with id ac5658f098330d046a6097237d6cad8d4c9fc69c18760e5a20f79899a8bf7bbf Oct 31 12:38:18.254103 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:18.254062 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl" event={"ID":"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a","Type":"ContainerStarted","Data":"ac5658f098330d046a6097237d6cad8d4c9fc69c18760e5a20f79899a8bf7bbf"} Oct 31 12:38:18.255687 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:18.255655 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-s2ldm" event={"ID":"e7c757c0-2ab1-4639-9594-42ab52830a01","Type":"ContainerStarted","Data":"123098598f14d9be2decbbe41bb6f1c689aa8d5d82f208c7f22a0fc025c7f693"} Oct 31 12:38:18.262211 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:18.257649 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-j72kb" event={"ID":"96de151e-d180-45de-8c17-a934e66ddc50","Type":"ContainerStarted","Data":"09d65e9b21c41b07470b0ed6e6ccac5d695ed577002d169c681a7e2dfd2dd12a"} Oct 31 12:38:18.264072 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:18.264042 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-jm7lb" event={"ID":"ecf5ba9f-127b-45de-969f-8128243e0bff","Type":"ContainerStarted","Data":"d0a036429ee1fa6585cafc6fc60001c1b17c9ef6d9d8cd82c70ee0e18780b9a5"} Oct 31 12:38:18.552682 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:18.552644 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:18.751294 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:18.751246 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:20.049945 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:38:20.049907 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:38:21.291438 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:21.291397 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-j72kb" event={"ID":"96de151e-d180-45de-8c17-a934e66ddc50","Type":"ContainerStarted","Data":"4e09923d06e276a84efb512b0584738fa8c870aaa40e8588695b3f3a5b17a701"} Oct 31 12:38:21.291910 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:21.291469 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-j72kb" Oct 31 12:38:21.293467 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:21.293438 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-jm7lb" event={"ID":"ecf5ba9f-127b-45de-969f-8128243e0bff","Type":"ContainerStarted","Data":"d0ff6c77f24e5cab27426f0d8424357daf15b6674378914de18afff405febaad"} Oct 31 12:38:21.293598 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:21.293507 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-jm7lb" Oct 31 12:38:21.312806 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:21.312746 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-j72kb" podStartSLOduration=2.364939003 podStartE2EDuration="5.312727977s" totalImagesPullingTime="2.947788974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:38:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:38:21.311810117 +0000 UTC m=+347.794933020" watchObservedRunningTime="2026-10-31 12:38:21.312727977 +0000 UTC m=+347.795850879" Oct 31 12:38:21.334499 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:21.334448 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-jm7lb" podStartSLOduration=1.810956097 podStartE2EDuration="4.334434654s" totalImagesPullingTime="2.523478557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:38:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:38:21.333599196 +0000 UTC m=+347.816722098" watchObservedRunningTime="2026-10-31 12:38:21.334434654 +0000 UTC m=+347.817557556" Oct 31 12:38:22.046968 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:22.046937 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:38:22.047061 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:22.047017 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:38:22.299670 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:22.299571 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl" event={"ID":"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a","Type":"ContainerStarted","Data":"ec47d83b2a8d004a41e2bcac35dd09bd59247e6910a753825e12c6ae207f5632"} Oct 31 12:38:22.300226 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:22.299685 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl" Oct 31 12:38:22.301242 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:22.301216 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-s2ldm" event={"ID":"e7c757c0-2ab1-4639-9594-42ab52830a01","Type":"ContainerStarted","Data":"2da78a7952617b1e6a4d7d494d7b5db70942174013b31a28b2689cfdc4afd5fe"} Oct 31 12:38:22.301413 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:22.301392 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-s2ldm" Oct 31 12:38:22.326007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:22.325952 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl" podStartSLOduration=1.376317869 podStartE2EDuration="5.325940425s" totalImagesPullingTime="3.949622556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:38:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:38:22.323895975 +0000 UTC m=+348.807018878" watchObservedRunningTime="2026-10-31 12:38:22.325940425 +0000 UTC m=+348.809063323" Oct 31 12:38:22.343211 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:22.343164 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-s2ldm" podStartSLOduration=1.7161959919999998 podStartE2EDuration="5.343151576s" totalImagesPullingTime="3.626955584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:38:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:38:22.342205121 +0000 UTC m=+348.825328045" watchObservedRunningTime="2026-10-31 12:38:22.343151576 +0000 UTC m=+348.826274477" Oct 31 12:38:23.308526 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:23.308498 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl" Oct 31 12:38:32.305496 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:32.305463 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-jm7lb" Oct 31 12:38:33.309515 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:33.309487 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-s2ldm" Oct 31 12:38:35.052993 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:35.052948 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:48.235290 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:48.235247 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:48.432911 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:48.432873 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:38:50.033752 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:38:50.033720 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:38:52.306502 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:38:52.306476 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-j72kb" Oct 31 12:39:20.033421 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:39:20.033383 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:39:35.107803 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:39:35.107766 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:39:50.033298 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:39:50.033260 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:40:20.032199 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:40:20.032153 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:40:35.158721 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:40:35.158681 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:40:50.032669 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:40:50.032629 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:41:11.629885 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:41:11.629843 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:20.033467 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:41:20.033427 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:41:21.319774 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:41:21.319738 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:34.669745 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:41:34.669701 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:34.867624 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:41:34.867590 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:35.203544 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:41:35.203513 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:41:50.033584 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:41:50.033496 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:42:20.033810 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:42:20.033773 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:42:35.266940 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:42:35.266903 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:39.629194 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:42:39.629161 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:39.831265 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:42:39.831233 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:42:50.032639 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:42:50.032603 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:43:20.033513 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:43:20.033481 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:43:28.344636 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:43:28.344601 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:28.539737 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:43:28.539700 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:31.568135 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:43:31.568091 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:35.312903 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:43:35.312861 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:43:50.033901 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:43:50.033859 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:44:20.033504 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:44:20.033469 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:44:35.378766 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:44:35.378721 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:44:50.034023 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:44:50.033924 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:45:20.033080 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:45:20.033038 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:45:35.447362 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:45:35.447320 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:45:50.033115 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:45:50.033078 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:46:20.032594 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:46:20.032556 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:46:35.485080 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:46:35.485041 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:46:50.033881 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:46:50.033844 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:47:20.033847 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:47:20.033809 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:47:35.449179 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:47:35.449137 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:47:50.032935 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:47:50.032858 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:48:00.190549 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:48:00.190503 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:00.392599 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:48:00.392562 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:01.059535 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:48:01.059492 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:01.263899 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:48:01.263854 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:19.391048 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:48:19.391015 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:19.592457 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:48:19.592399 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:20.034639 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:48:20.034594 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:48:20.173229 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:48:20.173193 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:35.596715 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:48:35.596676 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:41.251767 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:48:41.251730 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:41.457679 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:48:41.457643 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:41.654047 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:48:41.653955 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:46.376954 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:48:46.376914 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:46.586192 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:48:46.585870 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:47.952912 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:48:47.952874 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:48.152762 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:48:48.152719 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:48:50.033185 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:48:50.033156 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:48:59.724559 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:48:59.724521 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:18.174950 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:49:18.174906 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:18.380365 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:49:18.380325 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:19.466357 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:49:19.466319 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:19.671911 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:49:19.671878 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:20.033309 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:49:20.033273 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:49:31.853487 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:49:31.853434 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:32.058523 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:49:32.058484 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:35.662284 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:49:35.662248 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:50.033116 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:49:50.033080 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:49:54.451387 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:49:54.451345 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:49:54.653714 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:49:54.653657 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:09.217344 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:09.217308 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:09.426954 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:09.426923 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:13.412672 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:13.412616 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:13.629261 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:13.629228 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:20.033618 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:50:20.033584 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:50:30.638121 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:30.638080 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:35.700526 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:35.700486 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:38.391554 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:38.391500 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:38.906950 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:38.906901 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:40.188259 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:40.188225 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:40.392450 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:40.392409 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:43.740530 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:43.740491 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:48.050234 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:48.050145 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:49.082874 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:49.082833 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-bcnss"] Oct 31 12:50:49.086827 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:49.086805 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bcnss" Oct 31 12:50:49.096786 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:49.096760 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 31 12:50:49.096956 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:49.096824 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 31 12:50:49.096956 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:49.096827 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 31 12:50:49.098878 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:49.098858 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-bcnss"] Oct 31 12:50:49.100086 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:49.100066 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-26vsr\"" Oct 31 12:50:49.127476 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:49.127453 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69440ba8-a13f-4afd-a71f-dfec01ec06eb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bcnss\" (UID: \"69440ba8-a13f-4afd-a71f-dfec01ec06eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bcnss" Oct 31 12:50:49.127596 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:49.127480 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vcw7\" (UniqueName: \"kubernetes.io/projected/69440ba8-a13f-4afd-a71f-dfec01ec06eb-kube-api-access-8vcw7\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bcnss\" (UID: \"69440ba8-a13f-4afd-a71f-dfec01ec06eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bcnss" Oct 31 12:50:49.127596 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:49.127517 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69440ba8-a13f-4afd-a71f-dfec01ec06eb-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bcnss\" (UID: \"69440ba8-a13f-4afd-a71f-dfec01ec06eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bcnss" Oct 31 12:50:49.127672 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:49.127624 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69440ba8-a13f-4afd-a71f-dfec01ec06eb-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bcnss\" (UID: \"69440ba8-a13f-4afd-a71f-dfec01ec06eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bcnss" Oct 31 12:50:49.127672 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:49.127655 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69440ba8-a13f-4afd-a71f-dfec01ec06eb-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bcnss\" (UID: \"69440ba8-a13f-4afd-a71f-dfec01ec06eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bcnss" Oct 31 12:50:49.127734 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:49.127676 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69440ba8-a13f-4afd-a71f-dfec01ec06eb-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bcnss\" (UID: \"69440ba8-a13f-4afd-a71f-dfec01ec06eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bcnss" Oct 31 12:50:49.228941 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:49.228908 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69440ba8-a13f-4afd-a71f-dfec01ec06eb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bcnss\" (UID: \"69440ba8-a13f-4afd-a71f-dfec01ec06eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bcnss" Oct 31 12:50:49.228941 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:49.228946 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vcw7\" (UniqueName: \"kubernetes.io/projected/69440ba8-a13f-4afd-a71f-dfec01ec06eb-kube-api-access-8vcw7\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bcnss\" (UID: \"69440ba8-a13f-4afd-a71f-dfec01ec06eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bcnss" Oct 31 12:50:49.229231 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:49.229046 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69440ba8-a13f-4afd-a71f-dfec01ec06eb-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bcnss\" (UID: \"69440ba8-a13f-4afd-a71f-dfec01ec06eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bcnss" Oct 31 12:50:49.229231 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:49.229105 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69440ba8-a13f-4afd-a71f-dfec01ec06eb-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bcnss\" (UID: \"69440ba8-a13f-4afd-a71f-dfec01ec06eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bcnss" Oct 31 12:50:49.229231 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:49.229135 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69440ba8-a13f-4afd-a71f-dfec01ec06eb-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bcnss\" (UID: \"69440ba8-a13f-4afd-a71f-dfec01ec06eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bcnss" Oct 31 12:50:49.229231 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:49.229163 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69440ba8-a13f-4afd-a71f-dfec01ec06eb-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bcnss\" (UID: \"69440ba8-a13f-4afd-a71f-dfec01ec06eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bcnss" Oct 31 12:50:49.229441 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:49.229414 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69440ba8-a13f-4afd-a71f-dfec01ec06eb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bcnss\" (UID: \"69440ba8-a13f-4afd-a71f-dfec01ec06eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bcnss" Oct 31 12:50:49.229748 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:49.229716 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69440ba8-a13f-4afd-a71f-dfec01ec06eb-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bcnss\" (UID: \"69440ba8-a13f-4afd-a71f-dfec01ec06eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bcnss" Oct 31 12:50:49.229860 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:49.229758 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69440ba8-a13f-4afd-a71f-dfec01ec06eb-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bcnss\" (UID: \"69440ba8-a13f-4afd-a71f-dfec01ec06eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bcnss" Oct 31 12:50:49.231528 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:49.231505 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69440ba8-a13f-4afd-a71f-dfec01ec06eb-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bcnss\" (UID: \"69440ba8-a13f-4afd-a71f-dfec01ec06eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bcnss" Oct 31 12:50:49.231848 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:49.231832 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69440ba8-a13f-4afd-a71f-dfec01ec06eb-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bcnss\" (UID: \"69440ba8-a13f-4afd-a71f-dfec01ec06eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bcnss" Oct 31 12:50:49.241397 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:49.241368 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vcw7\" (UniqueName: \"kubernetes.io/projected/69440ba8-a13f-4afd-a71f-dfec01ec06eb-kube-api-access-8vcw7\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bcnss\" (UID: \"69440ba8-a13f-4afd-a71f-dfec01ec06eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bcnss" Oct 31 12:50:49.257439 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:49.257396 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:49.401157 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:49.401083 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bcnss" Oct 31 12:50:49.679100 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:49.679075 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-bcnss"] Oct 31 12:50:49.681247 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:50:49.681222 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69440ba8_a13f_4afd_a71f_dfec01ec06eb.slice/crio-1bcb32030c8ae8a48b59f7780adf9e1218f0f4f23c9c641522cbbc1e146709f1 WatchSource:0}: Error finding container 1bcb32030c8ae8a48b59f7780adf9e1218f0f4f23c9c641522cbbc1e146709f1: Status 404 returned error can't find the container with id 1bcb32030c8ae8a48b59f7780adf9e1218f0f4f23c9c641522cbbc1e146709f1 Oct 31 12:50:49.682751 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:49.682728 6841 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 12:50:49.858207 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:49.858174 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bcnss" event={"ID":"69440ba8-a13f-4afd-a71f-dfec01ec06eb","Type":"ContainerStarted","Data":"1bcb32030c8ae8a48b59f7780adf9e1218f0f4f23c9c641522cbbc1e146709f1"} Oct 31 12:50:50.041943 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:50:50.041905 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:50:50.890648 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:50.890592 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:51.876457 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:51.876424 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bcnss" event={"ID":"69440ba8-a13f-4afd-a71f-dfec01ec06eb","Type":"ContainerStarted","Data":"d1e5819d0f1bcbc0377ccac47e8c3a90908a8d0f56809e8c01f5e79641e5b62c"} Oct 31 12:50:51.901243 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:51.901188 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bcnss" podStartSLOduration=1.737967143 podStartE2EDuration="2.901174034s" totalImagesPullingTime="1.163206891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:50:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:50:51.90070707 +0000 UTC m=+1098.383829974" watchObservedRunningTime="2026-10-31 12:50:51.901174034 +0000 UTC m=+1098.384296936" Oct 31 12:50:52.385617 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:52.385573 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:52.401389 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:52.401357 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bcnss" Oct 31 12:50:53.355794 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:53.355758 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-l6cx8"] Oct 31 12:50:53.359608 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:53.359582 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-l6cx8" Oct 31 12:50:53.369476 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:53.369444 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 31 12:50:53.371431 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:53.371387 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-l6cx8"] Oct 31 12:50:53.468631 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:53.468590 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dce909fe-da4f-4d24-ad92-c4b70cce4ae2-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-l6cx8\" (UID: \"dce909fe-da4f-4d24-ad92-c4b70cce4ae2\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l6cx8" Oct 31 12:50:53.468631 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:53.468632 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dce909fe-da4f-4d24-ad92-c4b70cce4ae2-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-l6cx8\" (UID: \"dce909fe-da4f-4d24-ad92-c4b70cce4ae2\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l6cx8" Oct 31 12:50:53.468813 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:53.468716 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dce909fe-da4f-4d24-ad92-c4b70cce4ae2-home\") pod \"echo-simulator-kserve-64b6f8f96c-l6cx8\" (UID: \"dce909fe-da4f-4d24-ad92-c4b70cce4ae2\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l6cx8" Oct 31 12:50:53.468813 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:53.468775 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87tmc\" (UniqueName: \"kubernetes.io/projected/dce909fe-da4f-4d24-ad92-c4b70cce4ae2-kube-api-access-87tmc\") pod \"echo-simulator-kserve-64b6f8f96c-l6cx8\" (UID: \"dce909fe-da4f-4d24-ad92-c4b70cce4ae2\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l6cx8" Oct 31 12:50:53.468900 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:53.468846 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dce909fe-da4f-4d24-ad92-c4b70cce4ae2-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-l6cx8\" (UID: \"dce909fe-da4f-4d24-ad92-c4b70cce4ae2\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l6cx8" Oct 31 12:50:53.468900 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:53.468884 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dce909fe-da4f-4d24-ad92-c4b70cce4ae2-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-l6cx8\" (UID: \"dce909fe-da4f-4d24-ad92-c4b70cce4ae2\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l6cx8" Oct 31 12:50:53.525768 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:53.525700 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:53.570204 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:53.570178 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87tmc\" (UniqueName: \"kubernetes.io/projected/dce909fe-da4f-4d24-ad92-c4b70cce4ae2-kube-api-access-87tmc\") pod \"echo-simulator-kserve-64b6f8f96c-l6cx8\" (UID: \"dce909fe-da4f-4d24-ad92-c4b70cce4ae2\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l6cx8" Oct 31 12:50:53.570379 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:53.570214 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dce909fe-da4f-4d24-ad92-c4b70cce4ae2-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-l6cx8\" (UID: \"dce909fe-da4f-4d24-ad92-c4b70cce4ae2\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l6cx8" Oct 31 12:50:53.570379 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:53.570246 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dce909fe-da4f-4d24-ad92-c4b70cce4ae2-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-l6cx8\" (UID: \"dce909fe-da4f-4d24-ad92-c4b70cce4ae2\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l6cx8" Oct 31 12:50:53.570379 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:53.570274 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dce909fe-da4f-4d24-ad92-c4b70cce4ae2-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-l6cx8\" (UID: \"dce909fe-da4f-4d24-ad92-c4b70cce4ae2\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l6cx8" Oct 31 12:50:53.570379 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:53.570292 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dce909fe-da4f-4d24-ad92-c4b70cce4ae2-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-l6cx8\" (UID: \"dce909fe-da4f-4d24-ad92-c4b70cce4ae2\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l6cx8" Oct 31 12:50:53.570379 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:53.570325 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dce909fe-da4f-4d24-ad92-c4b70cce4ae2-home\") pod \"echo-simulator-kserve-64b6f8f96c-l6cx8\" (UID: \"dce909fe-da4f-4d24-ad92-c4b70cce4ae2\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l6cx8" Oct 31 12:50:53.570879 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:53.570745 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dce909fe-da4f-4d24-ad92-c4b70cce4ae2-home\") pod \"echo-simulator-kserve-64b6f8f96c-l6cx8\" (UID: \"dce909fe-da4f-4d24-ad92-c4b70cce4ae2\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l6cx8" Oct 31 12:50:53.570879 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:53.570789 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dce909fe-da4f-4d24-ad92-c4b70cce4ae2-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-l6cx8\" (UID: \"dce909fe-da4f-4d24-ad92-c4b70cce4ae2\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l6cx8" Oct 31 12:50:53.570879 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:53.570837 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dce909fe-da4f-4d24-ad92-c4b70cce4ae2-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-l6cx8\" (UID: \"dce909fe-da4f-4d24-ad92-c4b70cce4ae2\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l6cx8" Oct 31 12:50:53.572647 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:53.572620 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dce909fe-da4f-4d24-ad92-c4b70cce4ae2-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-l6cx8\" (UID: \"dce909fe-da4f-4d24-ad92-c4b70cce4ae2\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l6cx8" Oct 31 12:50:53.573225 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:53.573203 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dce909fe-da4f-4d24-ad92-c4b70cce4ae2-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-l6cx8\" (UID: \"dce909fe-da4f-4d24-ad92-c4b70cce4ae2\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l6cx8" Oct 31 12:50:53.583495 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:53.583471 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87tmc\" (UniqueName: \"kubernetes.io/projected/dce909fe-da4f-4d24-ad92-c4b70cce4ae2-kube-api-access-87tmc\") pod \"echo-simulator-kserve-64b6f8f96c-l6cx8\" (UID: \"dce909fe-da4f-4d24-ad92-c4b70cce4ae2\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l6cx8" Oct 31 12:50:53.677406 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:53.677338 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-l6cx8" Oct 31 12:50:53.952427 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:53.952405 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-l6cx8"] Oct 31 12:50:53.955109 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:50:53.955083 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce909fe_da4f_4d24_ad92_c4b70cce4ae2.slice/crio-f0ec2ca0141c1fba340b3d403fc508faef9fbbe8eca49a8a934a55f20430d6cd WatchSource:0}: Error finding container f0ec2ca0141c1fba340b3d403fc508faef9fbbe8eca49a8a934a55f20430d6cd: Status 404 returned error can't find the container with id f0ec2ca0141c1fba340b3d403fc508faef9fbbe8eca49a8a934a55f20430d6cd Oct 31 12:50:54.893074 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:54.893026 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-l6cx8" event={"ID":"dce909fe-da4f-4d24-ad92-c4b70cce4ae2","Type":"ContainerStarted","Data":"f0ec2ca0141c1fba340b3d403fc508faef9fbbe8eca49a8a934a55f20430d6cd"} Oct 31 12:50:55.899976 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:55.899895 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-l6cx8" event={"ID":"dce909fe-da4f-4d24-ad92-c4b70cce4ae2","Type":"ContainerStarted","Data":"c2821c6fb0c68c3c90d607bfe60662c079ba88c94566856881c0c80861b70353"} Oct 31 12:50:55.921933 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:55.921887 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-l6cx8" podStartSLOduration=1.257560054 podStartE2EDuration="2.921874258s" totalImagesPullingTime="1.664314204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:50:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:50:55.920174375 +0000 UTC m=+1102.403297282" watchObservedRunningTime="2026-10-31 12:50:55.921874258 +0000 UTC m=+1102.404997159" Oct 31 12:50:56.049830 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:56.049781 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg"] Oct 31 12:50:56.055080 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:56.055058 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg" Oct 31 12:50:56.062744 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:56.062706 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 31 12:50:56.064414 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:56.064395 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg"] Oct 31 12:50:56.093841 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:56.093814 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a38d24a-a652-4f2d-8bf1-913516c5f581-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg\" (UID: \"1a38d24a-a652-4f2d-8bf1-913516c5f581\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg" Oct 31 12:50:56.093966 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:56.093850 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a38d24a-a652-4f2d-8bf1-913516c5f581-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg\" (UID: \"1a38d24a-a652-4f2d-8bf1-913516c5f581\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg" Oct 31 12:50:56.093966 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:56.093871 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a38d24a-a652-4f2d-8bf1-913516c5f581-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg\" (UID: \"1a38d24a-a652-4f2d-8bf1-913516c5f581\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg" Oct 31 12:50:56.093966 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:56.093899 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4729\" (UniqueName: \"kubernetes.io/projected/1a38d24a-a652-4f2d-8bf1-913516c5f581-kube-api-access-b4729\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg\" (UID: \"1a38d24a-a652-4f2d-8bf1-913516c5f581\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg" Oct 31 12:50:56.094156 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:56.093976 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a38d24a-a652-4f2d-8bf1-913516c5f581-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg\" (UID: \"1a38d24a-a652-4f2d-8bf1-913516c5f581\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg" Oct 31 12:50:56.094156 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:56.094104 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a38d24a-a652-4f2d-8bf1-913516c5f581-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg\" (UID: \"1a38d24a-a652-4f2d-8bf1-913516c5f581\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg" Oct 31 12:50:56.192390 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:56.192362 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:56.195519 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:56.195498 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a38d24a-a652-4f2d-8bf1-913516c5f581-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg\" (UID: \"1a38d24a-a652-4f2d-8bf1-913516c5f581\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg" Oct 31 12:50:56.195603 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:56.195531 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a38d24a-a652-4f2d-8bf1-913516c5f581-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg\" (UID: \"1a38d24a-a652-4f2d-8bf1-913516c5f581\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg" Oct 31 12:50:56.195603 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:56.195551 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a38d24a-a652-4f2d-8bf1-913516c5f581-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg\" (UID: \"1a38d24a-a652-4f2d-8bf1-913516c5f581\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg" Oct 31 12:50:56.195710 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:56.195627 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4729\" (UniqueName: \"kubernetes.io/projected/1a38d24a-a652-4f2d-8bf1-913516c5f581-kube-api-access-b4729\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg\" (UID: \"1a38d24a-a652-4f2d-8bf1-913516c5f581\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg" Oct 31 12:50:56.195710 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:56.195666 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a38d24a-a652-4f2d-8bf1-913516c5f581-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg\" (UID: \"1a38d24a-a652-4f2d-8bf1-913516c5f581\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg" Oct 31 12:50:56.195710 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:56.195693 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a38d24a-a652-4f2d-8bf1-913516c5f581-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg\" (UID: \"1a38d24a-a652-4f2d-8bf1-913516c5f581\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg" Oct 31 12:50:56.196095 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:56.196072 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a38d24a-a652-4f2d-8bf1-913516c5f581-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg\" (UID: \"1a38d24a-a652-4f2d-8bf1-913516c5f581\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg" Oct 31 12:50:56.196095 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:56.196087 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a38d24a-a652-4f2d-8bf1-913516c5f581-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg\" (UID: \"1a38d24a-a652-4f2d-8bf1-913516c5f581\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg" Oct 31 12:50:56.196213 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:56.196119 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a38d24a-a652-4f2d-8bf1-913516c5f581-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg\" (UID: \"1a38d24a-a652-4f2d-8bf1-913516c5f581\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg" Oct 31 12:50:56.197861 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:56.197841 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a38d24a-a652-4f2d-8bf1-913516c5f581-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg\" (UID: \"1a38d24a-a652-4f2d-8bf1-913516c5f581\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg" Oct 31 12:50:56.198064 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:56.198047 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a38d24a-a652-4f2d-8bf1-913516c5f581-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg\" (UID: \"1a38d24a-a652-4f2d-8bf1-913516c5f581\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg" Oct 31 12:50:56.207503 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:56.207480 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4729\" (UniqueName: \"kubernetes.io/projected/1a38d24a-a652-4f2d-8bf1-913516c5f581-kube-api-access-b4729\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg\" (UID: \"1a38d24a-a652-4f2d-8bf1-913516c5f581\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg" Oct 31 12:50:56.369009 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:56.368952 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg" Oct 31 12:50:56.645136 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:56.645113 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg"] Oct 31 12:50:56.678109 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:56.678078 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-l6cx8" Oct 31 12:50:56.905834 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:56.905751 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg" event={"ID":"1a38d24a-a652-4f2d-8bf1-913516c5f581","Type":"ContainerStarted","Data":"686cebdadd00763fab14e74f9a35eb76667c32c7cec7b519c6c4f0906f066e07"} Oct 31 12:50:57.724755 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:57.724713 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:50:57.914892 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:57.914850 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg" event={"ID":"1a38d24a-a652-4f2d-8bf1-913516c5f581","Type":"ContainerStarted","Data":"4a3dad6ed6fc83bb0ff5a9220c9479c7d027a5894f05ceb63145cb2799c58538"} Oct 31 12:50:57.936599 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:57.936553 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg" podStartSLOduration=1.658194615 podStartE2EDuration="1.936540761s" totalImagesPullingTime="278.346146ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:50:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:50:57.934972424 +0000 UTC m=+1104.418095324" watchObservedRunningTime="2026-10-31 12:50:57.936540761 +0000 UTC m=+1104.419663698" Oct 31 12:50:58.369998 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:58.369947 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg" Oct 31 12:50:59.402133 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:59.402086 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bcnss" Oct 31 12:50:59.414625 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:59.414597 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bcnss" Oct 31 12:50:59.936862 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:50:59.936830 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bcnss" Oct 31 12:51:01.849458 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:01.849425 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx"] Oct 31 12:51:01.853129 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:01.853105 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" Oct 31 12:51:01.863583 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:01.863564 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-2bzjw\"" Oct 31 12:51:01.870172 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:01.870150 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx"] Oct 31 12:51:01.949837 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:01.949809 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f79621-82da-431c-8c50-12eb405fc80e-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-ghnhx\" (UID: \"66f79621-82da-431c-8c50-12eb405fc80e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" Oct 31 12:51:01.950018 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:01.949854 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66f79621-82da-431c-8c50-12eb405fc80e-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-ghnhx\" (UID: \"66f79621-82da-431c-8c50-12eb405fc80e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" Oct 31 12:51:01.950018 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:01.949877 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66f79621-82da-431c-8c50-12eb405fc80e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-ghnhx\" (UID: \"66f79621-82da-431c-8c50-12eb405fc80e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" Oct 31 12:51:01.950018 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:01.950011 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f79621-82da-431c-8c50-12eb405fc80e-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-ghnhx\" (UID: \"66f79621-82da-431c-8c50-12eb405fc80e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" Oct 31 12:51:01.950145 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:01.950064 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66f79621-82da-431c-8c50-12eb405fc80e-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-ghnhx\" (UID: \"66f79621-82da-431c-8c50-12eb405fc80e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" Oct 31 12:51:01.950145 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:01.950112 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66f79621-82da-431c-8c50-12eb405fc80e-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-ghnhx\" (UID: \"66f79621-82da-431c-8c50-12eb405fc80e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" Oct 31 12:51:01.950145 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:01.950138 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66f79621-82da-431c-8c50-12eb405fc80e-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-ghnhx\" (UID: \"66f79621-82da-431c-8c50-12eb405fc80e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" Oct 31 12:51:01.950262 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:01.950247 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qz4z\" (UniqueName: \"kubernetes.io/projected/66f79621-82da-431c-8c50-12eb405fc80e-kube-api-access-6qz4z\") pod \"maas-default-gateway-openshift-default-fc8c56547-ghnhx\" (UID: \"66f79621-82da-431c-8c50-12eb405fc80e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" Oct 31 12:51:01.950296 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:01.950270 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66f79621-82da-431c-8c50-12eb405fc80e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-ghnhx\" (UID: \"66f79621-82da-431c-8c50-12eb405fc80e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" Oct 31 12:51:02.050898 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:02.050861 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66f79621-82da-431c-8c50-12eb405fc80e-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-ghnhx\" (UID: \"66f79621-82da-431c-8c50-12eb405fc80e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" Oct 31 12:51:02.050898 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:02.050902 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66f79621-82da-431c-8c50-12eb405fc80e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-ghnhx\" (UID: \"66f79621-82da-431c-8c50-12eb405fc80e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" Oct 31 12:51:02.051231 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:02.050927 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f79621-82da-431c-8c50-12eb405fc80e-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-ghnhx\" (UID: \"66f79621-82da-431c-8c50-12eb405fc80e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" Oct 31 12:51:02.051231 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:02.050977 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66f79621-82da-431c-8c50-12eb405fc80e-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-ghnhx\" (UID: \"66f79621-82da-431c-8c50-12eb405fc80e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" Oct 31 12:51:02.051231 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:02.051040 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66f79621-82da-431c-8c50-12eb405fc80e-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-ghnhx\" (UID: \"66f79621-82da-431c-8c50-12eb405fc80e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" Oct 31 12:51:02.051231 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:02.051065 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66f79621-82da-431c-8c50-12eb405fc80e-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-ghnhx\" (UID: \"66f79621-82da-431c-8c50-12eb405fc80e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" Oct 31 12:51:02.051231 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:02.051115 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qz4z\" (UniqueName: \"kubernetes.io/projected/66f79621-82da-431c-8c50-12eb405fc80e-kube-api-access-6qz4z\") pod \"maas-default-gateway-openshift-default-fc8c56547-ghnhx\" (UID: \"66f79621-82da-431c-8c50-12eb405fc80e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" Oct 31 12:51:02.051231 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:02.051144 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66f79621-82da-431c-8c50-12eb405fc80e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-ghnhx\" (UID: \"66f79621-82da-431c-8c50-12eb405fc80e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" Oct 31 12:51:02.051231 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:02.051170 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f79621-82da-431c-8c50-12eb405fc80e-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-ghnhx\" (UID: \"66f79621-82da-431c-8c50-12eb405fc80e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" Oct 31 12:51:02.051593 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:02.051326 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66f79621-82da-431c-8c50-12eb405fc80e-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-ghnhx\" (UID: \"66f79621-82da-431c-8c50-12eb405fc80e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" Oct 31 12:51:02.051593 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:02.051452 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66f79621-82da-431c-8c50-12eb405fc80e-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-ghnhx\" (UID: \"66f79621-82da-431c-8c50-12eb405fc80e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" Oct 31 12:51:02.051593 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:02.051538 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f79621-82da-431c-8c50-12eb405fc80e-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-ghnhx\" (UID: \"66f79621-82da-431c-8c50-12eb405fc80e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" Oct 31 12:51:02.051736 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:02.051659 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f79621-82da-431c-8c50-12eb405fc80e-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-ghnhx\" (UID: \"66f79621-82da-431c-8c50-12eb405fc80e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" Oct 31 12:51:02.051736 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:02.051671 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66f79621-82da-431c-8c50-12eb405fc80e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-ghnhx\" (UID: \"66f79621-82da-431c-8c50-12eb405fc80e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" Oct 31 12:51:02.053290 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:02.053266 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66f79621-82da-431c-8c50-12eb405fc80e-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-ghnhx\" (UID: \"66f79621-82da-431c-8c50-12eb405fc80e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" Oct 31 12:51:02.053542 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:02.053522 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66f79621-82da-431c-8c50-12eb405fc80e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-ghnhx\" (UID: \"66f79621-82da-431c-8c50-12eb405fc80e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" Oct 31 12:51:02.063079 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:02.063057 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66f79621-82da-431c-8c50-12eb405fc80e-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-ghnhx\" (UID: \"66f79621-82da-431c-8c50-12eb405fc80e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" Oct 31 12:51:02.063314 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:02.063296 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qz4z\" (UniqueName: \"kubernetes.io/projected/66f79621-82da-431c-8c50-12eb405fc80e-kube-api-access-6qz4z\") pod \"maas-default-gateway-openshift-default-fc8c56547-ghnhx\" (UID: \"66f79621-82da-431c-8c50-12eb405fc80e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" Oct 31 12:51:02.166157 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:02.166076 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" Oct 31 12:51:02.483423 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:02.483394 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx"] Oct 31 12:51:02.484922 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:51:02.484888 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66f79621_82da_431c_8c50_12eb405fc80e.slice/crio-80fb4791d4140e35975193ef8dd0938c53166400e9b1b8dd1870003cc1f2f2ab WatchSource:0}: Error finding container 80fb4791d4140e35975193ef8dd0938c53166400e9b1b8dd1870003cc1f2f2ab: Status 404 returned error can't find the container with id 80fb4791d4140e35975193ef8dd0938c53166400e9b1b8dd1870003cc1f2f2ab Oct 31 12:51:02.942482 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:02.942453 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" event={"ID":"66f79621-82da-431c-8c50-12eb405fc80e","Type":"ContainerStarted","Data":"80fb4791d4140e35975193ef8dd0938c53166400e9b1b8dd1870003cc1f2f2ab"} Oct 31 12:51:03.677689 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:03.677660 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-l6cx8" Oct 31 12:51:03.690010 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:03.689969 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-l6cx8" Oct 31 12:51:03.959765 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:03.959739 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-l6cx8" Oct 31 12:51:06.370155 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:06.370124 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg" Oct 31 12:51:06.384345 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:06.384315 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg" Oct 31 12:51:06.722883 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:06.722846 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:51:06.723040 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:06.722929 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:51:06.723040 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:06.722973 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:51:06.971157 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:06.971064 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" event={"ID":"66f79621-82da-431c-8c50-12eb405fc80e","Type":"ContainerStarted","Data":"11e28327d752bee8b194d902b9a722da8c96e5e59de7a3e66cb238b33c1f1a37"} Oct 31 12:51:06.982597 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:06.982573 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg" Oct 31 12:51:06.994566 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:06.994524 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" podStartSLOduration=1.759136742 podStartE2EDuration="5.99451022s" totalImagesPullingTime="4.235373478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:51:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:51:06.993735966 +0000 UTC m=+1113.476858868" watchObservedRunningTime="2026-10-31 12:51:06.99451022 +0000 UTC m=+1113.477633185" Oct 31 12:51:07.167159 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:07.167123 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" Oct 31 12:51:07.168426 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:07.168397 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-ghnhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 31 12:51:07.168578 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:07.168496 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" podUID="66f79621-82da-431c-8c50-12eb405fc80e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 31 12:51:08.166588 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:08.166550 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-ghnhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 31 12:51:08.167008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:08.166609 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" podUID="66f79621-82da-431c-8c50-12eb405fc80e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 31 12:51:09.167497 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:09.167452 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-ghnhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 31 12:51:09.167899 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:09.167538 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" podUID="66f79621-82da-431c-8c50-12eb405fc80e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 31 12:51:10.166568 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:10.166532 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-ghnhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 31 12:51:10.166753 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:10.166603 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" podUID="66f79621-82da-431c-8c50-12eb405fc80e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 31 12:51:11.167493 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:11.167455 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-ghnhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 31 12:51:11.167888 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:11.167513 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" podUID="66f79621-82da-431c-8c50-12eb405fc80e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 31 12:51:12.166555 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:12.166519 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-ghnhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 31 12:51:12.166775 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:12.166578 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" podUID="66f79621-82da-431c-8c50-12eb405fc80e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 31 12:51:13.166858 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:13.166821 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-ghnhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 31 12:51:13.167380 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:13.166895 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" podUID="66f79621-82da-431c-8c50-12eb405fc80e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 31 12:51:14.167235 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:14.167196 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-ghnhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 31 12:51:14.167612 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:14.167252 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" podUID="66f79621-82da-431c-8c50-12eb405fc80e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 31 12:51:15.170847 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:15.170816 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" Oct 31 12:51:15.171300 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:15.171103 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" Oct 31 12:51:15.171784 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:15.171766 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" Oct 31 12:51:15.434388 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:15.434303 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:15.654427 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:15.654394 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:16.566992 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:16.566956 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:16.774717 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:16.774689 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:20.033955 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:51:20.033920 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:51:31.920354 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:31.920317 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb"] Oct 31 12:51:31.923822 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:31.923805 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" Oct 31 12:51:31.942427 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:31.942402 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ad44f3-de70-4b31-8d39-ceeed24efd4a-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-6lxfb\" (UID: \"d0ad44f3-de70-4b31-8d39-ceeed24efd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" Oct 31 12:51:31.942649 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:31.942621 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0ad44f3-de70-4b31-8d39-ceeed24efd4a-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-6lxfb\" (UID: \"d0ad44f3-de70-4b31-8d39-ceeed24efd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" Oct 31 12:51:31.942790 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:31.942662 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0ad44f3-de70-4b31-8d39-ceeed24efd4a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-6lxfb\" (UID: \"d0ad44f3-de70-4b31-8d39-ceeed24efd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" Oct 31 12:51:31.942790 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:31.942707 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ad44f3-de70-4b31-8d39-ceeed24efd4a-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-6lxfb\" (UID: \"d0ad44f3-de70-4b31-8d39-ceeed24efd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" Oct 31 12:51:31.942903 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:31.942803 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0ad44f3-de70-4b31-8d39-ceeed24efd4a-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-6lxfb\" (UID: \"d0ad44f3-de70-4b31-8d39-ceeed24efd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" Oct 31 12:51:31.942940 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:31.942903 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0ad44f3-de70-4b31-8d39-ceeed24efd4a-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-6lxfb\" (UID: \"d0ad44f3-de70-4b31-8d39-ceeed24efd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" Oct 31 12:51:31.943015 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:31.942941 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0ad44f3-de70-4b31-8d39-ceeed24efd4a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-6lxfb\" (UID: \"d0ad44f3-de70-4b31-8d39-ceeed24efd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" Oct 31 12:51:31.943061 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:31.943018 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0ad44f3-de70-4b31-8d39-ceeed24efd4a-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-6lxfb\" (UID: \"d0ad44f3-de70-4b31-8d39-ceeed24efd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" Oct 31 12:51:31.943106 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:31.943068 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98htp\" (UniqueName: \"kubernetes.io/projected/d0ad44f3-de70-4b31-8d39-ceeed24efd4a-kube-api-access-98htp\") pod \"maas-default-gateway-openshift-default-fc8c56547-6lxfb\" (UID: \"d0ad44f3-de70-4b31-8d39-ceeed24efd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" Oct 31 12:51:31.948728 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:31.948702 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb"] Oct 31 12:51:31.983092 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:31.983059 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx"] Oct 31 12:51:31.987240 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:31.987217 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" Oct 31 12:51:32.001492 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.001469 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx"] Oct 31 12:51:32.043488 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.043457 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ad44f3-de70-4b31-8d39-ceeed24efd4a-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-6lxfb\" (UID: \"d0ad44f3-de70-4b31-8d39-ceeed24efd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" Oct 31 12:51:32.043663 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.043495 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa7e0a48-9dc5-4735-b1ae-cafd28ff982f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-fk2tx\" (UID: \"aa7e0a48-9dc5-4735-b1ae-cafd28ff982f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" Oct 31 12:51:32.043663 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.043515 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa7e0a48-9dc5-4735-b1ae-cafd28ff982f-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-fk2tx\" (UID: \"aa7e0a48-9dc5-4735-b1ae-cafd28ff982f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" Oct 31 12:51:32.043663 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.043532 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa7e0a48-9dc5-4735-b1ae-cafd28ff982f-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-fk2tx\" (UID: \"aa7e0a48-9dc5-4735-b1ae-cafd28ff982f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" Oct 31 12:51:32.043849 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.043673 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0ad44f3-de70-4b31-8d39-ceeed24efd4a-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-6lxfb\" (UID: \"d0ad44f3-de70-4b31-8d39-ceeed24efd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" Oct 31 12:51:32.043849 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.043722 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa7e0a48-9dc5-4735-b1ae-cafd28ff982f-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-fk2tx\" (UID: \"aa7e0a48-9dc5-4735-b1ae-cafd28ff982f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" Oct 31 12:51:32.043849 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.043750 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0ad44f3-de70-4b31-8d39-ceeed24efd4a-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-6lxfb\" (UID: \"d0ad44f3-de70-4b31-8d39-ceeed24efd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" Oct 31 12:51:32.043849 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.043831 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0ad44f3-de70-4b31-8d39-ceeed24efd4a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-6lxfb\" (UID: \"d0ad44f3-de70-4b31-8d39-ceeed24efd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" Oct 31 12:51:32.044091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.043853 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ad44f3-de70-4b31-8d39-ceeed24efd4a-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-6lxfb\" (UID: \"d0ad44f3-de70-4b31-8d39-ceeed24efd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" Oct 31 12:51:32.044091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.043871 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa7e0a48-9dc5-4735-b1ae-cafd28ff982f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-fk2tx\" (UID: \"aa7e0a48-9dc5-4735-b1ae-cafd28ff982f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" Oct 31 12:51:32.044091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.043905 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0ad44f3-de70-4b31-8d39-ceeed24efd4a-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-6lxfb\" (UID: \"d0ad44f3-de70-4b31-8d39-ceeed24efd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" Oct 31 12:51:32.044091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.043997 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98htp\" (UniqueName: \"kubernetes.io/projected/d0ad44f3-de70-4b31-8d39-ceeed24efd4a-kube-api-access-98htp\") pod \"maas-default-gateway-openshift-default-fc8c56547-6lxfb\" (UID: \"d0ad44f3-de70-4b31-8d39-ceeed24efd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" Oct 31 12:51:32.044091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.044039 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa7e0a48-9dc5-4735-b1ae-cafd28ff982f-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-fk2tx\" (UID: \"aa7e0a48-9dc5-4735-b1ae-cafd28ff982f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" Oct 31 12:51:32.044091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.044069 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ad44f3-de70-4b31-8d39-ceeed24efd4a-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-6lxfb\" (UID: \"d0ad44f3-de70-4b31-8d39-ceeed24efd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" Oct 31 12:51:32.044405 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.044096 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0ad44f3-de70-4b31-8d39-ceeed24efd4a-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-6lxfb\" (UID: \"d0ad44f3-de70-4b31-8d39-ceeed24efd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" Oct 31 12:51:32.044405 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.044150 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa7e0a48-9dc5-4735-b1ae-cafd28ff982f-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-fk2tx\" (UID: \"aa7e0a48-9dc5-4735-b1ae-cafd28ff982f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" Oct 31 12:51:32.044405 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.044173 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0ad44f3-de70-4b31-8d39-ceeed24efd4a-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-6lxfb\" (UID: \"d0ad44f3-de70-4b31-8d39-ceeed24efd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" Oct 31 12:51:32.044405 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.044188 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0ad44f3-de70-4b31-8d39-ceeed24efd4a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-6lxfb\" (UID: \"d0ad44f3-de70-4b31-8d39-ceeed24efd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" Oct 31 12:51:32.044405 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.044249 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa7e0a48-9dc5-4735-b1ae-cafd28ff982f-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-fk2tx\" (UID: \"aa7e0a48-9dc5-4735-b1ae-cafd28ff982f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" Oct 31 12:51:32.044405 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.044276 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9rrj\" (UniqueName: \"kubernetes.io/projected/aa7e0a48-9dc5-4735-b1ae-cafd28ff982f-kube-api-access-s9rrj\") pod \"maas-default-gateway-openshift-default-fc8c56547-fk2tx\" (UID: \"aa7e0a48-9dc5-4735-b1ae-cafd28ff982f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" Oct 31 12:51:32.044716 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.044437 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0ad44f3-de70-4b31-8d39-ceeed24efd4a-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-6lxfb\" (UID: \"d0ad44f3-de70-4b31-8d39-ceeed24efd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" Oct 31 12:51:32.044716 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.044503 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ad44f3-de70-4b31-8d39-ceeed24efd4a-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-6lxfb\" (UID: \"d0ad44f3-de70-4b31-8d39-ceeed24efd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" Oct 31 12:51:32.044716 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.044535 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0ad44f3-de70-4b31-8d39-ceeed24efd4a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-6lxfb\" (UID: \"d0ad44f3-de70-4b31-8d39-ceeed24efd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" Oct 31 12:51:32.046517 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.046492 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0ad44f3-de70-4b31-8d39-ceeed24efd4a-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-6lxfb\" (UID: \"d0ad44f3-de70-4b31-8d39-ceeed24efd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" Oct 31 12:51:32.046639 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.046614 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0ad44f3-de70-4b31-8d39-ceeed24efd4a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-6lxfb\" (UID: \"d0ad44f3-de70-4b31-8d39-ceeed24efd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" Oct 31 12:51:32.058778 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.058757 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98htp\" (UniqueName: \"kubernetes.io/projected/d0ad44f3-de70-4b31-8d39-ceeed24efd4a-kube-api-access-98htp\") pod \"maas-default-gateway-openshift-default-fc8c56547-6lxfb\" (UID: \"d0ad44f3-de70-4b31-8d39-ceeed24efd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" Oct 31 12:51:32.064344 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.064321 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0ad44f3-de70-4b31-8d39-ceeed24efd4a-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-6lxfb\" (UID: \"d0ad44f3-de70-4b31-8d39-ceeed24efd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" Oct 31 12:51:32.144845 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.144814 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa7e0a48-9dc5-4735-b1ae-cafd28ff982f-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-fk2tx\" (UID: \"aa7e0a48-9dc5-4735-b1ae-cafd28ff982f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" Oct 31 12:51:32.144845 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.144848 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa7e0a48-9dc5-4735-b1ae-cafd28ff982f-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-fk2tx\" (UID: \"aa7e0a48-9dc5-4735-b1ae-cafd28ff982f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" Oct 31 12:51:32.145122 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.144870 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa7e0a48-9dc5-4735-b1ae-cafd28ff982f-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-fk2tx\" (UID: \"aa7e0a48-9dc5-4735-b1ae-cafd28ff982f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" Oct 31 12:51:32.145122 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.145002 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9rrj\" (UniqueName: \"kubernetes.io/projected/aa7e0a48-9dc5-4735-b1ae-cafd28ff982f-kube-api-access-s9rrj\") pod \"maas-default-gateway-openshift-default-fc8c56547-fk2tx\" (UID: \"aa7e0a48-9dc5-4735-b1ae-cafd28ff982f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" Oct 31 12:51:32.145122 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.145053 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa7e0a48-9dc5-4735-b1ae-cafd28ff982f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-fk2tx\" (UID: \"aa7e0a48-9dc5-4735-b1ae-cafd28ff982f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" Oct 31 12:51:32.145122 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.145075 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa7e0a48-9dc5-4735-b1ae-cafd28ff982f-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-fk2tx\" (UID: \"aa7e0a48-9dc5-4735-b1ae-cafd28ff982f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" Oct 31 12:51:32.145122 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.145095 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa7e0a48-9dc5-4735-b1ae-cafd28ff982f-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-fk2tx\" (UID: \"aa7e0a48-9dc5-4735-b1ae-cafd28ff982f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" Oct 31 12:51:32.145379 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.145183 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa7e0a48-9dc5-4735-b1ae-cafd28ff982f-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-fk2tx\" (UID: \"aa7e0a48-9dc5-4735-b1ae-cafd28ff982f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" Oct 31 12:51:32.145379 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.145242 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa7e0a48-9dc5-4735-b1ae-cafd28ff982f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-fk2tx\" (UID: \"aa7e0a48-9dc5-4735-b1ae-cafd28ff982f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" Oct 31 12:51:32.145379 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.145357 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa7e0a48-9dc5-4735-b1ae-cafd28ff982f-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-fk2tx\" (UID: \"aa7e0a48-9dc5-4735-b1ae-cafd28ff982f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" Oct 31 12:51:32.145496 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.145450 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa7e0a48-9dc5-4735-b1ae-cafd28ff982f-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-fk2tx\" (UID: \"aa7e0a48-9dc5-4735-b1ae-cafd28ff982f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" Oct 31 12:51:32.145593 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.145566 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa7e0a48-9dc5-4735-b1ae-cafd28ff982f-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-fk2tx\" (UID: \"aa7e0a48-9dc5-4735-b1ae-cafd28ff982f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" Oct 31 12:51:32.145722 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.145653 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa7e0a48-9dc5-4735-b1ae-cafd28ff982f-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-fk2tx\" (UID: \"aa7e0a48-9dc5-4735-b1ae-cafd28ff982f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" Oct 31 12:51:32.145900 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.145870 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa7e0a48-9dc5-4735-b1ae-cafd28ff982f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-fk2tx\" (UID: \"aa7e0a48-9dc5-4735-b1ae-cafd28ff982f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" Oct 31 12:51:32.147195 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.147175 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa7e0a48-9dc5-4735-b1ae-cafd28ff982f-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-fk2tx\" (UID: \"aa7e0a48-9dc5-4735-b1ae-cafd28ff982f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" Oct 31 12:51:32.147600 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.147584 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa7e0a48-9dc5-4735-b1ae-cafd28ff982f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-fk2tx\" (UID: \"aa7e0a48-9dc5-4735-b1ae-cafd28ff982f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" Oct 31 12:51:32.157944 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.157925 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa7e0a48-9dc5-4735-b1ae-cafd28ff982f-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-fk2tx\" (UID: \"aa7e0a48-9dc5-4735-b1ae-cafd28ff982f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" Oct 31 12:51:32.158052 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.158028 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9rrj\" (UniqueName: \"kubernetes.io/projected/aa7e0a48-9dc5-4735-b1ae-cafd28ff982f-kube-api-access-s9rrj\") pod \"maas-default-gateway-openshift-default-fc8c56547-fk2tx\" (UID: \"aa7e0a48-9dc5-4735-b1ae-cafd28ff982f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" Oct 31 12:51:32.236737 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.236701 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" Oct 31 12:51:32.307577 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.307510 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" Oct 31 12:51:32.517407 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.517379 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb"] Oct 31 12:51:32.518974 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:51:32.518944 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ad44f3_de70_4b31_8d39_ceeed24efd4a.slice/crio-aa302a305ae41772f0709ca699a5b1106f2bed2324a33eb8fdd09466e2c24280 WatchSource:0}: Error finding container aa302a305ae41772f0709ca699a5b1106f2bed2324a33eb8fdd09466e2c24280: Status 404 returned error can't find the container with id aa302a305ae41772f0709ca699a5b1106f2bed2324a33eb8fdd09466e2c24280 Oct 31 12:51:32.522084 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.522054 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:51:32.522175 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.522114 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:51:32.522175 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.522150 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:51:32.593513 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.593485 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx"] Oct 31 12:51:32.594585 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:51:32.594563 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa7e0a48_9dc5_4735_b1ae_cafd28ff982f.slice/crio-79db0cf42000a58417b3bcb72902503956e5baca7ad184fc38155f44fcad31d6 WatchSource:0}: Error finding container 79db0cf42000a58417b3bcb72902503956e5baca7ad184fc38155f44fcad31d6: Status 404 returned error can't find the container with id 79db0cf42000a58417b3bcb72902503956e5baca7ad184fc38155f44fcad31d6 Oct 31 12:51:32.597724 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.597685 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:51:32.597822 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.597758 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:51:32.597822 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:32.597785 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:51:33.103583 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:33.103545 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" event={"ID":"aa7e0a48-9dc5-4735-b1ae-cafd28ff982f","Type":"ContainerStarted","Data":"f5dd3528da938a87d9ee40f71f6e425511c2b6cccd72bddf632a80b3b45ddf64"} Oct 31 12:51:33.104587 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:33.104559 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" event={"ID":"aa7e0a48-9dc5-4735-b1ae-cafd28ff982f","Type":"ContainerStarted","Data":"79db0cf42000a58417b3bcb72902503956e5baca7ad184fc38155f44fcad31d6"} Oct 31 12:51:33.113078 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:33.113044 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" event={"ID":"d0ad44f3-de70-4b31-8d39-ceeed24efd4a","Type":"ContainerStarted","Data":"b2b294349309839f4127ce8578816223d50d979572a28b3704dfc41bb8f22fd1"} Oct 31 12:51:33.113375 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:33.113334 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" event={"ID":"d0ad44f3-de70-4b31-8d39-ceeed24efd4a","Type":"ContainerStarted","Data":"aa302a305ae41772f0709ca699a5b1106f2bed2324a33eb8fdd09466e2c24280"} Oct 31 12:51:33.133457 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:33.133402 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" podStartSLOduration=2.133387237 podStartE2EDuration="2.133387237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:51:33.131976946 +0000 UTC m=+1139.615099848" watchObservedRunningTime="2026-10-31 12:51:33.133387237 +0000 UTC m=+1139.616510222" Oct 31 12:51:33.237320 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:33.237285 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" Oct 31 12:51:33.239012 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:33.238975 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-6lxfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 31 12:51:33.239139 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:33.239035 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" podUID="d0ad44f3-de70-4b31-8d39-ceeed24efd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 31 12:51:33.307884 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:33.307855 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" Oct 31 12:51:33.309585 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:33.309330 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fk2tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 12:51:33.309585 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:33.309382 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" podUID="aa7e0a48-9dc5-4735-b1ae-cafd28ff982f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 12:51:34.237537 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:34.237503 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-6lxfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 31 12:51:34.237940 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:34.237555 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" podUID="d0ad44f3-de70-4b31-8d39-ceeed24efd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 31 12:51:34.308422 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:34.308379 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fk2tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 12:51:34.308624 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:34.308440 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" podUID="aa7e0a48-9dc5-4735-b1ae-cafd28ff982f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 12:51:35.237778 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:35.237742 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-6lxfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 31 12:51:35.238186 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:35.237806 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" podUID="d0ad44f3-de70-4b31-8d39-ceeed24efd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 31 12:51:35.308278 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:35.308237 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fk2tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 12:51:35.308523 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:35.308492 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" podUID="aa7e0a48-9dc5-4735-b1ae-cafd28ff982f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 12:51:35.781006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:35.772669 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:36.237086 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:36.237052 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-6lxfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 31 12:51:36.237258 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:36.237117 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" podUID="d0ad44f3-de70-4b31-8d39-ceeed24efd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 31 12:51:36.308135 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:36.308102 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fk2tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 12:51:36.308540 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:36.308151 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" podUID="aa7e0a48-9dc5-4735-b1ae-cafd28ff982f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 12:51:37.237421 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:37.237387 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-6lxfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 31 12:51:37.237623 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:37.237443 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" podUID="d0ad44f3-de70-4b31-8d39-ceeed24efd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 31 12:51:37.308030 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:37.307999 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fk2tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 12:51:37.308203 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:37.308056 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" podUID="aa7e0a48-9dc5-4735-b1ae-cafd28ff982f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 12:51:38.238073 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:38.238039 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-6lxfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 31 12:51:38.238332 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:38.238308 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" podUID="d0ad44f3-de70-4b31-8d39-ceeed24efd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 31 12:51:38.307852 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:38.307816 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fk2tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 12:51:38.308041 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:38.307863 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" podUID="aa7e0a48-9dc5-4735-b1ae-cafd28ff982f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 12:51:39.237595 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:39.237558 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-6lxfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 31 12:51:39.238021 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:39.237624 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" podUID="d0ad44f3-de70-4b31-8d39-ceeed24efd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 31 12:51:39.308627 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:39.308588 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fk2tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 12:51:39.308794 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:39.308656 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" podUID="aa7e0a48-9dc5-4735-b1ae-cafd28ff982f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 12:51:40.834001 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:40.833953 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" Oct 31 12:51:40.834470 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:40.834372 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" Oct 31 12:51:40.835097 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:40.835073 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" Oct 31 12:51:40.873900 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:40.873833 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" podStartSLOduration=9.873815731 podStartE2EDuration="9.873815731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:51:33.160429886 +0000 UTC m=+1139.643552784" watchObservedRunningTime="2026-10-31 12:51:40.873815731 +0000 UTC m=+1147.356938632" Oct 31 12:51:41.001862 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:41.001823 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" Oct 31 12:51:41.002301 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:41.002275 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" Oct 31 12:51:41.003336 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:41.003313 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" Oct 31 12:51:41.904907 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:41.904871 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:42.427831 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.427798 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj"] Oct 31 12:51:42.431643 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.431623 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:51:42.441146 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.441121 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b03bbf98-openshift-default-dockercfg-tvn2t\"" Oct 31 12:51:42.455386 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.455357 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj"] Oct 31 12:51:42.520915 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.520881 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a039273-4b94-4793-b713-1524da46eed6-istiod-ca-cert\") pod \"e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj\" (UID: \"0a039273-4b94-4793-b713-1524da46eed6\") " pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:51:42.521164 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.520922 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a039273-4b94-4793-b713-1524da46eed6-istio-data\") pod \"e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj\" (UID: \"0a039273-4b94-4793-b713-1524da46eed6\") " pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:51:42.521164 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.521040 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a039273-4b94-4793-b713-1524da46eed6-workload-certs\") pod \"e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj\" (UID: \"0a039273-4b94-4793-b713-1524da46eed6\") " pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:51:42.521164 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.521086 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a039273-4b94-4793-b713-1524da46eed6-istio-podinfo\") pod \"e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj\" (UID: \"0a039273-4b94-4793-b713-1524da46eed6\") " pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:51:42.521164 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.521152 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a039273-4b94-4793-b713-1524da46eed6-istio-token\") pod \"e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj\" (UID: \"0a039273-4b94-4793-b713-1524da46eed6\") " pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:51:42.521404 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.521218 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckfnb\" (UniqueName: \"kubernetes.io/projected/0a039273-4b94-4793-b713-1524da46eed6-kube-api-access-ckfnb\") pod \"e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj\" (UID: \"0a039273-4b94-4793-b713-1524da46eed6\") " pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:51:42.521404 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.521248 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a039273-4b94-4793-b713-1524da46eed6-credential-socket\") pod \"e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj\" (UID: \"0a039273-4b94-4793-b713-1524da46eed6\") " pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:51:42.521404 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.521285 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a039273-4b94-4793-b713-1524da46eed6-workload-socket\") pod \"e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj\" (UID: \"0a039273-4b94-4793-b713-1524da46eed6\") " pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:51:42.521404 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.521358 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a039273-4b94-4793-b713-1524da46eed6-istio-envoy\") pod \"e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj\" (UID: \"0a039273-4b94-4793-b713-1524da46eed6\") " pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:51:42.548844 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.548809 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr"] Oct 31 12:51:42.557266 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.557235 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:51:42.566460 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.566432 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr"] Oct 31 12:51:42.567083 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.567063 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d48f454a-openshift-default-dockercfg-6zjbw\"" Oct 31 12:51:42.622671 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.622636 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a039273-4b94-4793-b713-1524da46eed6-istio-podinfo\") pod \"e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj\" (UID: \"0a039273-4b94-4793-b713-1524da46eed6\") " pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:51:42.622846 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.622678 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a039273-4b94-4793-b713-1524da46eed6-istio-token\") pod \"e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj\" (UID: \"0a039273-4b94-4793-b713-1524da46eed6\") " pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:51:42.622846 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.622717 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckfnb\" (UniqueName: \"kubernetes.io/projected/0a039273-4b94-4793-b713-1524da46eed6-kube-api-access-ckfnb\") pod \"e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj\" (UID: \"0a039273-4b94-4793-b713-1524da46eed6\") " pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:51:42.622846 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.622755 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a039273-4b94-4793-b713-1524da46eed6-credential-socket\") pod \"e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj\" (UID: \"0a039273-4b94-4793-b713-1524da46eed6\") " pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:51:42.622846 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.622790 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a039273-4b94-4793-b713-1524da46eed6-workload-socket\") pod \"e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj\" (UID: \"0a039273-4b94-4793-b713-1524da46eed6\") " pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:51:42.622846 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.622840 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a039273-4b94-4793-b713-1524da46eed6-istio-envoy\") pod \"e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj\" (UID: \"0a039273-4b94-4793-b713-1524da46eed6\") " pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:51:42.623153 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.622901 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a039273-4b94-4793-b713-1524da46eed6-istiod-ca-cert\") pod \"e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj\" (UID: \"0a039273-4b94-4793-b713-1524da46eed6\") " pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:51:42.623153 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.622940 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a039273-4b94-4793-b713-1524da46eed6-istio-data\") pod \"e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj\" (UID: \"0a039273-4b94-4793-b713-1524da46eed6\") " pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:51:42.623153 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.623026 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a039273-4b94-4793-b713-1524da46eed6-workload-certs\") pod \"e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj\" (UID: \"0a039273-4b94-4793-b713-1524da46eed6\") " pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:51:42.623389 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.623362 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a039273-4b94-4793-b713-1524da46eed6-workload-socket\") pod \"e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj\" (UID: \"0a039273-4b94-4793-b713-1524da46eed6\") " pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:51:42.623526 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.623507 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a039273-4b94-4793-b713-1524da46eed6-workload-certs\") pod \"e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj\" (UID: \"0a039273-4b94-4793-b713-1524da46eed6\") " pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:51:42.623664 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.623643 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a039273-4b94-4793-b713-1524da46eed6-credential-socket\") pod \"e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj\" (UID: \"0a039273-4b94-4793-b713-1524da46eed6\") " pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:51:42.624121 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.624096 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a039273-4b94-4793-b713-1524da46eed6-istiod-ca-cert\") pod \"e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj\" (UID: \"0a039273-4b94-4793-b713-1524da46eed6\") " pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:51:42.625339 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.625318 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a039273-4b94-4793-b713-1524da46eed6-istio-data\") pod \"e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj\" (UID: \"0a039273-4b94-4793-b713-1524da46eed6\") " pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:51:42.625860 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.625836 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a039273-4b94-4793-b713-1524da46eed6-istio-envoy\") pod \"e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj\" (UID: \"0a039273-4b94-4793-b713-1524da46eed6\") " pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:51:42.626143 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.626123 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a039273-4b94-4793-b713-1524da46eed6-istio-podinfo\") pod \"e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj\" (UID: \"0a039273-4b94-4793-b713-1524da46eed6\") " pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:51:42.638431 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.638402 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckfnb\" (UniqueName: \"kubernetes.io/projected/0a039273-4b94-4793-b713-1524da46eed6-kube-api-access-ckfnb\") pod \"e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj\" (UID: \"0a039273-4b94-4793-b713-1524da46eed6\") " pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:51:42.638828 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.638807 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a039273-4b94-4793-b713-1524da46eed6-istio-token\") pod \"e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj\" (UID: \"0a039273-4b94-4793-b713-1524da46eed6\") " pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:51:42.724116 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.724081 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-istio-podinfo\") pod \"e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr\" (UID: \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\") " pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:51:42.724283 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.724123 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-credential-socket\") pod \"e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr\" (UID: \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\") " pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:51:42.724283 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.724145 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-istio-data\") pod \"e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr\" (UID: \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\") " pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:51:42.724366 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.724292 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-workload-certs\") pod \"e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr\" (UID: \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\") " pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:51:42.724366 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.724335 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-istiod-ca-cert\") pod \"e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr\" (UID: \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\") " pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:51:42.724366 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.724357 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-istio-envoy\") pod \"e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr\" (UID: \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\") " pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:51:42.724507 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.724406 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4jfk\" (UniqueName: \"kubernetes.io/projected/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-kube-api-access-g4jfk\") pod \"e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr\" (UID: \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\") " pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:51:42.724562 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.724537 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-workload-socket\") pod \"e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr\" (UID: \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\") " pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:51:42.724620 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.724577 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-istio-token\") pod \"e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr\" (UID: \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\") " pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:51:42.745143 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.745112 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:51:42.825712 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.825670 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-workload-socket\") pod \"e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr\" (UID: \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\") " pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:51:42.825849 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.825716 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-istio-token\") pod \"e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr\" (UID: \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\") " pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:51:42.825849 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.825759 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-istio-podinfo\") pod \"e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr\" (UID: \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\") " pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:51:42.825849 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.825786 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-credential-socket\") pod \"e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr\" (UID: \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\") " pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:51:42.825849 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.825807 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-istio-data\") pod \"e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr\" (UID: \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\") " pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:51:42.825849 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.825843 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-workload-certs\") pod \"e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr\" (UID: \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\") " pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:51:42.826069 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.825866 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-istiod-ca-cert\") pod \"e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr\" (UID: \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\") " pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:51:42.826069 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.825887 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-istio-envoy\") pod \"e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr\" (UID: \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\") " pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:51:42.826069 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.825916 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4jfk\" (UniqueName: \"kubernetes.io/projected/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-kube-api-access-g4jfk\") pod \"e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr\" (UID: \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\") " pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:51:42.826677 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.826635 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-istio-data\") pod \"e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr\" (UID: \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\") " pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:51:42.826817 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.826708 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-workload-socket\") pod \"e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr\" (UID: \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\") " pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:51:42.826969 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.826945 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-workload-certs\") pod \"e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr\" (UID: \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\") " pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:51:42.827283 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.827261 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-credential-socket\") pod \"e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr\" (UID: \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\") " pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:51:42.828017 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.827931 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-istiod-ca-cert\") pod \"e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr\" (UID: \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\") " pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:51:42.830291 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.830271 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-istio-envoy\") pod \"e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr\" (UID: \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\") " pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:51:42.830391 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.830323 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-istio-podinfo\") pod \"e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr\" (UID: \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\") " pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:51:42.845917 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.845893 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4jfk\" (UniqueName: \"kubernetes.io/projected/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-kube-api-access-g4jfk\") pod \"e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr\" (UID: \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\") " pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:51:42.846527 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.846505 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-istio-token\") pod \"e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr\" (UID: \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\") " pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:51:42.872631 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:42.872602 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:51:43.039390 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:43.039358 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj"] Oct 31 12:51:43.040124 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:51:43.040094 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a039273_4b94_4793_b713_1524da46eed6.slice/crio-c3ffb88a4cb1bd3f951d58d818a54c31d3a31ae78b05a230fd22a9d949763a8c WatchSource:0}: Error finding container c3ffb88a4cb1bd3f951d58d818a54c31d3a31ae78b05a230fd22a9d949763a8c: Status 404 returned error can't find the container with id c3ffb88a4cb1bd3f951d58d818a54c31d3a31ae78b05a230fd22a9d949763a8c Oct 31 12:51:43.042272 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:43.042242 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:51:43.042350 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:43.042308 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:51:43.042350 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:43.042339 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:51:43.160805 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:43.160767 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr"] Oct 31 12:51:43.165013 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:51:43.164493 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83131c5_7e8a_4e3f_8a87_aa1d3c091023.slice/crio-dd45ac1ad02d11c6da412d92b3397db372e6162eb5d8a7d407e7522bfccd0404 WatchSource:0}: Error finding container dd45ac1ad02d11c6da412d92b3397db372e6162eb5d8a7d407e7522bfccd0404: Status 404 returned error can't find the container with id dd45ac1ad02d11c6da412d92b3397db372e6162eb5d8a7d407e7522bfccd0404 Oct 31 12:51:43.167372 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:43.167332 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:51:43.167450 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:43.167416 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:51:43.167487 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:43.167457 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:51:43.169533 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:43.169486 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" event={"ID":"0a039273-4b94-4793-b713-1524da46eed6","Type":"ContainerStarted","Data":"ee4b9ffc8ecf462fba7eabafa696da97a95f0aad3e447f4092653e885593b28d"} Oct 31 12:51:43.169626 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:43.169544 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" event={"ID":"0a039273-4b94-4793-b713-1524da46eed6","Type":"ContainerStarted","Data":"c3ffb88a4cb1bd3f951d58d818a54c31d3a31ae78b05a230fd22a9d949763a8c"} Oct 31 12:51:43.196890 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:43.196838 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" podStartSLOduration=1.196820183 podStartE2EDuration="1.196820183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:51:43.194768777 +0000 UTC m=+1149.677891689" watchObservedRunningTime="2026-10-31 12:51:43.196820183 +0000 UTC m=+1149.679943084" Oct 31 12:51:43.538270 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:43.538229 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:43.745875 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:43.745834 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:51:43.822577 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:43.822482 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:44.177785 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:44.177696 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" event={"ID":"a83131c5-7e8a-4e3f-8a87-aa1d3c091023","Type":"ContainerStarted","Data":"7e2ff5b60ceef687ac667f8ddc28842f82e7db34d773c2d3e94fd1e2d5d2635b"} Oct 31 12:51:44.177785 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:44.177764 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" event={"ID":"a83131c5-7e8a-4e3f-8a87-aa1d3c091023","Type":"ContainerStarted","Data":"dd45ac1ad02d11c6da412d92b3397db372e6162eb5d8a7d407e7522bfccd0404"} Oct 31 12:51:44.237102 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:44.237047 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" podStartSLOduration=2.237031054 podStartE2EDuration="2.237031054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:51:44.231856159 +0000 UTC m=+1150.714979062" watchObservedRunningTime="2026-10-31 12:51:44.237031054 +0000 UTC m=+1150.720153955" Oct 31 12:51:44.237384 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:44.237365 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:44.560601 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:44.560565 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:44.750405 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:44.750374 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:51:44.873087 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:44.873001 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:51:44.878036 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:44.878011 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:51:44.887854 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:44.887833 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:45.118881 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:45.118842 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:45.213265 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:45.213227 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:51:45.213670 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:45.213417 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:51:45.214262 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:45.214236 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:51:45.214385 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:45.214290 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:51:45.796581 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:45.796541 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:46.025766 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:46.025733 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:46.230644 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:46.230608 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cf7e6604-5c746d87ff-lz4s4"] Oct 31 12:51:46.234567 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:46.234547 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cf7e6604-5c746d87ff-lz4s4" Oct 31 12:51:46.246315 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:46.246291 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cf7e6604-5c746d87ff-lz4s4"] Oct 31 12:51:46.249808 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:46.249742 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:46.310586 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:46.310554 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-cf7e6604\"" Oct 31 12:51:46.310826 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:46.310643 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-cf7e6604-dockercfg-p74c5\"" Oct 31 12:51:46.375016 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:46.374963 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdb5g\" (UniqueName: \"kubernetes.io/projected/c8df39d7-db98-4f69-94b3-d96c15f2ff11-kube-api-access-mdb5g\") pod \"payload-processing-e2e-migrate-cf7e6604-5c746d87ff-lz4s4\" (UID: \"c8df39d7-db98-4f69-94b3-d96c15f2ff11\") " pod="openshift-ingress/payload-processing-e2e-migrate-cf7e6604-5c746d87ff-lz4s4" Oct 31 12:51:46.375177 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:46.375050 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8df39d7-db98-4f69-94b3-d96c15f2ff11-plugins-config-volume\") pod \"payload-processing-e2e-migrate-cf7e6604-5c746d87ff-lz4s4\" (UID: \"c8df39d7-db98-4f69-94b3-d96c15f2ff11\") " pod="openshift-ingress/payload-processing-e2e-migrate-cf7e6604-5c746d87ff-lz4s4" Oct 31 12:51:46.476478 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:46.476442 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdb5g\" (UniqueName: \"kubernetes.io/projected/c8df39d7-db98-4f69-94b3-d96c15f2ff11-kube-api-access-mdb5g\") pod \"payload-processing-e2e-migrate-cf7e6604-5c746d87ff-lz4s4\" (UID: \"c8df39d7-db98-4f69-94b3-d96c15f2ff11\") " pod="openshift-ingress/payload-processing-e2e-migrate-cf7e6604-5c746d87ff-lz4s4" Oct 31 12:51:46.476642 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:46.476487 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8df39d7-db98-4f69-94b3-d96c15f2ff11-plugins-config-volume\") pod \"payload-processing-e2e-migrate-cf7e6604-5c746d87ff-lz4s4\" (UID: \"c8df39d7-db98-4f69-94b3-d96c15f2ff11\") " pod="openshift-ingress/payload-processing-e2e-migrate-cf7e6604-5c746d87ff-lz4s4" Oct 31 12:51:46.477277 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:46.477258 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8df39d7-db98-4f69-94b3-d96c15f2ff11-plugins-config-volume\") pod \"payload-processing-e2e-migrate-cf7e6604-5c746d87ff-lz4s4\" (UID: \"c8df39d7-db98-4f69-94b3-d96c15f2ff11\") " pod="openshift-ingress/payload-processing-e2e-migrate-cf7e6604-5c746d87ff-lz4s4" Oct 31 12:51:46.515440 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:46.515370 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdb5g\" (UniqueName: \"kubernetes.io/projected/c8df39d7-db98-4f69-94b3-d96c15f2ff11-kube-api-access-mdb5g\") pod \"payload-processing-e2e-migrate-cf7e6604-5c746d87ff-lz4s4\" (UID: \"c8df39d7-db98-4f69-94b3-d96c15f2ff11\") " pod="openshift-ingress/payload-processing-e2e-migrate-cf7e6604-5c746d87ff-lz4s4" Oct 31 12:51:46.549723 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:46.549696 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cf7e6604-5c746d87ff-lz4s4" Oct 31 12:51:46.625881 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:46.625843 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:46.839968 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:46.839941 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cf7e6604-5c746d87ff-lz4s4"] Oct 31 12:51:46.842117 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:51:46.842092 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8df39d7_db98_4f69_94b3_d96c15f2ff11.slice/crio-a5fe8e7b196ef5c5559a727ea1a85fa1adadb1a375a8958684dc419bb3969016 WatchSource:0}: Error finding container a5fe8e7b196ef5c5559a727ea1a85fa1adadb1a375a8958684dc419bb3969016: Status 404 returned error can't find the container with id a5fe8e7b196ef5c5559a727ea1a85fa1adadb1a375a8958684dc419bb3969016 Oct 31 12:51:46.862707 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:46.862675 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:47.001071 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.001038 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q"] Oct 31 12:51:47.006485 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.006459 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" Oct 31 12:51:47.035501 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.035463 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q"] Oct 31 12:51:47.134544 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.134464 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:47.183244 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.183208 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47a4ec22-fe77-4a7b-a66a-5debadbf5800-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-ssz2q\" (UID: \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" Oct 31 12:51:47.183409 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.183258 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47a4ec22-fe77-4a7b-a66a-5debadbf5800-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-ssz2q\" (UID: \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" Oct 31 12:51:47.183485 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.183422 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47a4ec22-fe77-4a7b-a66a-5debadbf5800-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-ssz2q\" (UID: \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" Oct 31 12:51:47.183485 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.183478 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47a4ec22-fe77-4a7b-a66a-5debadbf5800-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-ssz2q\" (UID: \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" Oct 31 12:51:47.183612 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.183590 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47a4ec22-fe77-4a7b-a66a-5debadbf5800-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-ssz2q\" (UID: \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" Oct 31 12:51:47.183676 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.183641 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvsn5\" (UniqueName: \"kubernetes.io/projected/47a4ec22-fe77-4a7b-a66a-5debadbf5800-kube-api-access-wvsn5\") pod \"maas-default-gateway-openshift-default-fc8c56547-ssz2q\" (UID: \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" Oct 31 12:51:47.183725 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.183702 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a4ec22-fe77-4a7b-a66a-5debadbf5800-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-ssz2q\" (UID: \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" Oct 31 12:51:47.183763 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.183743 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47a4ec22-fe77-4a7b-a66a-5debadbf5800-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-ssz2q\" (UID: \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" Oct 31 12:51:47.183816 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.183802 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a4ec22-fe77-4a7b-a66a-5debadbf5800-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-ssz2q\" (UID: \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" Oct 31 12:51:47.226923 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.226892 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cf7e6604-5c746d87ff-lz4s4" event={"ID":"c8df39d7-db98-4f69-94b3-d96c15f2ff11","Type":"ContainerStarted","Data":"a5fe8e7b196ef5c5559a727ea1a85fa1adadb1a375a8958684dc419bb3969016"} Oct 31 12:51:47.285393 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.285350 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47a4ec22-fe77-4a7b-a66a-5debadbf5800-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-ssz2q\" (UID: \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" Oct 31 12:51:47.285393 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.285395 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47a4ec22-fe77-4a7b-a66a-5debadbf5800-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-ssz2q\" (UID: \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" Oct 31 12:51:47.285924 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.285458 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47a4ec22-fe77-4a7b-a66a-5debadbf5800-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-ssz2q\" (UID: \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" Oct 31 12:51:47.285924 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.285512 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47a4ec22-fe77-4a7b-a66a-5debadbf5800-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-ssz2q\" (UID: \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" Oct 31 12:51:47.285924 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.285555 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47a4ec22-fe77-4a7b-a66a-5debadbf5800-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-ssz2q\" (UID: \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" Oct 31 12:51:47.285924 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.285587 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvsn5\" (UniqueName: \"kubernetes.io/projected/47a4ec22-fe77-4a7b-a66a-5debadbf5800-kube-api-access-wvsn5\") pod \"maas-default-gateway-openshift-default-fc8c56547-ssz2q\" (UID: \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" Oct 31 12:51:47.285924 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.285614 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a4ec22-fe77-4a7b-a66a-5debadbf5800-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-ssz2q\" (UID: \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" Oct 31 12:51:47.285924 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.285641 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47a4ec22-fe77-4a7b-a66a-5debadbf5800-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-ssz2q\" (UID: \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" Oct 31 12:51:47.285924 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.285685 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a4ec22-fe77-4a7b-a66a-5debadbf5800-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-ssz2q\" (UID: \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" Oct 31 12:51:47.286332 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.286308 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47a4ec22-fe77-4a7b-a66a-5debadbf5800-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-ssz2q\" (UID: \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" Oct 31 12:51:47.286750 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.286566 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a4ec22-fe77-4a7b-a66a-5debadbf5800-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-ssz2q\" (UID: \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" Oct 31 12:51:47.286750 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.286622 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a4ec22-fe77-4a7b-a66a-5debadbf5800-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-ssz2q\" (UID: \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" Oct 31 12:51:47.286750 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.286702 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47a4ec22-fe77-4a7b-a66a-5debadbf5800-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-ssz2q\" (UID: \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" Oct 31 12:51:47.287047 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.286782 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47a4ec22-fe77-4a7b-a66a-5debadbf5800-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-ssz2q\" (UID: \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" Oct 31 12:51:47.288383 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.288357 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47a4ec22-fe77-4a7b-a66a-5debadbf5800-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-ssz2q\" (UID: \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" Oct 31 12:51:47.288694 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.288674 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47a4ec22-fe77-4a7b-a66a-5debadbf5800-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-ssz2q\" (UID: \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" Oct 31 12:51:47.309187 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.309162 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvsn5\" (UniqueName: \"kubernetes.io/projected/47a4ec22-fe77-4a7b-a66a-5debadbf5800-kube-api-access-wvsn5\") pod \"maas-default-gateway-openshift-default-fc8c56547-ssz2q\" (UID: \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" Oct 31 12:51:47.315710 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.315686 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47a4ec22-fe77-4a7b-a66a-5debadbf5800-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-ssz2q\" (UID: \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" Oct 31 12:51:47.329475 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.329452 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" Oct 31 12:51:47.516519 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.516461 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:47.691345 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.688083 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q"] Oct 31 12:51:47.698563 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:51:47.698507 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47a4ec22_fe77_4a7b_a66a_5debadbf5800.slice/crio-469a7aea880d06bbd38b74d5658004d062559676d294170c09af68394bdab998 WatchSource:0}: Error finding container 469a7aea880d06bbd38b74d5658004d062559676d294170c09af68394bdab998: Status 404 returned error can't find the container with id 469a7aea880d06bbd38b74d5658004d062559676d294170c09af68394bdab998 Oct 31 12:51:47.702585 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.702545 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:51:47.702792 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.702769 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:51:47.702929 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.702911 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:51:47.838774 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:47.838713 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:48.168229 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:48.168136 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:48.269787 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:48.267011 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" event={"ID":"47a4ec22-fe77-4a7b-a66a-5debadbf5800","Type":"ContainerStarted","Data":"77c4cd2f036dbb6ba64e32c1632e11d9bc4d0de3b426f7eee5bca3a340c296f3"} Oct 31 12:51:48.269787 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:48.269734 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" event={"ID":"47a4ec22-fe77-4a7b-a66a-5debadbf5800","Type":"ContainerStarted","Data":"469a7aea880d06bbd38b74d5658004d062559676d294170c09af68394bdab998"} Oct 31 12:51:48.306752 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:48.306693 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" podStartSLOduration=2.30667724 podStartE2EDuration="2.30667724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:51:48.306270797 +0000 UTC m=+1154.789393697" watchObservedRunningTime="2026-10-31 12:51:48.30667724 +0000 UTC m=+1154.789800142" Oct 31 12:51:48.330510 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:48.330476 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" Oct 31 12:51:48.332074 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:48.332046 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-ssz2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 31 12:51:48.332223 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:48.332095 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" podUID="47a4ec22-fe77-4a7b-a66a-5debadbf5800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 31 12:51:48.376101 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:48.376054 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:48.662519 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:48.662434 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:48.870442 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:48.870404 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:49.330334 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:49.330302 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-ssz2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 31 12:51:49.330736 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:49.330351 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" podUID="47a4ec22-fe77-4a7b-a66a-5debadbf5800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 31 12:51:49.571494 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:49.571463 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:50.034925 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:51:50.034895 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:51:50.228240 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:50.228210 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d48f454a-f87456947-s49tk"] Oct 31 12:51:50.234079 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:50.233141 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d48f454a-f87456947-s49tk" Oct 31 12:51:50.272273 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:50.272243 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 12:51:50.272451 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:50.272297 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 12:51:50.272451 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:50.272310 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d48f454a-serving-cert\"" Oct 31 12:51:50.272451 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:50.272241 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 12:51:50.272451 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:50.272363 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rhftr\"" Oct 31 12:51:50.275877 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:50.275748 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d48f454a-f87456947-s49tk"] Oct 31 12:51:50.300118 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:50.300089 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cf7e6604-5c746d87ff-lz4s4" event={"ID":"c8df39d7-db98-4f69-94b3-d96c15f2ff11","Type":"ContainerStarted","Data":"ccd8785535f78314ba0aa6f8a5e4380e576e629213448e7a0dc044f71e9ccbb1"} Oct 31 12:51:50.300408 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:50.300390 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-cf7e6604-5c746d87ff-lz4s4" Oct 31 12:51:50.325402 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:50.325352 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-cf7e6604-5c746d87ff-lz4s4" podStartSLOduration=1.286623877 podStartE2EDuration="4.325337237s" totalImagesPullingTime="3.03871336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:51:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:51:50.32427945 +0000 UTC m=+1156.807402353" watchObservedRunningTime="2026-10-31 12:51:50.325337237 +0000 UTC m=+1156.808460139" Oct 31 12:51:50.330364 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:50.330340 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-ssz2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 31 12:51:50.330871 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:50.330425 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" podUID="47a4ec22-fe77-4a7b-a66a-5debadbf5800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 31 12:51:50.333815 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:50.333795 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3689d1f7-970f-48db-8bee-c50ffae8ab74-maas-api-tls\") pod \"maas-api-e2e-ait-d48f454a-f87456947-s49tk\" (UID: \"3689d1f7-970f-48db-8bee-c50ffae8ab74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d48f454a-f87456947-s49tk" Oct 31 12:51:50.333885 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:50.333865 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mhbt\" (UniqueName: \"kubernetes.io/projected/3689d1f7-970f-48db-8bee-c50ffae8ab74-kube-api-access-5mhbt\") pod \"maas-api-e2e-ait-d48f454a-f87456947-s49tk\" (UID: \"3689d1f7-970f-48db-8bee-c50ffae8ab74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d48f454a-f87456947-s49tk" Oct 31 12:51:50.334029 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:50.334009 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3689d1f7-970f-48db-8bee-c50ffae8ab74-openshift-service-ca\") pod \"maas-api-e2e-ait-d48f454a-f87456947-s49tk\" (UID: \"3689d1f7-970f-48db-8bee-c50ffae8ab74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d48f454a-f87456947-s49tk" Oct 31 12:51:50.334071 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:50.334042 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3689d1f7-970f-48db-8bee-c50ffae8ab74-metrics-certs\") pod \"maas-api-e2e-ait-d48f454a-f87456947-s49tk\" (UID: \"3689d1f7-970f-48db-8bee-c50ffae8ab74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d48f454a-f87456947-s49tk" Oct 31 12:51:50.395796 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:50.395755 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:50.435813 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:50.435775 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mhbt\" (UniqueName: \"kubernetes.io/projected/3689d1f7-970f-48db-8bee-c50ffae8ab74-kube-api-access-5mhbt\") pod \"maas-api-e2e-ait-d48f454a-f87456947-s49tk\" (UID: \"3689d1f7-970f-48db-8bee-c50ffae8ab74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d48f454a-f87456947-s49tk" Oct 31 12:51:50.436105 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:50.435860 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3689d1f7-970f-48db-8bee-c50ffae8ab74-openshift-service-ca\") pod \"maas-api-e2e-ait-d48f454a-f87456947-s49tk\" (UID: \"3689d1f7-970f-48db-8bee-c50ffae8ab74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d48f454a-f87456947-s49tk" Oct 31 12:51:50.436105 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:50.435900 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3689d1f7-970f-48db-8bee-c50ffae8ab74-metrics-certs\") pod \"maas-api-e2e-ait-d48f454a-f87456947-s49tk\" (UID: \"3689d1f7-970f-48db-8bee-c50ffae8ab74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d48f454a-f87456947-s49tk" Oct 31 12:51:50.436105 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:50.436063 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3689d1f7-970f-48db-8bee-c50ffae8ab74-maas-api-tls\") pod \"maas-api-e2e-ait-d48f454a-f87456947-s49tk\" (UID: \"3689d1f7-970f-48db-8bee-c50ffae8ab74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d48f454a-f87456947-s49tk" Oct 31 12:51:50.437178 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:51:50.437035 6841 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-d48f454a-serving-cert: secret "maas-api-e2e-ait-d48f454a-serving-cert" not found Oct 31 12:51:50.437178 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:51:50.437114 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3689d1f7-970f-48db-8bee-c50ffae8ab74-maas-api-tls podName:3689d1f7-970f-48db-8bee-c50ffae8ab74 nodeName:}" failed. No retries permitted until 2026-10-31 12:51:50.937095983 +0000 UTC m=+1157.420218864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3689d1f7-970f-48db-8bee-c50ffae8ab74-maas-api-tls") pod "maas-api-e2e-ait-d48f454a-f87456947-s49tk" (UID: "3689d1f7-970f-48db-8bee-c50ffae8ab74") : secret "maas-api-e2e-ait-d48f454a-serving-cert" not found Oct 31 12:51:50.439100 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:50.439042 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3689d1f7-970f-48db-8bee-c50ffae8ab74-openshift-service-ca\") pod \"maas-api-e2e-ait-d48f454a-f87456947-s49tk\" (UID: \"3689d1f7-970f-48db-8bee-c50ffae8ab74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d48f454a-f87456947-s49tk" Oct 31 12:51:50.500053 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:50.497155 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mhbt\" (UniqueName: \"kubernetes.io/projected/3689d1f7-970f-48db-8bee-c50ffae8ab74-kube-api-access-5mhbt\") pod \"maas-api-e2e-ait-d48f454a-f87456947-s49tk\" (UID: \"3689d1f7-970f-48db-8bee-c50ffae8ab74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d48f454a-f87456947-s49tk" Oct 31 12:51:50.500053 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:50.497646 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3689d1f7-970f-48db-8bee-c50ffae8ab74-metrics-certs\") pod \"maas-api-e2e-ait-d48f454a-f87456947-s49tk\" (UID: \"3689d1f7-970f-48db-8bee-c50ffae8ab74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d48f454a-f87456947-s49tk" Oct 31 12:51:50.603052 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:50.603010 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:50.814007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:50.813901 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:50.948008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:50.946592 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3689d1f7-970f-48db-8bee-c50ffae8ab74-maas-api-tls\") pod \"maas-api-e2e-ait-d48f454a-f87456947-s49tk\" (UID: \"3689d1f7-970f-48db-8bee-c50ffae8ab74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d48f454a-f87456947-s49tk" Oct 31 12:51:50.965005 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:50.964841 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3689d1f7-970f-48db-8bee-c50ffae8ab74-maas-api-tls\") pod \"maas-api-e2e-ait-d48f454a-f87456947-s49tk\" (UID: \"3689d1f7-970f-48db-8bee-c50ffae8ab74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d48f454a-f87456947-s49tk" Oct 31 12:51:51.012722 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:51.012688 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:51.185258 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:51.185182 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d48f454a-f87456947-s49tk" Oct 31 12:51:51.318250 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:51.318099 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:51.333098 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:51.333052 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-ssz2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 31 12:51:51.333613 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:51.333127 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" podUID="47a4ec22-fe77-4a7b-a66a-5debadbf5800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 31 12:51:51.524024 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:51.522669 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d48f454a-f87456947-s49tk"] Oct 31 12:51:51.524024 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:51:51.523241 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3689d1f7_970f_48db_8bee_c50ffae8ab74.slice/crio-2d8bda3ced22fe6f086e293d453bec92355b251a2198fbcf81588be5b7f406d4 WatchSource:0}: Error finding container 2d8bda3ced22fe6f086e293d453bec92355b251a2198fbcf81588be5b7f406d4: Status 404 returned error can't find the container with id 2d8bda3ced22fe6f086e293d453bec92355b251a2198fbcf81588be5b7f406d4 Oct 31 12:51:51.604820 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:51.604780 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:51.806572 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:51.806475 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g"] Oct 31 12:51:51.811582 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:51.811555 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:51:51.833223 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:51.833199 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-785c04b9-openshift-default-dockercfg-lhr2z\"" Oct 31 12:51:51.876016 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:51.874597 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3bee1b8-5152-4805-a59d-7d05594142de-workload-certs\") pod \"e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g\" (UID: \"f3bee1b8-5152-4805-a59d-7d05594142de\") " pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:51:51.876016 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:51.874645 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3bee1b8-5152-4805-a59d-7d05594142de-credential-socket\") pod \"e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g\" (UID: \"f3bee1b8-5152-4805-a59d-7d05594142de\") " pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:51:51.876016 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:51.874675 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3bee1b8-5152-4805-a59d-7d05594142de-istio-token\") pod \"e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g\" (UID: \"f3bee1b8-5152-4805-a59d-7d05594142de\") " pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:51:51.876016 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:51.874717 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3bee1b8-5152-4805-a59d-7d05594142de-workload-socket\") pod \"e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g\" (UID: \"f3bee1b8-5152-4805-a59d-7d05594142de\") " pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:51:51.876016 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:51.874742 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3bee1b8-5152-4805-a59d-7d05594142de-istio-envoy\") pod \"e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g\" (UID: \"f3bee1b8-5152-4805-a59d-7d05594142de\") " pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:51:51.876016 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:51.874764 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3bee1b8-5152-4805-a59d-7d05594142de-istio-data\") pod \"e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g\" (UID: \"f3bee1b8-5152-4805-a59d-7d05594142de\") " pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:51:51.876016 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:51.874851 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqbsc\" (UniqueName: \"kubernetes.io/projected/f3bee1b8-5152-4805-a59d-7d05594142de-kube-api-access-sqbsc\") pod \"e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g\" (UID: \"f3bee1b8-5152-4805-a59d-7d05594142de\") " pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:51:51.876016 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:51.874882 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3bee1b8-5152-4805-a59d-7d05594142de-istio-podinfo\") pod \"e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g\" (UID: \"f3bee1b8-5152-4805-a59d-7d05594142de\") " pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:51:51.876016 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:51.874912 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3bee1b8-5152-4805-a59d-7d05594142de-istiod-ca-cert\") pod \"e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g\" (UID: \"f3bee1b8-5152-4805-a59d-7d05594142de\") " pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:51:51.896700 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:51.895320 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g"] Oct 31 12:51:51.977342 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:51.977301 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3bee1b8-5152-4805-a59d-7d05594142de-workload-certs\") pod \"e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g\" (UID: \"f3bee1b8-5152-4805-a59d-7d05594142de\") " pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:51:51.977504 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:51.977353 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3bee1b8-5152-4805-a59d-7d05594142de-credential-socket\") pod \"e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g\" (UID: \"f3bee1b8-5152-4805-a59d-7d05594142de\") " pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:51:51.977504 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:51.977399 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3bee1b8-5152-4805-a59d-7d05594142de-istio-token\") pod \"e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g\" (UID: \"f3bee1b8-5152-4805-a59d-7d05594142de\") " pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:51:51.977504 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:51.977461 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3bee1b8-5152-4805-a59d-7d05594142de-workload-socket\") pod \"e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g\" (UID: \"f3bee1b8-5152-4805-a59d-7d05594142de\") " pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:51:51.977504 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:51.977489 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3bee1b8-5152-4805-a59d-7d05594142de-istio-envoy\") pod \"e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g\" (UID: \"f3bee1b8-5152-4805-a59d-7d05594142de\") " pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:51:51.977637 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:51.977528 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3bee1b8-5152-4805-a59d-7d05594142de-istio-data\") pod \"e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g\" (UID: \"f3bee1b8-5152-4805-a59d-7d05594142de\") " pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:51:51.977670 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:51.977642 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqbsc\" (UniqueName: \"kubernetes.io/projected/f3bee1b8-5152-4805-a59d-7d05594142de-kube-api-access-sqbsc\") pod \"e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g\" (UID: \"f3bee1b8-5152-4805-a59d-7d05594142de\") " pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:51:51.977703 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:51.977686 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3bee1b8-5152-4805-a59d-7d05594142de-istio-podinfo\") pod \"e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g\" (UID: \"f3bee1b8-5152-4805-a59d-7d05594142de\") " pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:51:51.977737 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:51.977718 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3bee1b8-5152-4805-a59d-7d05594142de-istiod-ca-cert\") pod \"e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g\" (UID: \"f3bee1b8-5152-4805-a59d-7d05594142de\") " pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:51:51.977778 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:51.977754 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3bee1b8-5152-4805-a59d-7d05594142de-workload-certs\") pod \"e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g\" (UID: \"f3bee1b8-5152-4805-a59d-7d05594142de\") " pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:51:51.978164 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:51.978135 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3bee1b8-5152-4805-a59d-7d05594142de-credential-socket\") pod \"e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g\" (UID: \"f3bee1b8-5152-4805-a59d-7d05594142de\") " pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:51:51.978401 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:51.978381 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3bee1b8-5152-4805-a59d-7d05594142de-workload-socket\") pod \"e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g\" (UID: \"f3bee1b8-5152-4805-a59d-7d05594142de\") " pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:51:51.978440 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:51.978393 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3bee1b8-5152-4805-a59d-7d05594142de-istio-data\") pod \"e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g\" (UID: \"f3bee1b8-5152-4805-a59d-7d05594142de\") " pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:51:51.979310 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:51.979286 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3bee1b8-5152-4805-a59d-7d05594142de-istiod-ca-cert\") pod \"e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g\" (UID: \"f3bee1b8-5152-4805-a59d-7d05594142de\") " pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:51:51.981178 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:51.981157 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3bee1b8-5152-4805-a59d-7d05594142de-istio-envoy\") pod \"e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g\" (UID: \"f3bee1b8-5152-4805-a59d-7d05594142de\") " pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:51:51.981448 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:51.981427 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3bee1b8-5152-4805-a59d-7d05594142de-istio-podinfo\") pod \"e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g\" (UID: \"f3bee1b8-5152-4805-a59d-7d05594142de\") " pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:51:52.008021 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:52.007970 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3bee1b8-5152-4805-a59d-7d05594142de-istio-token\") pod \"e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g\" (UID: \"f3bee1b8-5152-4805-a59d-7d05594142de\") " pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:51:52.014925 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:52.014897 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqbsc\" (UniqueName: \"kubernetes.io/projected/f3bee1b8-5152-4805-a59d-7d05594142de-kube-api-access-sqbsc\") pod \"e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g\" (UID: \"f3bee1b8-5152-4805-a59d-7d05594142de\") " pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:51:52.087641 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:52.087553 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:52.125652 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:52.125616 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:51:52.325291 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:52.325251 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:52.330369 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:52.330339 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-ssz2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 31 12:51:52.330490 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:52.330387 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" podUID="47a4ec22-fe77-4a7b-a66a-5debadbf5800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 31 12:51:52.337342 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:52.337317 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d48f454a-f87456947-s49tk" event={"ID":"3689d1f7-970f-48db-8bee-c50ffae8ab74","Type":"ContainerStarted","Data":"2d8bda3ced22fe6f086e293d453bec92355b251a2198fbcf81588be5b7f406d4"} Oct 31 12:51:52.465056 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:52.456909 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g"] Oct 31 12:51:52.476913 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:51:52.476487 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3bee1b8_5152_4805_a59d_7d05594142de.slice/crio-b9ff1eece98c46b9ecd49ae93dd61d191ca1d14d53e5a6560f781a4c0b63064f WatchSource:0}: Error finding container b9ff1eece98c46b9ecd49ae93dd61d191ca1d14d53e5a6560f781a4c0b63064f: Status 404 returned error can't find the container with id b9ff1eece98c46b9ecd49ae93dd61d191ca1d14d53e5a6560f781a4c0b63064f Oct 31 12:51:52.483138 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:52.482035 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:51:52.483138 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:52.482123 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:51:52.483138 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:52.482170 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:51:52.680832 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:52.680702 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:52.909765 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:52.909268 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:53.135007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:53.131332 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:53.330844 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:53.330812 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-ssz2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 31 12:51:53.331080 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:53.330869 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" podUID="47a4ec22-fe77-4a7b-a66a-5debadbf5800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 31 12:51:53.397792 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:53.396852 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" event={"ID":"f3bee1b8-5152-4805-a59d-7d05594142de","Type":"ContainerStarted","Data":"0d8ec6d196a15f540c960379f94253ba259464f3f63c7d92254846e3fd7769c1"} Oct 31 12:51:53.397792 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:53.396916 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" event={"ID":"f3bee1b8-5152-4805-a59d-7d05594142de","Type":"ContainerStarted","Data":"b9ff1eece98c46b9ecd49ae93dd61d191ca1d14d53e5a6560f781a4c0b63064f"} Oct 31 12:51:53.428884 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:53.428834 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" podStartSLOduration=2.4288203839999998 podStartE2EDuration="2.428820384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:51:53.427475719 +0000 UTC m=+1159.910598621" watchObservedRunningTime="2026-10-31 12:51:53.428820384 +0000 UTC m=+1159.911943284" Oct 31 12:51:53.594785 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:53.594646 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:54.126007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:54.125833 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:51:54.142008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:54.131951 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:51:54.329901 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:54.329863 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-ssz2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 31 12:51:54.330091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:54.329926 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" podUID="47a4ec22-fe77-4a7b-a66a-5debadbf5800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 31 12:51:54.409655 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:54.409579 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:51:54.410924 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:54.410728 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:51:54.593340 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:54.593293 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:55.121791 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:55.121746 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:55.326682 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:55.326644 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:55.330682 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:55.330653 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-ssz2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 31 12:51:55.330815 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:55.330703 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" podUID="47a4ec22-fe77-4a7b-a66a-5debadbf5800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 31 12:51:55.873425 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:55.873390 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:56.534539 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:56.534500 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:56.772931 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:56.772896 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:57.335009 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:57.334902 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-ssz2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:51:57.335009 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:57.334954 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" podUID="47a4ec22-fe77-4a7b-a66a-5debadbf5800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:51:57.360007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:57.357967 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" Oct 31 12:51:57.360007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:57.359292 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" Oct 31 12:51:57.365002 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:57.363208 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:51:57.367427 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:57.365296 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" Oct 31 12:51:57.439535 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:57.438722 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d48f454a-f87456947-s49tk" event={"ID":"3689d1f7-970f-48db-8bee-c50ffae8ab74","Type":"ContainerStarted","Data":"04bdfb681df390fde65b6629a1079dad14500a2051ae926944eb0cbbc4dd20d0"} Oct 31 12:51:57.439535 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:57.438788 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d48f454a-f87456947-s49tk" Oct 31 12:51:58.767677 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:51:58.767640 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:00.014437 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:00.014395 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:01.261478 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:01.261092 6841 patch_prober.go:29] interesting pod/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:52:01.261478 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:01.261162 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" podUID="0a039273-4b94-4793-b713-1524da46eed6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:52:01.267852 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:01.267817 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:01.355947 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:01.353011 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-cf7e6604-5c746d87ff-lz4s4" Oct 31 12:52:01.403719 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:01.403666 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d48f454a-f87456947-s49tk" podStartSLOduration=6.6046285959999995 podStartE2EDuration="11.403651927s" totalImagesPullingTime="4.799023331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:51:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:51:57.53414475 +0000 UTC m=+1164.017267651" watchObservedRunningTime="2026-10-31 12:52:01.403651927 +0000 UTC m=+1167.886774828" Oct 31 12:52:01.524434 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:01.524353 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:02.503534 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:02.495794 6841 generic.go:356] "Generic (PLEG): container finished" podID="0a039273-4b94-4793-b713-1524da46eed6" containerID="ee4b9ffc8ecf462fba7eabafa696da97a95f0aad3e447f4092653e885593b28d" exitCode=0 Oct 31 12:52:02.503534 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:02.495950 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" event={"ID":"0a039273-4b94-4793-b713-1524da46eed6","Type":"ContainerDied","Data":"ee4b9ffc8ecf462fba7eabafa696da97a95f0aad3e447f4092653e885593b28d"} Oct 31 12:52:02.503534 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:02.496497 6841 scope.go:122] "RemoveContainer" containerID="ee4b9ffc8ecf462fba7eabafa696da97a95f0aad3e447f4092653e885593b28d" Oct 31 12:52:02.503534 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:02.499382 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:52:02.503534 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:02.499472 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:52:02.503534 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:02.499520 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:52:02.743763 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:02.743726 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:03.503614 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:03.503580 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" event={"ID":"0a039273-4b94-4793-b713-1524da46eed6","Type":"ContainerStarted","Data":"7c98135a7d0af4e7a4a221ca51b8777502c31dd0af8844638c44faa715934a0e"} Oct 31 12:52:03.746796 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:03.746362 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:52:03.746796 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:03.746492 6841 patch_prober.go:29] interesting pod/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 31 12:52:03.746796 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:03.746536 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" podUID="0a039273-4b94-4793-b713-1524da46eed6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 31 12:52:03.940781 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:03.940685 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:04.479462 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:04.479426 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d48f454a-f87456947-s49tk" Oct 31 12:52:04.762060 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:04.753350 6841 patch_prober.go:29] interesting pod/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 31 12:52:04.762060 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:04.753411 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" podUID="0a039273-4b94-4793-b713-1524da46eed6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 31 12:52:05.083053 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:05.082836 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:05.758008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:05.747400 6841 patch_prober.go:29] interesting pod/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 31 12:52:05.758008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:05.747467 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" podUID="0a039273-4b94-4793-b713-1524da46eed6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 31 12:52:06.436562 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:06.436527 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:06.773007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:06.772424 6841 patch_prober.go:29] interesting pod/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 31 12:52:06.773007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:06.772486 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" podUID="0a039273-4b94-4793-b713-1524da46eed6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 31 12:52:07.534790 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:07.534761 6841 generic.go:356] "Generic (PLEG): container finished" podID="a83131c5-7e8a-4e3f-8a87-aa1d3c091023" containerID="7e2ff5b60ceef687ac667f8ddc28842f82e7db34d773c2d3e94fd1e2d5d2635b" exitCode=0 Oct 31 12:52:07.535238 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:07.534805 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" event={"ID":"a83131c5-7e8a-4e3f-8a87-aa1d3c091023","Type":"ContainerDied","Data":"7e2ff5b60ceef687ac667f8ddc28842f82e7db34d773c2d3e94fd1e2d5d2635b"} Oct 31 12:52:07.535238 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:07.535179 6841 scope.go:122] "RemoveContainer" containerID="7e2ff5b60ceef687ac667f8ddc28842f82e7db34d773c2d3e94fd1e2d5d2635b" Oct 31 12:52:07.536506 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:07.536474 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:52:07.536601 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:07.536541 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:52:07.536601 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:07.536575 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:52:07.692029 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:07.682582 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:07.767007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:07.766376 6841 patch_prober.go:29] interesting pod/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 31 12:52:07.767007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:07.766433 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" podUID="0a039273-4b94-4793-b713-1524da46eed6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 31 12:52:08.545007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:08.544580 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" event={"ID":"a83131c5-7e8a-4e3f-8a87-aa1d3c091023","Type":"ContainerStarted","Data":"774de72b8a36367c51bf6b72ab48f17f4d610d0dd1c14309547e707c5908468b"} Oct 31 12:52:08.657005 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:08.653469 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:08.852599 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:08.852510 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:52:08.854414 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:08.854377 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:52:08.863193 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:08.863163 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:52:08.873550 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:08.873517 6841 patch_prober.go:29] interesting pod/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 31 12:52:08.873845 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:08.873804 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" podUID="a83131c5-7e8a-4e3f-8a87-aa1d3c091023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 31 12:52:08.874337 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:08.873149 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:52:09.578643 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:09.578609 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:09.877064 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:09.876377 6841 patch_prober.go:29] interesting pod/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 31 12:52:09.877064 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:09.876459 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" podUID="a83131c5-7e8a-4e3f-8a87-aa1d3c091023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 31 12:52:10.476007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:10.465905 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:10.884195 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:10.867890 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cf7e6604-5c746d87ff-lz4s4"] Oct 31 12:52:10.884868 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:10.873339 6841 patch_prober.go:29] interesting pod/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 31 12:52:10.885008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:10.884913 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" podUID="a83131c5-7e8a-4e3f-8a87-aa1d3c091023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 31 12:52:11.102415 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:11.102349 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:11.303470 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:11.303433 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:11.717761 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:11.717725 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:11.874048 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:11.873997 6841 patch_prober.go:29] interesting pod/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 31 12:52:11.874236 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:11.874083 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" podUID="a83131c5-7e8a-4e3f-8a87-aa1d3c091023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 31 12:52:11.993079 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:11.992989 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:12.224906 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:12.224816 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:12.867021 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:12.866969 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:12.873236 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:12.873195 6841 patch_prober.go:29] interesting pod/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 31 12:52:12.873419 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:12.873264 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" podUID="a83131c5-7e8a-4e3f-8a87-aa1d3c091023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 31 12:52:13.107132 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:13.107094 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:13.522529 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:13.513638 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:13.722460 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:13.722407 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:13.874182 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:13.874087 6841 patch_prober.go:29] interesting pod/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 31 12:52:13.874182 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:13.874151 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" podUID="a83131c5-7e8a-4e3f-8a87-aa1d3c091023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 31 12:52:14.007046 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:14.007003 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:14.877647 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:14.877611 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:52:14.878167 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:14.877908 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:52:14.878540 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:14.878519 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:52:15.077509 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:15.077471 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:15.886136 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:15.886097 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-cf7e6604-5c746d87ff-lz4s4" podUID="c8df39d7-db98-4f69-94b3-d96c15f2ff11" containerName="payload-processing" containerID="cri-o://ccd8785535f78314ba0aa6f8a5e4380e576e629213448e7a0dc044f71e9ccbb1" gracePeriod=25 Oct 31 12:52:16.151192 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:16.151167 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:16.153240 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:16.153218 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cf7e6604-5c746d87ff-lz4s4" Oct 31 12:52:16.180661 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:16.180635 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8df39d7-db98-4f69-94b3-d96c15f2ff11-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8df39d7-db98-4f69-94b3-d96c15f2ff11-plugins-config-volume\") pod \"c8df39d7-db98-4f69-94b3-d96c15f2ff11\" (UID: \"c8df39d7-db98-4f69-94b3-d96c15f2ff11\") " Oct 31 12:52:16.180832 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:16.180745 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8df39d7-db98-4f69-94b3-d96c15f2ff11-kube-api-access-mdb5g\" (UniqueName: \"kubernetes.io/projected/c8df39d7-db98-4f69-94b3-d96c15f2ff11-kube-api-access-mdb5g\") pod \"c8df39d7-db98-4f69-94b3-d96c15f2ff11\" (UID: \"c8df39d7-db98-4f69-94b3-d96c15f2ff11\") " Oct 31 12:52:16.181102 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:16.181071 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8df39d7-db98-4f69-94b3-d96c15f2ff11-plugins-config-volume" pod "c8df39d7-db98-4f69-94b3-d96c15f2ff11" (UID: "c8df39d7-db98-4f69-94b3-d96c15f2ff11"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:52:16.183384 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:16.183336 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8df39d7-db98-4f69-94b3-d96c15f2ff11-kube-api-access-mdb5g" pod "c8df39d7-db98-4f69-94b3-d96c15f2ff11" (UID: "c8df39d7-db98-4f69-94b3-d96c15f2ff11"). InnerVolumeSpecName "kube-api-access-mdb5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:52:16.282172 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:16.282136 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdb5g\" (UniqueName: \"kubernetes.io/projected/c8df39d7-db98-4f69-94b3-d96c15f2ff11-kube-api-access-mdb5g\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:16.282172 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:16.282173 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8df39d7-db98-4f69-94b3-d96c15f2ff11-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:16.478809 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:16.478770 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:16.631725 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:16.631688 6841 generic.go:356] "Generic (PLEG): container finished" podID="c8df39d7-db98-4f69-94b3-d96c15f2ff11" containerID="ccd8785535f78314ba0aa6f8a5e4380e576e629213448e7a0dc044f71e9ccbb1" exitCode=0 Oct 31 12:52:16.631906 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:16.631755 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cf7e6604-5c746d87ff-lz4s4" Oct 31 12:52:16.631906 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:16.631774 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cf7e6604-5c746d87ff-lz4s4" event={"ID":"c8df39d7-db98-4f69-94b3-d96c15f2ff11","Type":"ContainerDied","Data":"ccd8785535f78314ba0aa6f8a5e4380e576e629213448e7a0dc044f71e9ccbb1"} Oct 31 12:52:16.631906 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:16.631823 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cf7e6604-5c746d87ff-lz4s4" event={"ID":"c8df39d7-db98-4f69-94b3-d96c15f2ff11","Type":"ContainerDied","Data":"a5fe8e7b196ef5c5559a727ea1a85fa1adadb1a375a8958684dc419bb3969016"} Oct 31 12:52:16.631906 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:16.631837 6841 scope.go:122] "RemoveContainer" containerID="ccd8785535f78314ba0aa6f8a5e4380e576e629213448e7a0dc044f71e9ccbb1" Oct 31 12:52:16.633729 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:16.633706 6841 generic.go:356] "Generic (PLEG): container finished" podID="f3bee1b8-5152-4805-a59d-7d05594142de" containerID="0d8ec6d196a15f540c960379f94253ba259464f3f63c7d92254846e3fd7769c1" exitCode=0 Oct 31 12:52:16.633845 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:16.633779 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" event={"ID":"f3bee1b8-5152-4805-a59d-7d05594142de","Type":"ContainerDied","Data":"0d8ec6d196a15f540c960379f94253ba259464f3f63c7d92254846e3fd7769c1"} Oct 31 12:52:16.634409 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:16.634388 6841 scope.go:122] "RemoveContainer" containerID="0d8ec6d196a15f540c960379f94253ba259464f3f63c7d92254846e3fd7769c1" Oct 31 12:52:16.635783 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:16.635747 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:52:16.635878 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:16.635862 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:52:16.635943 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:16.635919 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:52:16.642531 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:16.642509 6841 scope.go:122] "RemoveContainer" containerID="ccd8785535f78314ba0aa6f8a5e4380e576e629213448e7a0dc044f71e9ccbb1" Oct 31 12:52:16.642867 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:52:16.642823 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccd8785535f78314ba0aa6f8a5e4380e576e629213448e7a0dc044f71e9ccbb1\": container with ID starting with ccd8785535f78314ba0aa6f8a5e4380e576e629213448e7a0dc044f71e9ccbb1 not found: ID does not exist" containerID="ccd8785535f78314ba0aa6f8a5e4380e576e629213448e7a0dc044f71e9ccbb1" Oct 31 12:52:16.642960 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:16.642873 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccd8785535f78314ba0aa6f8a5e4380e576e629213448e7a0dc044f71e9ccbb1"} err="failed to get container status \"ccd8785535f78314ba0aa6f8a5e4380e576e629213448e7a0dc044f71e9ccbb1\": rpc error: code = NotFound desc = could not find container \"ccd8785535f78314ba0aa6f8a5e4380e576e629213448e7a0dc044f71e9ccbb1\": container with ID starting with ccd8785535f78314ba0aa6f8a5e4380e576e629213448e7a0dc044f71e9ccbb1 not found: ID does not exist" Oct 31 12:52:16.680731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:16.680699 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cf7e6604-5c746d87ff-lz4s4"] Oct 31 12:52:16.692212 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:16.692185 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cf7e6604-5c746d87ff-lz4s4"] Oct 31 12:52:16.696605 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:16.696575 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:17.367433 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:17.367354 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:17.654142 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:17.642292 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8df39d7-db98-4f69-94b3-d96c15f2ff11" path="/var/lib/kubelet/pods/c8df39d7-db98-4f69-94b3-d96c15f2ff11/volumes" Oct 31 12:52:17.672007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:17.669450 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" event={"ID":"f3bee1b8-5152-4805-a59d-7d05594142de","Type":"ContainerStarted","Data":"a3b81df16ff0d66c25dc0720c9d96f1af07657b55a784f380eea562bdce32435"} Oct 31 12:52:17.721957 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:17.716713 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:18.126766 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:18.126732 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:52:18.126937 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:18.126902 6841 patch_prober.go:29] interesting pod/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 31 12:52:18.127007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:18.126960 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" podUID="f3bee1b8-5152-4805-a59d-7d05594142de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 31 12:52:18.366021 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:18.363202 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:18.776436 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:18.776400 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:19.119545 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:19.119453 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:19.126495 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:19.126471 6841 patch_prober.go:29] interesting pod/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 31 12:52:19.126576 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:19.126533 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" podUID="f3bee1b8-5152-4805-a59d-7d05594142de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 31 12:52:19.916665 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:19.916625 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:20.033930 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:52:20.033893 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:52:20.126240 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:20.126210 6841 patch_prober.go:29] interesting pod/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 31 12:52:20.126378 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:20.126270 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" podUID="f3bee1b8-5152-4805-a59d-7d05594142de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 31 12:52:20.395247 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:20.395211 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj"] Oct 31 12:52:20.395439 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:20.395414 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" podUID="0a039273-4b94-4793-b713-1524da46eed6" containerName="istio-proxy" containerID="cri-o://7c98135a7d0af4e7a4a221ca51b8777502c31dd0af8844638c44faa715934a0e" gracePeriod=30 Oct 31 12:52:20.550007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:20.536818 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:21.126168 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.126128 6841 patch_prober.go:29] interesting pod/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 31 12:52:21.126552 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.126179 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" podUID="f3bee1b8-5152-4805-a59d-7d05594142de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 31 12:52:21.161114 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.160900 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:21.389218 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.389129 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:21.693786 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.693760 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v"] Oct 31 12:52:21.694381 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.694357 6841 cpu_manager.go:414] "containerMap: removing container" podUID="c8df39d7-db98-4f69-94b3-d96c15f2ff11" containerName="payload-processing" Oct 31 12:52:21.694560 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.694536 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8df39d7-db98-4f69-94b3-d96c15f2ff11" containerName="payload-processing" Oct 31 12:52:21.694708 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.694693 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8df39d7-db98-4f69-94b3-d96c15f2ff11" containerName="payload-processing" Oct 31 12:52:21.698655 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.698635 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:21.705613 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.705591 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-7ae9e7-openshift-default-dockercfg-pplr5\"" Oct 31 12:52:21.717235 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.717214 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v"] Oct 31 12:52:21.778178 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.778142 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2975edb7-e9cc-4793-acac-baa30b53613c-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v\" (UID: \"2975edb7-e9cc-4793-acac-baa30b53613c\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:21.778178 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.778180 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2975edb7-e9cc-4793-acac-baa30b53613c-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v\" (UID: \"2975edb7-e9cc-4793-acac-baa30b53613c\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:21.778375 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.778200 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2975edb7-e9cc-4793-acac-baa30b53613c-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v\" (UID: \"2975edb7-e9cc-4793-acac-baa30b53613c\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:21.778417 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.778367 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2975edb7-e9cc-4793-acac-baa30b53613c-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v\" (UID: \"2975edb7-e9cc-4793-acac-baa30b53613c\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:21.778417 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.778401 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2975edb7-e9cc-4793-acac-baa30b53613c-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v\" (UID: \"2975edb7-e9cc-4793-acac-baa30b53613c\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:21.778477 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.778458 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2975edb7-e9cc-4793-acac-baa30b53613c-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v\" (UID: \"2975edb7-e9cc-4793-acac-baa30b53613c\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:21.778510 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.778493 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dnhj\" (UniqueName: \"kubernetes.io/projected/2975edb7-e9cc-4793-acac-baa30b53613c-kube-api-access-4dnhj\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v\" (UID: \"2975edb7-e9cc-4793-acac-baa30b53613c\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:21.778543 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.778517 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2975edb7-e9cc-4793-acac-baa30b53613c-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v\" (UID: \"2975edb7-e9cc-4793-acac-baa30b53613c\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:21.778543 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.778537 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2975edb7-e9cc-4793-acac-baa30b53613c-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v\" (UID: \"2975edb7-e9cc-4793-acac-baa30b53613c\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:21.880034 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.878692 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:21.880034 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.879481 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2975edb7-e9cc-4793-acac-baa30b53613c-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v\" (UID: \"2975edb7-e9cc-4793-acac-baa30b53613c\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:21.881368 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.880686 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2975edb7-e9cc-4793-acac-baa30b53613c-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v\" (UID: \"2975edb7-e9cc-4793-acac-baa30b53613c\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:21.881368 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.880760 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2975edb7-e9cc-4793-acac-baa30b53613c-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v\" (UID: \"2975edb7-e9cc-4793-acac-baa30b53613c\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:21.881368 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.880828 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2975edb7-e9cc-4793-acac-baa30b53613c-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v\" (UID: \"2975edb7-e9cc-4793-acac-baa30b53613c\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:21.881368 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.881193 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2975edb7-e9cc-4793-acac-baa30b53613c-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v\" (UID: \"2975edb7-e9cc-4793-acac-baa30b53613c\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:21.881368 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.881237 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2975edb7-e9cc-4793-acac-baa30b53613c-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v\" (UID: \"2975edb7-e9cc-4793-acac-baa30b53613c\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:21.881368 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.881264 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2975edb7-e9cc-4793-acac-baa30b53613c-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v\" (UID: \"2975edb7-e9cc-4793-acac-baa30b53613c\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:21.881368 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.881292 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2975edb7-e9cc-4793-acac-baa30b53613c-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v\" (UID: \"2975edb7-e9cc-4793-acac-baa30b53613c\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:21.881822 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.881610 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2975edb7-e9cc-4793-acac-baa30b53613c-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v\" (UID: \"2975edb7-e9cc-4793-acac-baa30b53613c\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:21.882548 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.881885 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dnhj\" (UniqueName: \"kubernetes.io/projected/2975edb7-e9cc-4793-acac-baa30b53613c-kube-api-access-4dnhj\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v\" (UID: \"2975edb7-e9cc-4793-acac-baa30b53613c\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:21.882548 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.881918 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2975edb7-e9cc-4793-acac-baa30b53613c-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v\" (UID: \"2975edb7-e9cc-4793-acac-baa30b53613c\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:21.882548 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.881948 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2975edb7-e9cc-4793-acac-baa30b53613c-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v\" (UID: \"2975edb7-e9cc-4793-acac-baa30b53613c\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:21.882548 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.882331 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2975edb7-e9cc-4793-acac-baa30b53613c-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v\" (UID: \"2975edb7-e9cc-4793-acac-baa30b53613c\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:21.882826 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.882774 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2975edb7-e9cc-4793-acac-baa30b53613c-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v\" (UID: \"2975edb7-e9cc-4793-acac-baa30b53613c\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:21.882886 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.882871 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2975edb7-e9cc-4793-acac-baa30b53613c-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v\" (UID: \"2975edb7-e9cc-4793-acac-baa30b53613c\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:21.883998 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.883967 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2975edb7-e9cc-4793-acac-baa30b53613c-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v\" (UID: \"2975edb7-e9cc-4793-acac-baa30b53613c\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:21.904405 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.904371 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dnhj\" (UniqueName: \"kubernetes.io/projected/2975edb7-e9cc-4793-acac-baa30b53613c-kube-api-access-4dnhj\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v\" (UID: \"2975edb7-e9cc-4793-acac-baa30b53613c\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:21.905255 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:21.905230 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2975edb7-e9cc-4793-acac-baa30b53613c-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v\" (UID: \"2975edb7-e9cc-4793-acac-baa30b53613c\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:22.012684 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:22.012598 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:22.128796 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:22.128344 6841 patch_prober.go:29] interesting pod/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 31 12:52:22.128796 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:22.128406 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" podUID="f3bee1b8-5152-4805-a59d-7d05594142de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 31 12:52:22.133013 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:22.132909 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:22.370129 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:22.366613 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:22.370129 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:22.367375 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v"] Oct 31 12:52:22.372053 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:22.371968 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:52:22.372163 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:22.372088 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:52:22.372163 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:22.372120 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:52:22.712192 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:22.712156 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" event={"ID":"2975edb7-e9cc-4793-acac-baa30b53613c","Type":"ContainerStarted","Data":"0541c2e1998aca3a1d4d3ca605532a1e2e984bcfa0bd581f092d11f2fe0e4c3b"} Oct 31 12:52:22.712385 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:22.712230 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" event={"ID":"2975edb7-e9cc-4793-acac-baa30b53613c","Type":"ContainerStarted","Data":"14ef2df8dc3d761b1bc376fb59614dc958c4ed94dd062a83f6e4f9a5deab082e"} Oct 31 12:52:22.750005 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:22.748052 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" podStartSLOduration=1.748031962 podStartE2EDuration="1.748031962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:52:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:52:22.739093838 +0000 UTC m=+1189.222216742" watchObservedRunningTime="2026-10-31 12:52:22.748031962 +0000 UTC m=+1189.231154856" Oct 31 12:52:22.931003 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:22.930732 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:23.013709 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:23.013624 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:23.019508 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:23.019480 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:23.153541 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:23.153499 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:23.192166 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:23.187382 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:52:23.192166 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:23.188799 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:52:23.198004 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:23.194628 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:52:23.712713 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:23.712680 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:23.717827 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:23.717798 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:23.718803 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:23.718782 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:23.855952 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:23.855922 6841 patch_prober.go:29] interesting pod/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 12:52:23.856149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:23.855996 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" podUID="0a039273-4b94-4793-b713-1524da46eed6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 12:52:24.729415 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:24.729381 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:25.034865 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.034777 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:25.255933 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.255889 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:25.650442 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.650420 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:52:25.729763 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.729732 6841 generic.go:356] "Generic (PLEG): container finished" podID="0a039273-4b94-4793-b713-1524da46eed6" containerID="7c98135a7d0af4e7a4a221ca51b8777502c31dd0af8844638c44faa715934a0e" exitCode=0 Oct 31 12:52:25.730297 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.729800 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" Oct 31 12:52:25.730297 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.729817 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" event={"ID":"0a039273-4b94-4793-b713-1524da46eed6","Type":"ContainerDied","Data":"7c98135a7d0af4e7a4a221ca51b8777502c31dd0af8844638c44faa715934a0e"} Oct 31 12:52:25.730297 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.729897 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj" event={"ID":"0a039273-4b94-4793-b713-1524da46eed6","Type":"ContainerDied","Data":"c3ffb88a4cb1bd3f951d58d818a54c31d3a31ae78b05a230fd22a9d949763a8c"} Oct 31 12:52:25.730297 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.729933 6841 scope.go:122] "RemoveContainer" containerID="7c98135a7d0af4e7a4a221ca51b8777502c31dd0af8844638c44faa715934a0e" Oct 31 12:52:25.741190 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.740326 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a039273-4b94-4793-b713-1524da46eed6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a039273-4b94-4793-b713-1524da46eed6-workload-socket\") pod \"0a039273-4b94-4793-b713-1524da46eed6\" (UID: \"0a039273-4b94-4793-b713-1524da46eed6\") " Oct 31 12:52:25.741190 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.740400 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a039273-4b94-4793-b713-1524da46eed6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a039273-4b94-4793-b713-1524da46eed6-credential-socket\") pod \"0a039273-4b94-4793-b713-1524da46eed6\" (UID: \"0a039273-4b94-4793-b713-1524da46eed6\") " Oct 31 12:52:25.741190 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.740427 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a039273-4b94-4793-b713-1524da46eed6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a039273-4b94-4793-b713-1524da46eed6-istio-envoy\") pod \"0a039273-4b94-4793-b713-1524da46eed6\" (UID: \"0a039273-4b94-4793-b713-1524da46eed6\") " Oct 31 12:52:25.741190 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.740465 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a039273-4b94-4793-b713-1524da46eed6-istio-token\" (UniqueName: \"kubernetes.io/projected/0a039273-4b94-4793-b713-1524da46eed6-istio-token\") pod \"0a039273-4b94-4793-b713-1524da46eed6\" (UID: \"0a039273-4b94-4793-b713-1524da46eed6\") " Oct 31 12:52:25.741190 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.740496 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a039273-4b94-4793-b713-1524da46eed6-kube-api-access-ckfnb\" (UniqueName: \"kubernetes.io/projected/0a039273-4b94-4793-b713-1524da46eed6-kube-api-access-ckfnb\") pod \"0a039273-4b94-4793-b713-1524da46eed6\" (UID: \"0a039273-4b94-4793-b713-1524da46eed6\") " Oct 31 12:52:25.741190 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.740556 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a039273-4b94-4793-b713-1524da46eed6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a039273-4b94-4793-b713-1524da46eed6-istio-podinfo\") pod \"0a039273-4b94-4793-b713-1524da46eed6\" (UID: \"0a039273-4b94-4793-b713-1524da46eed6\") " Oct 31 12:52:25.741190 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.740609 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a039273-4b94-4793-b713-1524da46eed6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a039273-4b94-4793-b713-1524da46eed6-workload-certs\") pod \"0a039273-4b94-4793-b713-1524da46eed6\" (UID: \"0a039273-4b94-4793-b713-1524da46eed6\") " Oct 31 12:52:25.741190 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.740645 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a039273-4b94-4793-b713-1524da46eed6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a039273-4b94-4793-b713-1524da46eed6-istio-data\") pod \"0a039273-4b94-4793-b713-1524da46eed6\" (UID: \"0a039273-4b94-4793-b713-1524da46eed6\") " Oct 31 12:52:25.741190 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.740693 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a039273-4b94-4793-b713-1524da46eed6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a039273-4b94-4793-b713-1524da46eed6-istiod-ca-cert\") pod \"0a039273-4b94-4793-b713-1524da46eed6\" (UID: \"0a039273-4b94-4793-b713-1524da46eed6\") " Oct 31 12:52:25.741190 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.741022 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a039273-4b94-4793-b713-1524da46eed6-credential-socket" pod "0a039273-4b94-4793-b713-1524da46eed6" (UID: "0a039273-4b94-4793-b713-1524da46eed6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:52:25.741190 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.741116 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a039273-4b94-4793-b713-1524da46eed6-workload-socket" pod "0a039273-4b94-4793-b713-1524da46eed6" (UID: "0a039273-4b94-4793-b713-1524da46eed6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:52:25.741737 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.741199 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a039273-4b94-4793-b713-1524da46eed6-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:25.741737 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.741350 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a039273-4b94-4793-b713-1524da46eed6-workload-certs" pod "0a039273-4b94-4793-b713-1524da46eed6" (UID: "0a039273-4b94-4793-b713-1524da46eed6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:52:25.741737 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.741513 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a039273-4b94-4793-b713-1524da46eed6-istio-data" pod "0a039273-4b94-4793-b713-1524da46eed6" (UID: "0a039273-4b94-4793-b713-1524da46eed6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:52:25.742081 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.741965 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a039273-4b94-4793-b713-1524da46eed6-istiod-ca-cert" pod "0a039273-4b94-4793-b713-1524da46eed6" (UID: "0a039273-4b94-4793-b713-1524da46eed6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:52:25.742203 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.742181 6841 scope.go:122] "RemoveContainer" containerID="ee4b9ffc8ecf462fba7eabafa696da97a95f0aad3e447f4092653e885593b28d" Oct 31 12:52:25.743011 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.742957 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a039273-4b94-4793-b713-1524da46eed6-istio-token" pod "0a039273-4b94-4793-b713-1524da46eed6" (UID: "0a039273-4b94-4793-b713-1524da46eed6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:52:25.743259 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.743234 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a039273-4b94-4793-b713-1524da46eed6-istio-podinfo" pod "0a039273-4b94-4793-b713-1524da46eed6" (UID: "0a039273-4b94-4793-b713-1524da46eed6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:52:25.743335 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.743235 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a039273-4b94-4793-b713-1524da46eed6-istio-envoy" pod "0a039273-4b94-4793-b713-1524da46eed6" (UID: "0a039273-4b94-4793-b713-1524da46eed6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:52:25.743959 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.743939 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a039273-4b94-4793-b713-1524da46eed6-kube-api-access-ckfnb" pod "0a039273-4b94-4793-b713-1524da46eed6" (UID: "0a039273-4b94-4793-b713-1524da46eed6"). InnerVolumeSpecName "kube-api-access-ckfnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:52:25.763532 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.763516 6841 scope.go:122] "RemoveContainer" containerID="7c98135a7d0af4e7a4a221ca51b8777502c31dd0af8844638c44faa715934a0e" Oct 31 12:52:25.763810 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:52:25.763782 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c98135a7d0af4e7a4a221ca51b8777502c31dd0af8844638c44faa715934a0e\": container with ID starting with 7c98135a7d0af4e7a4a221ca51b8777502c31dd0af8844638c44faa715934a0e not found: ID does not exist" containerID="7c98135a7d0af4e7a4a221ca51b8777502c31dd0af8844638c44faa715934a0e" Oct 31 12:52:25.763901 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.763817 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c98135a7d0af4e7a4a221ca51b8777502c31dd0af8844638c44faa715934a0e"} err="failed to get container status \"7c98135a7d0af4e7a4a221ca51b8777502c31dd0af8844638c44faa715934a0e\": rpc error: code = NotFound desc = could not find container \"7c98135a7d0af4e7a4a221ca51b8777502c31dd0af8844638c44faa715934a0e\": container with ID starting with 7c98135a7d0af4e7a4a221ca51b8777502c31dd0af8844638c44faa715934a0e not found: ID does not exist" Oct 31 12:52:25.763901 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.763834 6841 scope.go:122] "RemoveContainer" containerID="ee4b9ffc8ecf462fba7eabafa696da97a95f0aad3e447f4092653e885593b28d" Oct 31 12:52:25.764085 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:52:25.764064 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee4b9ffc8ecf462fba7eabafa696da97a95f0aad3e447f4092653e885593b28d\": container with ID starting with ee4b9ffc8ecf462fba7eabafa696da97a95f0aad3e447f4092653e885593b28d not found: ID does not exist" containerID="ee4b9ffc8ecf462fba7eabafa696da97a95f0aad3e447f4092653e885593b28d" Oct 31 12:52:25.764128 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.764091 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee4b9ffc8ecf462fba7eabafa696da97a95f0aad3e447f4092653e885593b28d"} err="failed to get container status \"ee4b9ffc8ecf462fba7eabafa696da97a95f0aad3e447f4092653e885593b28d\": rpc error: code = NotFound desc = could not find container \"ee4b9ffc8ecf462fba7eabafa696da97a95f0aad3e447f4092653e885593b28d\": container with ID starting with ee4b9ffc8ecf462fba7eabafa696da97a95f0aad3e447f4092653e885593b28d not found: ID does not exist" Oct 31 12:52:25.799111 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.799082 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:25.842574 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.842551 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a039273-4b94-4793-b713-1524da46eed6-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:25.842574 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.842572 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a039273-4b94-4793-b713-1524da46eed6-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:25.842712 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.842586 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckfnb\" (UniqueName: \"kubernetes.io/projected/0a039273-4b94-4793-b713-1524da46eed6-kube-api-access-ckfnb\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:25.842712 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.842600 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a039273-4b94-4793-b713-1524da46eed6-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:25.842712 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.842611 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a039273-4b94-4793-b713-1524da46eed6-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:25.842712 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.842619 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a039273-4b94-4793-b713-1524da46eed6-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:25.842712 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.842627 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a039273-4b94-4793-b713-1524da46eed6-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:25.842712 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:25.842636 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a039273-4b94-4793-b713-1524da46eed6-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:26.059842 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:26.059807 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj"] Oct 31 12:52:26.063484 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:26.063453 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b03bbf98-openshift-default-645744c5b9-fskwj"] Oct 31 12:52:26.434507 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:26.434419 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:26.447523 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:26.447498 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7ae9e7-65f7d7db79-bsvx2"] Oct 31 12:52:26.447977 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:26.447962 6841 cpu_manager.go:414] "containerMap: removing container" podUID="0a039273-4b94-4793-b713-1524da46eed6" containerName="istio-proxy" Oct 31 12:52:26.447977 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:26.447978 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a039273-4b94-4793-b713-1524da46eed6" containerName="istio-proxy" Oct 31 12:52:26.447977 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:26.447999 6841 cpu_manager.go:414] "containerMap: removing container" podUID="0a039273-4b94-4793-b713-1524da46eed6" containerName="istio-proxy" Oct 31 12:52:26.448110 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:26.448004 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a039273-4b94-4793-b713-1524da46eed6" containerName="istio-proxy" Oct 31 12:52:26.448110 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:26.448077 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a039273-4b94-4793-b713-1524da46eed6" containerName="istio-proxy" Oct 31 12:52:26.448110 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:26.448086 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a039273-4b94-4793-b713-1524da46eed6" containerName="istio-proxy" Oct 31 12:52:26.452612 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:26.452596 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ae9e7-65f7d7db79-bsvx2" Oct 31 12:52:26.464536 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:26.464511 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-7ae9e7\"" Oct 31 12:52:26.473079 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:26.473048 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7ae9e7-65f7d7db79-bsvx2"] Oct 31 12:52:26.475113 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:26.475094 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-7ae9e7-dockercfg-pdpbs\"" Oct 31 12:52:26.576006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:26.575217 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7ae9e7-57b99fbcdc-rqs6p"] Oct 31 12:52:26.576006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:26.575789 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6xzz\" (UniqueName: \"kubernetes.io/projected/716df414-c063-4bf8-b85c-fcb3c319fb70-kube-api-access-r6xzz\") pod \"payload-pre-processing-e2e-worker-w4-7ae9e7-65f7d7db79-bsvx2\" (UID: \"716df414-c063-4bf8-b85c-fcb3c319fb70\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ae9e7-65f7d7db79-bsvx2" Oct 31 12:52:26.576006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:26.575935 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/716df414-c063-4bf8-b85c-fcb3c319fb70-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-7ae9e7-65f7d7db79-bsvx2\" (UID: \"716df414-c063-4bf8-b85c-fcb3c319fb70\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ae9e7-65f7d7db79-bsvx2" Oct 31 12:52:26.596038 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:26.591182 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7ae9e7-57b99fbcdc-rqs6p"] Oct 31 12:52:26.596038 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:26.591307 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ae9e7-57b99fbcdc-rqs6p" Oct 31 12:52:26.672129 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:26.672090 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:26.677288 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:26.677265 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f721bc13-8ce6-4d6b-b53a-65b9c0030e08-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-7ae9e7-57b99fbcdc-rqs6p\" (UID: \"f721bc13-8ce6-4d6b-b53a-65b9c0030e08\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7ae9e7-57b99fbcdc-rqs6p" Oct 31 12:52:26.677369 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:26.677305 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/716df414-c063-4bf8-b85c-fcb3c319fb70-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-7ae9e7-65f7d7db79-bsvx2\" (UID: \"716df414-c063-4bf8-b85c-fcb3c319fb70\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ae9e7-65f7d7db79-bsvx2" Oct 31 12:52:26.677471 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:26.677449 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw8cc\" (UniqueName: \"kubernetes.io/projected/f721bc13-8ce6-4d6b-b53a-65b9c0030e08-kube-api-access-gw8cc\") pod \"payload-processing-e2e-worker-w4-7ae9e7-57b99fbcdc-rqs6p\" (UID: \"f721bc13-8ce6-4d6b-b53a-65b9c0030e08\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7ae9e7-57b99fbcdc-rqs6p" Oct 31 12:52:26.677575 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:26.677557 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6xzz\" (UniqueName: \"kubernetes.io/projected/716df414-c063-4bf8-b85c-fcb3c319fb70-kube-api-access-r6xzz\") pod \"payload-pre-processing-e2e-worker-w4-7ae9e7-65f7d7db79-bsvx2\" (UID: \"716df414-c063-4bf8-b85c-fcb3c319fb70\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ae9e7-65f7d7db79-bsvx2" Oct 31 12:52:26.678080 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:26.678062 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/716df414-c063-4bf8-b85c-fcb3c319fb70-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-7ae9e7-65f7d7db79-bsvx2\" (UID: \"716df414-c063-4bf8-b85c-fcb3c319fb70\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ae9e7-65f7d7db79-bsvx2" Oct 31 12:52:26.698163 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:26.698132 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6xzz\" (UniqueName: \"kubernetes.io/projected/716df414-c063-4bf8-b85c-fcb3c319fb70-kube-api-access-r6xzz\") pod \"payload-pre-processing-e2e-worker-w4-7ae9e7-65f7d7db79-bsvx2\" (UID: \"716df414-c063-4bf8-b85c-fcb3c319fb70\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ae9e7-65f7d7db79-bsvx2" Oct 31 12:52:26.767241 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:26.767207 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ae9e7-65f7d7db79-bsvx2" Oct 31 12:52:26.789774 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:26.789654 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw8cc\" (UniqueName: \"kubernetes.io/projected/f721bc13-8ce6-4d6b-b53a-65b9c0030e08-kube-api-access-gw8cc\") pod \"payload-processing-e2e-worker-w4-7ae9e7-57b99fbcdc-rqs6p\" (UID: \"f721bc13-8ce6-4d6b-b53a-65b9c0030e08\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7ae9e7-57b99fbcdc-rqs6p" Oct 31 12:52:26.792957 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:26.792923 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f721bc13-8ce6-4d6b-b53a-65b9c0030e08-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-7ae9e7-57b99fbcdc-rqs6p\" (UID: \"f721bc13-8ce6-4d6b-b53a-65b9c0030e08\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7ae9e7-57b99fbcdc-rqs6p" Oct 31 12:52:26.794202 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:26.794127 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f721bc13-8ce6-4d6b-b53a-65b9c0030e08-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-7ae9e7-57b99fbcdc-rqs6p\" (UID: \"f721bc13-8ce6-4d6b-b53a-65b9c0030e08\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7ae9e7-57b99fbcdc-rqs6p" Oct 31 12:52:26.811755 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:26.811725 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw8cc\" (UniqueName: \"kubernetes.io/projected/f721bc13-8ce6-4d6b-b53a-65b9c0030e08-kube-api-access-gw8cc\") pod \"payload-processing-e2e-worker-w4-7ae9e7-57b99fbcdc-rqs6p\" (UID: \"f721bc13-8ce6-4d6b-b53a-65b9c0030e08\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7ae9e7-57b99fbcdc-rqs6p" Oct 31 12:52:26.893155 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:26.893119 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:26.921994 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:26.921960 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ae9e7-57b99fbcdc-rqs6p" Oct 31 12:52:27.069502 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:27.069478 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7ae9e7-65f7d7db79-bsvx2"] Oct 31 12:52:27.071712 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:52:27.071685 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod716df414_c063_4bf8_b85c_fcb3c319fb70.slice/crio-6514dc8ab28c2c35ed146bfff74cea74042ea0a49eed19dde899988479a8958f WatchSource:0}: Error finding container 6514dc8ab28c2c35ed146bfff74cea74042ea0a49eed19dde899988479a8958f: Status 404 returned error can't find the container with id 6514dc8ab28c2c35ed146bfff74cea74042ea0a49eed19dde899988479a8958f Oct 31 12:52:27.200936 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:27.200913 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7ae9e7-57b99fbcdc-rqs6p"] Oct 31 12:52:27.202390 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:52:27.202357 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf721bc13_8ce6_4d6b_b53a_65b9c0030e08.slice/crio-7d46bff8eebed35cb1a3897a36367f2aaaeebb50e641fedadad1e0cc2fdb896a WatchSource:0}: Error finding container 7d46bff8eebed35cb1a3897a36367f2aaaeebb50e641fedadad1e0cc2fdb896a: Status 404 returned error can't find the container with id 7d46bff8eebed35cb1a3897a36367f2aaaeebb50e641fedadad1e0cc2fdb896a Oct 31 12:52:27.450607 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:27.450532 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:27.642259 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:27.642185 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a039273-4b94-4793-b713-1524da46eed6" path="/var/lib/kubelet/pods/0a039273-4b94-4793-b713-1524da46eed6/volumes" Oct 31 12:52:27.696814 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:27.696784 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:27.768007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:27.765091 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ae9e7-65f7d7db79-bsvx2" event={"ID":"716df414-c063-4bf8-b85c-fcb3c319fb70","Type":"ContainerStarted","Data":"0effe3ab060992c2a9fe684dea2c2681dde7e8cada2b4b031c692d8d1be60392"} Oct 31 12:52:27.768007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:27.765160 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ae9e7-65f7d7db79-bsvx2" event={"ID":"716df414-c063-4bf8-b85c-fcb3c319fb70","Type":"ContainerStarted","Data":"6514dc8ab28c2c35ed146bfff74cea74042ea0a49eed19dde899988479a8958f"} Oct 31 12:52:27.768007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:27.766303 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ae9e7-65f7d7db79-bsvx2" Oct 31 12:52:27.781033 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:27.768849 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ae9e7-57b99fbcdc-rqs6p" event={"ID":"f721bc13-8ce6-4d6b-b53a-65b9c0030e08","Type":"ContainerStarted","Data":"a9e2f283a5b7e8d250af5e5eb65afe802b9cbda17666c5c1309c0cf2ba635c30"} Oct 31 12:52:27.781033 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:27.768902 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ae9e7-57b99fbcdc-rqs6p" event={"ID":"f721bc13-8ce6-4d6b-b53a-65b9c0030e08","Type":"ContainerStarted","Data":"7d46bff8eebed35cb1a3897a36367f2aaaeebb50e641fedadad1e0cc2fdb896a"} Oct 31 12:52:27.781033 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:27.769536 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ae9e7-57b99fbcdc-rqs6p" Oct 31 12:52:27.792757 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:27.792706 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ae9e7-65f7d7db79-bsvx2" podStartSLOduration=1.792690001 podStartE2EDuration="1.792690001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:52:27.790563976 +0000 UTC m=+1194.273686877" watchObservedRunningTime="2026-10-31 12:52:27.792690001 +0000 UTC m=+1194.275812902" Oct 31 12:52:27.814191 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:27.814134 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ae9e7-57b99fbcdc-rqs6p" podStartSLOduration=1.8141171790000001 podStartE2EDuration="1.814117179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:52:27.811927835 +0000 UTC m=+1194.295050737" watchObservedRunningTime="2026-10-31 12:52:27.814117179 +0000 UTC m=+1194.297240079" Oct 31 12:52:27.907808 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:27.907734 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:28.407732 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:28.407681 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:28.636503 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:28.636470 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:28.857169 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:28.857139 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:29.152244 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:29.152152 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:29.636442 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:29.636395 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:29.885028 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:29.884957 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:30.146069 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:30.146036 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:30.448603 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:30.448566 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:30.652624 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:30.652584 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d48f454a-f87456947-s49tk"] Oct 31 12:52:30.652903 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:30.652843 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d48f454a-f87456947-s49tk" podUID="3689d1f7-970f-48db-8bee-c50ffae8ab74" containerName="maas-api" containerID="cri-o://04bdfb681df390fde65b6629a1079dad14500a2051ae926944eb0cbbc4dd20d0" gracePeriod=30 Oct 31 12:52:30.788186 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:30.788155 6841 generic.go:356] "Generic (PLEG): container finished" podID="3689d1f7-970f-48db-8bee-c50ffae8ab74" containerID="04bdfb681df390fde65b6629a1079dad14500a2051ae926944eb0cbbc4dd20d0" exitCode=0 Oct 31 12:52:30.788363 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:30.788227 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d48f454a-f87456947-s49tk" event={"ID":"3689d1f7-970f-48db-8bee-c50ffae8ab74","Type":"ContainerDied","Data":"04bdfb681df390fde65b6629a1079dad14500a2051ae926944eb0cbbc4dd20d0"} Oct 31 12:52:30.832888 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:30.832846 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:30.959910 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:30.959879 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d48f454a-f87456947-s49tk" Oct 31 12:52:31.072965 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:31.072889 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3689d1f7-970f-48db-8bee-c50ffae8ab74-kube-api-access-5mhbt\" (UniqueName: \"kubernetes.io/projected/3689d1f7-970f-48db-8bee-c50ffae8ab74-kube-api-access-5mhbt\") pod \"3689d1f7-970f-48db-8bee-c50ffae8ab74\" (UID: \"3689d1f7-970f-48db-8bee-c50ffae8ab74\") " Oct 31 12:52:31.072965 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:31.072935 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3689d1f7-970f-48db-8bee-c50ffae8ab74-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3689d1f7-970f-48db-8bee-c50ffae8ab74-maas-api-tls\") pod \"3689d1f7-970f-48db-8bee-c50ffae8ab74\" (UID: \"3689d1f7-970f-48db-8bee-c50ffae8ab74\") " Oct 31 12:52:31.073167 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:31.073018 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3689d1f7-970f-48db-8bee-c50ffae8ab74-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3689d1f7-970f-48db-8bee-c50ffae8ab74-metrics-certs\") pod \"3689d1f7-970f-48db-8bee-c50ffae8ab74\" (UID: \"3689d1f7-970f-48db-8bee-c50ffae8ab74\") " Oct 31 12:52:31.073167 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:31.073084 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3689d1f7-970f-48db-8bee-c50ffae8ab74-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3689d1f7-970f-48db-8bee-c50ffae8ab74-openshift-service-ca\") pod \"3689d1f7-970f-48db-8bee-c50ffae8ab74\" (UID: \"3689d1f7-970f-48db-8bee-c50ffae8ab74\") " Oct 31 12:52:31.073484 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:31.073455 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3689d1f7-970f-48db-8bee-c50ffae8ab74-openshift-service-ca" pod "3689d1f7-970f-48db-8bee-c50ffae8ab74" (UID: "3689d1f7-970f-48db-8bee-c50ffae8ab74"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:52:31.075306 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:31.075275 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3689d1f7-970f-48db-8bee-c50ffae8ab74-metrics-certs" pod "3689d1f7-970f-48db-8bee-c50ffae8ab74" (UID: "3689d1f7-970f-48db-8bee-c50ffae8ab74"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:52:31.075306 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:31.075291 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3689d1f7-970f-48db-8bee-c50ffae8ab74-maas-api-tls" pod "3689d1f7-970f-48db-8bee-c50ffae8ab74" (UID: "3689d1f7-970f-48db-8bee-c50ffae8ab74"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:52:31.075306 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:31.075299 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3689d1f7-970f-48db-8bee-c50ffae8ab74-kube-api-access-5mhbt" pod "3689d1f7-970f-48db-8bee-c50ffae8ab74" (UID: "3689d1f7-970f-48db-8bee-c50ffae8ab74"). InnerVolumeSpecName "kube-api-access-5mhbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:52:31.172783 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:31.172734 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:31.173960 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:31.173935 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3689d1f7-970f-48db-8bee-c50ffae8ab74-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:31.174051 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:31.173966 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mhbt\" (UniqueName: \"kubernetes.io/projected/3689d1f7-970f-48db-8bee-c50ffae8ab74-kube-api-access-5mhbt\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:31.174051 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:31.173996 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3689d1f7-970f-48db-8bee-c50ffae8ab74-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:31.174051 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:31.174006 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3689d1f7-970f-48db-8bee-c50ffae8ab74-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:31.310588 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:31.310555 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr"] Oct 31 12:52:31.310882 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:31.310856 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" podUID="a83131c5-7e8a-4e3f-8a87-aa1d3c091023" containerName="istio-proxy" containerID="cri-o://774de72b8a36367c51bf6b72ab48f17f4d610d0dd1c14309547e707c5908468b" gracePeriod=30 Oct 31 12:52:31.433415 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:31.433313 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:31.647887 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:31.647849 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:31.794783 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:31.794744 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d48f454a-f87456947-s49tk" event={"ID":"3689d1f7-970f-48db-8bee-c50ffae8ab74","Type":"ContainerDied","Data":"2d8bda3ced22fe6f086e293d453bec92355b251a2198fbcf81588be5b7f406d4"} Oct 31 12:52:31.794932 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:31.794800 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d48f454a-f87456947-s49tk" Oct 31 12:52:31.794932 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:31.794817 6841 scope.go:122] "RemoveContainer" containerID="04bdfb681df390fde65b6629a1079dad14500a2051ae926944eb0cbbc4dd20d0" Oct 31 12:52:31.816400 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:31.816373 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d48f454a-f87456947-s49tk"] Oct 31 12:52:31.822279 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:31.822257 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d48f454a-f87456947-s49tk"] Oct 31 12:52:32.037950 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:32.037914 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:32.270760 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:32.270712 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:32.489235 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:32.489201 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:32.529163 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:32.529084 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g"] Oct 31 12:52:32.529382 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:32.529349 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" podUID="f3bee1b8-5152-4805-a59d-7d05594142de" containerName="istio-proxy" containerID="cri-o://a3b81df16ff0d66c25dc0720c9d96f1af07657b55a784f380eea562bdce32435" gracePeriod=30 Oct 31 12:52:32.717023 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:32.712572 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:32.960195 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:32.960159 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:33.166824 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:33.166779 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:33.447262 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:33.447226 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:33.667546 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:33.667501 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3689d1f7-970f-48db-8bee-c50ffae8ab74" path="/var/lib/kubelet/pods/3689d1f7-970f-48db-8bee-c50ffae8ab74/volumes" Oct 31 12:52:33.733442 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:33.733353 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:33.999512 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:33.999418 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:34.326312 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:34.326227 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:34.659726 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:34.659628 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:34.928940 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:34.928848 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-107955-86f6b68d45-ztxhj"] Oct 31 12:52:34.929489 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:34.929467 6841 cpu_manager.go:414] "containerMap: removing container" podUID="3689d1f7-970f-48db-8bee-c50ffae8ab74" containerName="maas-api" Oct 31 12:52:34.929589 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:34.929491 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3689d1f7-970f-48db-8bee-c50ffae8ab74" containerName="maas-api" Oct 31 12:52:34.929665 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:34.929652 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="3689d1f7-970f-48db-8bee-c50ffae8ab74" containerName="maas-api" Oct 31 12:52:34.933899 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:34.933878 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-107955-86f6b68d45-ztxhj" Oct 31 12:52:34.975407 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:34.975376 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-107955\"" Oct 31 12:52:34.975771 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:34.975745 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-107955-dockercfg-dp8wc\"" Oct 31 12:52:34.976741 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:34.976713 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:34.979617 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:34.979587 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-107955-86f6b68d45-ztxhj"] Oct 31 12:52:35.037746 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:35.037704 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm6zb\" (UniqueName: \"kubernetes.io/projected/481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe-kube-api-access-cm6zb\") pod \"payload-processing-e2e-worker-w0-107955-86f6b68d45-ztxhj\" (UID: \"481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-107955-86f6b68d45-ztxhj" Oct 31 12:52:35.047177 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:35.047133 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-107955-86f6b68d45-ztxhj\" (UID: \"481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-107955-86f6b68d45-ztxhj" Oct 31 12:52:35.164633 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:35.164591 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-107955-86f6b68d45-ztxhj\" (UID: \"481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-107955-86f6b68d45-ztxhj" Oct 31 12:52:35.165023 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:35.164974 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm6zb\" (UniqueName: \"kubernetes.io/projected/481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe-kube-api-access-cm6zb\") pod \"payload-processing-e2e-worker-w0-107955-86f6b68d45-ztxhj\" (UID: \"481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-107955-86f6b68d45-ztxhj" Oct 31 12:52:35.166470 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:35.166445 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-107955-86f6b68d45-ztxhj\" (UID: \"481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-107955-86f6b68d45-ztxhj" Oct 31 12:52:35.177676 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:35.177633 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm6zb\" (UniqueName: \"kubernetes.io/projected/481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe-kube-api-access-cm6zb\") pod \"payload-processing-e2e-worker-w0-107955-86f6b68d45-ztxhj\" (UID: \"481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-107955-86f6b68d45-ztxhj" Oct 31 12:52:35.188227 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:35.188164 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:35.281370 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:35.281335 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-107955-86f6b68d45-ztxhj" Oct 31 12:52:35.571016 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:35.570949 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-107955-86f6b68d45-ztxhj"] Oct 31 12:52:35.688326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:35.684267 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-107955-fc749446-559d9"] Oct 31 12:52:35.690379 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:35.690349 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-107955-fc749446-559d9" Oct 31 12:52:35.693114 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:35.693083 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-107955-fc749446-559d9"] Oct 31 12:52:35.694750 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:35.694724 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-107955-serving-cert\"" Oct 31 12:52:35.695104 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:35.695085 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rhftr\"" Oct 31 12:52:35.695250 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:35.695231 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 12:52:35.695329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:35.695317 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 12:52:35.700201 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:35.698635 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 12:52:35.788031 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:35.787974 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0573f69-b5ed-475b-b579-4c084744071d-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-107955-fc749446-559d9\" (UID: \"c0573f69-b5ed-475b-b579-4c084744071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-107955-fc749446-559d9" Oct 31 12:52:35.788197 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:35.788109 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0573f69-b5ed-475b-b579-4c084744071d-metrics-certs\") pod \"maas-api-e2e-worker-w0-107955-fc749446-559d9\" (UID: \"c0573f69-b5ed-475b-b579-4c084744071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-107955-fc749446-559d9" Oct 31 12:52:35.788197 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:35.788157 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0573f69-b5ed-475b-b579-4c084744071d-maas-api-tls\") pod \"maas-api-e2e-worker-w0-107955-fc749446-559d9\" (UID: \"c0573f69-b5ed-475b-b579-4c084744071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-107955-fc749446-559d9" Oct 31 12:52:35.788270 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:35.788220 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9ln7\" (UniqueName: \"kubernetes.io/projected/c0573f69-b5ed-475b-b579-4c084744071d-kube-api-access-p9ln7\") pod \"maas-api-e2e-worker-w0-107955-fc749446-559d9\" (UID: \"c0573f69-b5ed-475b-b579-4c084744071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-107955-fc749446-559d9" Oct 31 12:52:35.823076 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:35.822995 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:35.824746 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:35.824722 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-107955-86f6b68d45-ztxhj" event={"ID":"481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe","Type":"ContainerStarted","Data":"940d9c7a7e36edbdf013cfaf00a94be25cb3828c044f69f676a2736bb67af8d4"} Oct 31 12:52:35.824868 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:35.824777 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-107955-86f6b68d45-ztxhj" event={"ID":"481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe","Type":"ContainerStarted","Data":"08eec4465150d73f872aad4fcbb0869cad9f0e413b290082ba07ea55944d9023"} Oct 31 12:52:35.824868 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:35.824808 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-107955-86f6b68d45-ztxhj" Oct 31 12:52:35.874005 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:35.865413 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-107955-86f6b68d45-ztxhj" podStartSLOduration=1.865393565 podStartE2EDuration="1.865393565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:52:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:52:35.865234625 +0000 UTC m=+1202.348357527" watchObservedRunningTime="2026-10-31 12:52:35.865393565 +0000 UTC m=+1202.348516465" Oct 31 12:52:35.902006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:35.895159 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0573f69-b5ed-475b-b579-4c084744071d-maas-api-tls\") pod \"maas-api-e2e-worker-w0-107955-fc749446-559d9\" (UID: \"c0573f69-b5ed-475b-b579-4c084744071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-107955-fc749446-559d9" Oct 31 12:52:35.902006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:35.895223 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9ln7\" (UniqueName: \"kubernetes.io/projected/c0573f69-b5ed-475b-b579-4c084744071d-kube-api-access-p9ln7\") pod \"maas-api-e2e-worker-w0-107955-fc749446-559d9\" (UID: \"c0573f69-b5ed-475b-b579-4c084744071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-107955-fc749446-559d9" Oct 31 12:52:35.902006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:35.895278 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0573f69-b5ed-475b-b579-4c084744071d-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-107955-fc749446-559d9\" (UID: \"c0573f69-b5ed-475b-b579-4c084744071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-107955-fc749446-559d9" Oct 31 12:52:35.902006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:35.895413 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0573f69-b5ed-475b-b579-4c084744071d-metrics-certs\") pod \"maas-api-e2e-worker-w0-107955-fc749446-559d9\" (UID: \"c0573f69-b5ed-475b-b579-4c084744071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-107955-fc749446-559d9" Oct 31 12:52:35.902006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:35.899523 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0573f69-b5ed-475b-b579-4c084744071d-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-107955-fc749446-559d9\" (UID: \"c0573f69-b5ed-475b-b579-4c084744071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-107955-fc749446-559d9" Oct 31 12:52:35.902006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:35.899853 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0573f69-b5ed-475b-b579-4c084744071d-metrics-certs\") pod \"maas-api-e2e-worker-w0-107955-fc749446-559d9\" (UID: \"c0573f69-b5ed-475b-b579-4c084744071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-107955-fc749446-559d9" Oct 31 12:52:35.903095 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:35.902444 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0573f69-b5ed-475b-b579-4c084744071d-maas-api-tls\") pod \"maas-api-e2e-worker-w0-107955-fc749446-559d9\" (UID: \"c0573f69-b5ed-475b-b579-4c084744071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-107955-fc749446-559d9" Oct 31 12:52:35.909252 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:35.909224 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9ln7\" (UniqueName: \"kubernetes.io/projected/c0573f69-b5ed-475b-b579-4c084744071d-kube-api-access-p9ln7\") pod \"maas-api-e2e-worker-w0-107955-fc749446-559d9\" (UID: \"c0573f69-b5ed-475b-b579-4c084744071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-107955-fc749446-559d9" Oct 31 12:52:36.012867 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.012833 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-107955-fc749446-559d9" Oct 31 12:52:36.031165 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.031132 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:36.233149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.233112 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:36.322376 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.322308 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-107955-fc749446-559d9"] Oct 31 12:52:36.672898 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.672869 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:52:36.820577 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.820543 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-istio-data\") pod \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\" (UID: \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\") " Oct 31 12:52:36.820723 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.820600 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-istio-token\" (UniqueName: \"kubernetes.io/projected/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-istio-token\") pod \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\" (UID: \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\") " Oct 31 12:52:36.820723 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.820665 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-istio-podinfo\") pod \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\" (UID: \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\") " Oct 31 12:52:36.820723 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.820703 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-istio-envoy\") pod \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\" (UID: \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\") " Oct 31 12:52:36.820889 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.820751 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-workload-certs\") pod \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\" (UID: \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\") " Oct 31 12:52:36.820889 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.820792 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-credential-socket\") pod \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\" (UID: \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\") " Oct 31 12:52:36.821024 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.820955 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-istiod-ca-cert\") pod \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\" (UID: \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\") " Oct 31 12:52:36.821088 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.821028 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-kube-api-access-g4jfk\" (UniqueName: \"kubernetes.io/projected/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-kube-api-access-g4jfk\") pod \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\" (UID: \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\") " Oct 31 12:52:36.822083 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.822056 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-workload-socket\") pod \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\" (UID: \"a83131c5-7e8a-4e3f-8a87-aa1d3c091023\") " Oct 31 12:52:36.822399 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.822371 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-workload-certs" pod "a83131c5-7e8a-4e3f-8a87-aa1d3c091023" (UID: "a83131c5-7e8a-4e3f-8a87-aa1d3c091023"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:52:36.822631 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.822605 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-workload-socket" pod "a83131c5-7e8a-4e3f-8a87-aa1d3c091023" (UID: "a83131c5-7e8a-4e3f-8a87-aa1d3c091023"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:52:36.822795 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.822775 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-istiod-ca-cert" pod "a83131c5-7e8a-4e3f-8a87-aa1d3c091023" (UID: "a83131c5-7e8a-4e3f-8a87-aa1d3c091023"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:52:36.823026 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.823009 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-credential-socket" pod "a83131c5-7e8a-4e3f-8a87-aa1d3c091023" (UID: "a83131c5-7e8a-4e3f-8a87-aa1d3c091023"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:52:36.823089 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.823054 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-istio-data" pod "a83131c5-7e8a-4e3f-8a87-aa1d3c091023" (UID: "a83131c5-7e8a-4e3f-8a87-aa1d3c091023"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:52:36.824325 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.824297 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-istio-envoy" pod "a83131c5-7e8a-4e3f-8a87-aa1d3c091023" (UID: "a83131c5-7e8a-4e3f-8a87-aa1d3c091023"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:52:36.824597 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.824573 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-istio-podinfo" pod "a83131c5-7e8a-4e3f-8a87-aa1d3c091023" (UID: "a83131c5-7e8a-4e3f-8a87-aa1d3c091023"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:52:36.825378 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.825352 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-kube-api-access-g4jfk" pod "a83131c5-7e8a-4e3f-8a87-aa1d3c091023" (UID: "a83131c5-7e8a-4e3f-8a87-aa1d3c091023"). InnerVolumeSpecName "kube-api-access-g4jfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:52:36.826152 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.826126 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-istio-token" pod "a83131c5-7e8a-4e3f-8a87-aa1d3c091023" (UID: "a83131c5-7e8a-4e3f-8a87-aa1d3c091023"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:52:36.833690 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.833665 6841 generic.go:356] "Generic (PLEG): container finished" podID="a83131c5-7e8a-4e3f-8a87-aa1d3c091023" containerID="774de72b8a36367c51bf6b72ab48f17f4d610d0dd1c14309547e707c5908468b" exitCode=0 Oct 31 12:52:36.833778 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.833717 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" Oct 31 12:52:36.833778 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.833744 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" event={"ID":"a83131c5-7e8a-4e3f-8a87-aa1d3c091023","Type":"ContainerDied","Data":"774de72b8a36367c51bf6b72ab48f17f4d610d0dd1c14309547e707c5908468b"} Oct 31 12:52:36.833876 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.833800 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr" event={"ID":"a83131c5-7e8a-4e3f-8a87-aa1d3c091023","Type":"ContainerDied","Data":"dd45ac1ad02d11c6da412d92b3397db372e6162eb5d8a7d407e7522bfccd0404"} Oct 31 12:52:36.833876 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.833830 6841 scope.go:122] "RemoveContainer" containerID="774de72b8a36367c51bf6b72ab48f17f4d610d0dd1c14309547e707c5908468b" Oct 31 12:52:36.835243 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.835188 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-107955-fc749446-559d9" event={"ID":"c0573f69-b5ed-475b-b579-4c084744071d","Type":"ContainerStarted","Data":"5146abf42ca284e3a03d1b086826171ce51d2a730429ceaff05bbf3429a968a1"} Oct 31 12:52:36.872527 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.872432 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr"] Oct 31 12:52:36.874472 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.874448 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:36.874554 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.874537 6841 scope.go:122] "RemoveContainer" containerID="7e2ff5b60ceef687ac667f8ddc28842f82e7db34d773c2d3e94fd1e2d5d2635b" Oct 31 12:52:36.876710 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.876685 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d48f454a-openshift-default-5f4ff66b7c-tb8rr"] Oct 31 12:52:36.891767 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.891748 6841 scope.go:122] "RemoveContainer" containerID="774de72b8a36367c51bf6b72ab48f17f4d610d0dd1c14309547e707c5908468b" Oct 31 12:52:36.892094 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:52:36.892069 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"774de72b8a36367c51bf6b72ab48f17f4d610d0dd1c14309547e707c5908468b\": container with ID starting with 774de72b8a36367c51bf6b72ab48f17f4d610d0dd1c14309547e707c5908468b not found: ID does not exist" containerID="774de72b8a36367c51bf6b72ab48f17f4d610d0dd1c14309547e707c5908468b" Oct 31 12:52:36.892203 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.892103 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"774de72b8a36367c51bf6b72ab48f17f4d610d0dd1c14309547e707c5908468b"} err="failed to get container status \"774de72b8a36367c51bf6b72ab48f17f4d610d0dd1c14309547e707c5908468b\": rpc error: code = NotFound desc = could not find container \"774de72b8a36367c51bf6b72ab48f17f4d610d0dd1c14309547e707c5908468b\": container with ID starting with 774de72b8a36367c51bf6b72ab48f17f4d610d0dd1c14309547e707c5908468b not found: ID does not exist" Oct 31 12:52:36.892203 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.892121 6841 scope.go:122] "RemoveContainer" containerID="7e2ff5b60ceef687ac667f8ddc28842f82e7db34d773c2d3e94fd1e2d5d2635b" Oct 31 12:52:36.892386 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:52:36.892366 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e2ff5b60ceef687ac667f8ddc28842f82e7db34d773c2d3e94fd1e2d5d2635b\": container with ID starting with 7e2ff5b60ceef687ac667f8ddc28842f82e7db34d773c2d3e94fd1e2d5d2635b not found: ID does not exist" containerID="7e2ff5b60ceef687ac667f8ddc28842f82e7db34d773c2d3e94fd1e2d5d2635b" Oct 31 12:52:36.892451 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.892397 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e2ff5b60ceef687ac667f8ddc28842f82e7db34d773c2d3e94fd1e2d5d2635b"} err="failed to get container status \"7e2ff5b60ceef687ac667f8ddc28842f82e7db34d773c2d3e94fd1e2d5d2635b\": rpc error: code = NotFound desc = could not find container \"7e2ff5b60ceef687ac667f8ddc28842f82e7db34d773c2d3e94fd1e2d5d2635b\": container with ID starting with 7e2ff5b60ceef687ac667f8ddc28842f82e7db34d773c2d3e94fd1e2d5d2635b not found: ID does not exist" Oct 31 12:52:36.922878 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.922850 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:36.922878 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.922874 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:36.922878 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.922885 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:36.922878 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.922893 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:36.923181 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.922902 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:36.923181 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.922910 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4jfk\" (UniqueName: \"kubernetes.io/projected/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-kube-api-access-g4jfk\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:36.923181 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.922918 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:36.923181 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.922926 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:36.923181 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:36.922934 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a83131c5-7e8a-4e3f-8a87-aa1d3c091023-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:37.390482 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:37.390442 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:37.636288 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:37.636244 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:37.670627 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:37.670590 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a83131c5-7e8a-4e3f-8a87-aa1d3c091023" path="/var/lib/kubelet/pods/a83131c5-7e8a-4e3f-8a87-aa1d3c091023/volumes" Oct 31 12:52:37.690870 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:52:37.690822 6841 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83131c5_7e8a_4e3f_8a87_aa1d3c091023.slice/crio-774de72b8a36367c51bf6b72ab48f17f4d610d0dd1c14309547e707c5908468b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3bee1b8_5152_4805_a59d_7d05594142de.slice/crio-conmon-a3b81df16ff0d66c25dc0720c9d96f1af07657b55a784f380eea562bdce32435.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83131c5_7e8a_4e3f_8a87_aa1d3c091023.slice/crio-conmon-774de72b8a36367c51bf6b72ab48f17f4d610d0dd1c14309547e707c5908468b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83131c5_7e8a_4e3f_8a87_aa1d3c091023.slice\": RecentStats: unable to find data in memory cache]" Oct 31 12:52:37.691349 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:52:37.690879 6841 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83131c5_7e8a_4e3f_8a87_aa1d3c091023.slice/crio-774de72b8a36367c51bf6b72ab48f17f4d610d0dd1c14309547e707c5908468b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83131c5_7e8a_4e3f_8a87_aa1d3c091023.slice\": RecentStats: unable to find data in memory cache]" Oct 31 12:52:37.879756 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:37.879726 6841 generic.go:356] "Generic (PLEG): container finished" podID="f3bee1b8-5152-4805-a59d-7d05594142de" containerID="a3b81df16ff0d66c25dc0720c9d96f1af07657b55a784f380eea562bdce32435" exitCode=0 Oct 31 12:52:37.879886 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:37.879845 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" event={"ID":"f3bee1b8-5152-4805-a59d-7d05594142de","Type":"ContainerDied","Data":"a3b81df16ff0d66c25dc0720c9d96f1af07657b55a784f380eea562bdce32435"} Oct 31 12:52:37.879927 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:37.879888 6841 scope.go:122] "RemoveContainer" containerID="0d8ec6d196a15f540c960379f94253ba259464f3f63c7d92254846e3fd7769c1" Oct 31 12:52:37.882515 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:37.882490 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-107955-fc749446-559d9" event={"ID":"c0573f69-b5ed-475b-b579-4c084744071d","Type":"ContainerStarted","Data":"422b62b920308dffe52358f6b3789f68b12b68c364efa998fc42205a60435118"} Oct 31 12:52:37.882700 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:37.882682 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-107955-fc749446-559d9" Oct 31 12:52:37.887482 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:37.887459 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:52:37.908905 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:37.908859 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-107955-fc749446-559d9" podStartSLOduration=2.440155564 podStartE2EDuration="2.908847398s" totalImagesPullingTime="468.691834ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:52:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:52:37.907792159 +0000 UTC m=+1204.390915063" watchObservedRunningTime="2026-10-31 12:52:37.908847398 +0000 UTC m=+1204.391970298" Oct 31 12:52:37.940341 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:37.940308 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3bee1b8-5152-4805-a59d-7d05594142de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3bee1b8-5152-4805-a59d-7d05594142de-istio-data\") pod \"f3bee1b8-5152-4805-a59d-7d05594142de\" (UID: \"f3bee1b8-5152-4805-a59d-7d05594142de\") " Oct 31 12:52:37.940498 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:37.940370 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3bee1b8-5152-4805-a59d-7d05594142de-istio-token\" (UniqueName: \"kubernetes.io/projected/f3bee1b8-5152-4805-a59d-7d05594142de-istio-token\") pod \"f3bee1b8-5152-4805-a59d-7d05594142de\" (UID: \"f3bee1b8-5152-4805-a59d-7d05594142de\") " Oct 31 12:52:37.940498 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:37.940453 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3bee1b8-5152-4805-a59d-7d05594142de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3bee1b8-5152-4805-a59d-7d05594142de-istio-podinfo\") pod \"f3bee1b8-5152-4805-a59d-7d05594142de\" (UID: \"f3bee1b8-5152-4805-a59d-7d05594142de\") " Oct 31 12:52:37.940498 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:37.940488 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3bee1b8-5152-4805-a59d-7d05594142de-kube-api-access-sqbsc\" (UniqueName: \"kubernetes.io/projected/f3bee1b8-5152-4805-a59d-7d05594142de-kube-api-access-sqbsc\") pod \"f3bee1b8-5152-4805-a59d-7d05594142de\" (UID: \"f3bee1b8-5152-4805-a59d-7d05594142de\") " Oct 31 12:52:37.940695 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:37.940532 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3bee1b8-5152-4805-a59d-7d05594142de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3bee1b8-5152-4805-a59d-7d05594142de-istiod-ca-cert\") pod \"f3bee1b8-5152-4805-a59d-7d05594142de\" (UID: \"f3bee1b8-5152-4805-a59d-7d05594142de\") " Oct 31 12:52:37.940695 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:37.940590 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3bee1b8-5152-4805-a59d-7d05594142de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3bee1b8-5152-4805-a59d-7d05594142de-istio-envoy\") pod \"f3bee1b8-5152-4805-a59d-7d05594142de\" (UID: \"f3bee1b8-5152-4805-a59d-7d05594142de\") " Oct 31 12:52:37.940695 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:37.940619 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3bee1b8-5152-4805-a59d-7d05594142de-istio-data" pod "f3bee1b8-5152-4805-a59d-7d05594142de" (UID: "f3bee1b8-5152-4805-a59d-7d05594142de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:52:37.940695 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:37.940637 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3bee1b8-5152-4805-a59d-7d05594142de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3bee1b8-5152-4805-a59d-7d05594142de-credential-socket\") pod \"f3bee1b8-5152-4805-a59d-7d05594142de\" (UID: \"f3bee1b8-5152-4805-a59d-7d05594142de\") " Oct 31 12:52:37.940946 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:37.940697 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3bee1b8-5152-4805-a59d-7d05594142de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3bee1b8-5152-4805-a59d-7d05594142de-workload-certs\") pod \"f3bee1b8-5152-4805-a59d-7d05594142de\" (UID: \"f3bee1b8-5152-4805-a59d-7d05594142de\") " Oct 31 12:52:37.940946 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:37.940724 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3bee1b8-5152-4805-a59d-7d05594142de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3bee1b8-5152-4805-a59d-7d05594142de-workload-socket\") pod \"f3bee1b8-5152-4805-a59d-7d05594142de\" (UID: \"f3bee1b8-5152-4805-a59d-7d05594142de\") " Oct 31 12:52:37.941081 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:37.940961 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3bee1b8-5152-4805-a59d-7d05594142de-istiod-ca-cert" pod "f3bee1b8-5152-4805-a59d-7d05594142de" (UID: "f3bee1b8-5152-4805-a59d-7d05594142de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:52:37.957050 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:37.956286 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3bee1b8-5152-4805-a59d-7d05594142de-workload-certs" pod "f3bee1b8-5152-4805-a59d-7d05594142de" (UID: "f3bee1b8-5152-4805-a59d-7d05594142de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:52:37.957050 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:37.956471 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3bee1b8-5152-4805-a59d-7d05594142de-credential-socket" pod "f3bee1b8-5152-4805-a59d-7d05594142de" (UID: "f3bee1b8-5152-4805-a59d-7d05594142de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:52:37.957050 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:37.956639 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3bee1b8-5152-4805-a59d-7d05594142de-workload-socket" pod "f3bee1b8-5152-4805-a59d-7d05594142de" (UID: "f3bee1b8-5152-4805-a59d-7d05594142de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:52:37.983033 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:37.961758 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3bee1b8-5152-4805-a59d-7d05594142de-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:38.000446 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:38.000418 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3bee1b8-5152-4805-a59d-7d05594142de-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:38.000650 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:37.982839 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3bee1b8-5152-4805-a59d-7d05594142de-istio-token" pod "f3bee1b8-5152-4805-a59d-7d05594142de" (UID: "f3bee1b8-5152-4805-a59d-7d05594142de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:52:38.000761 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:38.000706 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3bee1b8-5152-4805-a59d-7d05594142de-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:38.000858 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:38.000843 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3bee1b8-5152-4805-a59d-7d05594142de-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:38.000934 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:38.000924 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3bee1b8-5152-4805-a59d-7d05594142de-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:38.001170 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:38.001131 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3bee1b8-5152-4805-a59d-7d05594142de-istio-envoy" pod "f3bee1b8-5152-4805-a59d-7d05594142de" (UID: "f3bee1b8-5152-4805-a59d-7d05594142de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:52:38.005574 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:38.005540 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3bee1b8-5152-4805-a59d-7d05594142de-istio-podinfo" pod "f3bee1b8-5152-4805-a59d-7d05594142de" (UID: "f3bee1b8-5152-4805-a59d-7d05594142de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:52:38.005852 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:38.005811 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3bee1b8-5152-4805-a59d-7d05594142de-kube-api-access-sqbsc" pod "f3bee1b8-5152-4805-a59d-7d05594142de" (UID: "f3bee1b8-5152-4805-a59d-7d05594142de"). InnerVolumeSpecName "kube-api-access-sqbsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:52:38.024506 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:38.024472 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:38.104513 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:38.104427 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3bee1b8-5152-4805-a59d-7d05594142de-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:38.104679 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:38.104581 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqbsc\" (UniqueName: \"kubernetes.io/projected/f3bee1b8-5152-4805-a59d-7d05594142de-kube-api-access-sqbsc\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:38.104679 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:38.104608 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3bee1b8-5152-4805-a59d-7d05594142de-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:38.104679 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:38.104639 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3bee1b8-5152-4805-a59d-7d05594142de-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:38.282374 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:38.282265 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:38.503848 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:38.503808 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:38.899140 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:38.895967 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" Oct 31 12:52:38.899140 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:38.896069 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g" event={"ID":"f3bee1b8-5152-4805-a59d-7d05594142de","Type":"ContainerDied","Data":"b9ff1eece98c46b9ecd49ae93dd61d191ca1d14d53e5a6560f781a4c0b63064f"} Oct 31 12:52:38.899140 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:38.896119 6841 scope.go:122] "RemoveContainer" containerID="a3b81df16ff0d66c25dc0720c9d96f1af07657b55a784f380eea562bdce32435" Oct 31 12:52:38.968812 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:38.968698 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g"] Oct 31 12:52:38.970977 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:38.970952 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-785c04b9-openshift-default-5f9679596f-fq49g"] Oct 31 12:52:39.367188 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.367150 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-622e03-55955d5bc9-nfrwx"] Oct 31 12:52:39.367869 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.367848 6841 cpu_manager.go:414] "containerMap: removing container" podUID="a83131c5-7e8a-4e3f-8a87-aa1d3c091023" containerName="istio-proxy" Oct 31 12:52:39.367919 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.367874 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83131c5-7e8a-4e3f-8a87-aa1d3c091023" containerName="istio-proxy" Oct 31 12:52:39.367919 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.367889 6841 cpu_manager.go:414] "containerMap: removing container" podUID="f3bee1b8-5152-4805-a59d-7d05594142de" containerName="istio-proxy" Oct 31 12:52:39.367919 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.367897 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3bee1b8-5152-4805-a59d-7d05594142de" containerName="istio-proxy" Oct 31 12:52:39.368026 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.367933 6841 cpu_manager.go:414] "containerMap: removing container" podUID="f3bee1b8-5152-4805-a59d-7d05594142de" containerName="istio-proxy" Oct 31 12:52:39.368026 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.367942 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3bee1b8-5152-4805-a59d-7d05594142de" containerName="istio-proxy" Oct 31 12:52:39.368085 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.368076 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3bee1b8-5152-4805-a59d-7d05594142de" containerName="istio-proxy" Oct 31 12:52:39.368118 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.368090 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="a83131c5-7e8a-4e3f-8a87-aa1d3c091023" containerName="istio-proxy" Oct 31 12:52:39.368118 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.368104 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="a83131c5-7e8a-4e3f-8a87-aa1d3c091023" containerName="istio-proxy" Oct 31 12:52:39.372991 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.372958 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-622e03-55955d5bc9-nfrwx" Oct 31 12:52:39.377698 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.377677 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-622e03-dockercfg-xr4j4\"" Oct 31 12:52:39.378396 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.378329 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-622e03-55955d5bc9-nfrwx"] Oct 31 12:52:39.379743 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.379717 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-622e03\"" Oct 31 12:52:39.385281 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.385253 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:39.397026 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.397005 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-622e03-7574857797-f4fdg"] Oct 31 12:52:39.397479 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.397465 6841 cpu_manager.go:414] "containerMap: removing container" podUID="a83131c5-7e8a-4e3f-8a87-aa1d3c091023" containerName="istio-proxy" Oct 31 12:52:39.397520 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.397480 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83131c5-7e8a-4e3f-8a87-aa1d3c091023" containerName="istio-proxy" Oct 31 12:52:39.397606 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.397596 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3bee1b8-5152-4805-a59d-7d05594142de" containerName="istio-proxy" Oct 31 12:52:39.400913 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.400893 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-622e03-7574857797-f4fdg" Oct 31 12:52:39.418497 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.418470 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-622e03-7574857797-f4fdg"] Oct 31 12:52:39.521747 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.521702 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xktpv\" (UniqueName: \"kubernetes.io/projected/fb6f239c-4a3d-4f98-9b5e-4d775a4c561c-kube-api-access-xktpv\") pod \"payload-processing-e2e-worker-w1-622e03-7574857797-f4fdg\" (UID: \"fb6f239c-4a3d-4f98-9b5e-4d775a4c561c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-622e03-7574857797-f4fdg" Oct 31 12:52:39.522099 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.522071 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96229665-4486-4750-becf-d30f28c47dd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-622e03-55955d5bc9-nfrwx\" (UID: \"96229665-4486-4750-becf-d30f28c47dd5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-622e03-55955d5bc9-nfrwx" Oct 31 12:52:39.522189 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.522139 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmtbz\" (UniqueName: \"kubernetes.io/projected/96229665-4486-4750-becf-d30f28c47dd5-kube-api-access-dmtbz\") pod \"payload-pre-processing-e2e-worker-w1-622e03-55955d5bc9-nfrwx\" (UID: \"96229665-4486-4750-becf-d30f28c47dd5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-622e03-55955d5bc9-nfrwx" Oct 31 12:52:39.522189 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.522170 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb6f239c-4a3d-4f98-9b5e-4d775a4c561c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-622e03-7574857797-f4fdg\" (UID: \"fb6f239c-4a3d-4f98-9b5e-4d775a4c561c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-622e03-7574857797-f4fdg" Oct 31 12:52:39.623163 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.623071 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xktpv\" (UniqueName: \"kubernetes.io/projected/fb6f239c-4a3d-4f98-9b5e-4d775a4c561c-kube-api-access-xktpv\") pod \"payload-processing-e2e-worker-w1-622e03-7574857797-f4fdg\" (UID: \"fb6f239c-4a3d-4f98-9b5e-4d775a4c561c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-622e03-7574857797-f4fdg" Oct 31 12:52:39.623163 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.623154 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96229665-4486-4750-becf-d30f28c47dd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-622e03-55955d5bc9-nfrwx\" (UID: \"96229665-4486-4750-becf-d30f28c47dd5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-622e03-55955d5bc9-nfrwx" Oct 31 12:52:39.623403 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.623189 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmtbz\" (UniqueName: \"kubernetes.io/projected/96229665-4486-4750-becf-d30f28c47dd5-kube-api-access-dmtbz\") pod \"payload-pre-processing-e2e-worker-w1-622e03-55955d5bc9-nfrwx\" (UID: \"96229665-4486-4750-becf-d30f28c47dd5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-622e03-55955d5bc9-nfrwx" Oct 31 12:52:39.623403 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.623207 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb6f239c-4a3d-4f98-9b5e-4d775a4c561c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-622e03-7574857797-f4fdg\" (UID: \"fb6f239c-4a3d-4f98-9b5e-4d775a4c561c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-622e03-7574857797-f4fdg" Oct 31 12:52:39.623858 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.623839 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb6f239c-4a3d-4f98-9b5e-4d775a4c561c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-622e03-7574857797-f4fdg\" (UID: \"fb6f239c-4a3d-4f98-9b5e-4d775a4c561c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-622e03-7574857797-f4fdg" Oct 31 12:52:39.623897 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.623843 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96229665-4486-4750-becf-d30f28c47dd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-622e03-55955d5bc9-nfrwx\" (UID: \"96229665-4486-4750-becf-d30f28c47dd5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-622e03-55955d5bc9-nfrwx" Oct 31 12:52:39.639409 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.639376 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xktpv\" (UniqueName: \"kubernetes.io/projected/fb6f239c-4a3d-4f98-9b5e-4d775a4c561c-kube-api-access-xktpv\") pod \"payload-processing-e2e-worker-w1-622e03-7574857797-f4fdg\" (UID: \"fb6f239c-4a3d-4f98-9b5e-4d775a4c561c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-622e03-7574857797-f4fdg" Oct 31 12:52:39.640638 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.640613 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmtbz\" (UniqueName: \"kubernetes.io/projected/96229665-4486-4750-becf-d30f28c47dd5-kube-api-access-dmtbz\") pod \"payload-pre-processing-e2e-worker-w1-622e03-55955d5bc9-nfrwx\" (UID: \"96229665-4486-4750-becf-d30f28c47dd5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-622e03-55955d5bc9-nfrwx" Oct 31 12:52:39.667699 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.667150 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3bee1b8-5152-4805-a59d-7d05594142de" path="/var/lib/kubelet/pods/f3bee1b8-5152-4805-a59d-7d05594142de/volumes" Oct 31 12:52:39.671245 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.671109 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:39.689961 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.689939 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-622e03-55955d5bc9-nfrwx" Oct 31 12:52:39.722787 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.722754 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-622e03-7574857797-f4fdg" Oct 31 12:52:39.784504 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.784478 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ae9e7-57b99fbcdc-rqs6p" Oct 31 12:52:39.784635 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:39.784564 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ae9e7-65f7d7db79-bsvx2" Oct 31 12:52:40.020414 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:40.020386 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-622e03-55955d5bc9-nfrwx"] Oct 31 12:52:40.021679 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:52:40.021653 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96229665_4486_4750_becf_d30f28c47dd5.slice/crio-1213b6c878fa186eb454338ff986e19fec610adccda6160969b5cb1041d3d584 WatchSource:0}: Error finding container 1213b6c878fa186eb454338ff986e19fec610adccda6160969b5cb1041d3d584: Status 404 returned error can't find the container with id 1213b6c878fa186eb454338ff986e19fec610adccda6160969b5cb1041d3d584 Oct 31 12:52:40.041475 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:40.041402 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:40.049100 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:40.049078 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-622e03-7574857797-f4fdg"] Oct 31 12:52:40.051219 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:52:40.051189 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb6f239c_4a3d_4f98_9b5e_4d775a4c561c.slice/crio-316527ca7f4739546addbf4f9485ae462f44537b9433ee496261650d98a8cd03 WatchSource:0}: Error finding container 316527ca7f4739546addbf4f9485ae462f44537b9433ee496261650d98a8cd03: Status 404 returned error can't find the container with id 316527ca7f4739546addbf4f9485ae462f44537b9433ee496261650d98a8cd03 Oct 31 12:52:40.375455 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:40.375381 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8"] Oct 31 12:52:40.379389 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:40.379367 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8" Oct 31 12:52:40.387149 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:40.387046 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-3dd1e5-serving-cert\"" Oct 31 12:52:40.396868 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:40.396838 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8"] Oct 31 12:52:40.432864 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:40.432825 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4brs\" (UniqueName: \"kubernetes.io/projected/4e342df0-b108-4369-9c0a-0a2bf7b850bf-kube-api-access-c4brs\") pod \"maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8\" (UID: \"4e342df0-b108-4369-9c0a-0a2bf7b850bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8" Oct 31 12:52:40.433076 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:40.432932 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e342df0-b108-4369-9c0a-0a2bf7b850bf-maas-api-tls\") pod \"maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8\" (UID: \"4e342df0-b108-4369-9c0a-0a2bf7b850bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8" Oct 31 12:52:40.433076 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:40.432967 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e342df0-b108-4369-9c0a-0a2bf7b850bf-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8\" (UID: \"4e342df0-b108-4369-9c0a-0a2bf7b850bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8" Oct 31 12:52:40.433076 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:40.433032 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e342df0-b108-4369-9c0a-0a2bf7b850bf-metrics-certs\") pod \"maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8\" (UID: \"4e342df0-b108-4369-9c0a-0a2bf7b850bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8" Oct 31 12:52:40.534785 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:40.534739 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4brs\" (UniqueName: \"kubernetes.io/projected/4e342df0-b108-4369-9c0a-0a2bf7b850bf-kube-api-access-c4brs\") pod \"maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8\" (UID: \"4e342df0-b108-4369-9c0a-0a2bf7b850bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8" Oct 31 12:52:40.534946 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:40.534833 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e342df0-b108-4369-9c0a-0a2bf7b850bf-maas-api-tls\") pod \"maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8\" (UID: \"4e342df0-b108-4369-9c0a-0a2bf7b850bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8" Oct 31 12:52:40.534946 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:40.534874 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e342df0-b108-4369-9c0a-0a2bf7b850bf-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8\" (UID: \"4e342df0-b108-4369-9c0a-0a2bf7b850bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8" Oct 31 12:52:40.534946 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:40.534920 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e342df0-b108-4369-9c0a-0a2bf7b850bf-metrics-certs\") pod \"maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8\" (UID: \"4e342df0-b108-4369-9c0a-0a2bf7b850bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8" Oct 31 12:52:40.535121 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:52:40.534975 6841 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-3dd1e5-serving-cert: secret "maas-api-e2e-worker-w2-3dd1e5-serving-cert" not found Oct 31 12:52:40.535121 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:52:40.535050 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e342df0-b108-4369-9c0a-0a2bf7b850bf-maas-api-tls podName:4e342df0-b108-4369-9c0a-0a2bf7b850bf nodeName:}" failed. No retries permitted until 2026-10-31 12:52:41.035033526 +0000 UTC m=+1207.518156404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4e342df0-b108-4369-9c0a-0a2bf7b850bf-maas-api-tls") pod "maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8" (UID: "4e342df0-b108-4369-9c0a-0a2bf7b850bf") : secret "maas-api-e2e-worker-w2-3dd1e5-serving-cert" not found Oct 31 12:52:40.535605 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:40.535582 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e342df0-b108-4369-9c0a-0a2bf7b850bf-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8\" (UID: \"4e342df0-b108-4369-9c0a-0a2bf7b850bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8" Oct 31 12:52:40.537594 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:40.537572 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e342df0-b108-4369-9c0a-0a2bf7b850bf-metrics-certs\") pod \"maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8\" (UID: \"4e342df0-b108-4369-9c0a-0a2bf7b850bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8" Oct 31 12:52:40.550466 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:40.550430 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4brs\" (UniqueName: \"kubernetes.io/projected/4e342df0-b108-4369-9c0a-0a2bf7b850bf-kube-api-access-c4brs\") pod \"maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8\" (UID: \"4e342df0-b108-4369-9c0a-0a2bf7b850bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8" Oct 31 12:52:40.552767 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:40.552726 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:40.788661 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:40.788620 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:40.915409 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:40.915372 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-622e03-55955d5bc9-nfrwx" event={"ID":"96229665-4486-4750-becf-d30f28c47dd5","Type":"ContainerStarted","Data":"d4baf7017ccf84c7ec94e5ee6a988983ff78fc8dab38fc6ef15de6fde9b6d2bb"} Oct 31 12:52:40.915590 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:40.915441 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-622e03-55955d5bc9-nfrwx" Oct 31 12:52:40.915590 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:40.915468 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-622e03-55955d5bc9-nfrwx" event={"ID":"96229665-4486-4750-becf-d30f28c47dd5","Type":"ContainerStarted","Data":"1213b6c878fa186eb454338ff986e19fec610adccda6160969b5cb1041d3d584"} Oct 31 12:52:40.917571 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:40.917544 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-622e03-7574857797-f4fdg" event={"ID":"fb6f239c-4a3d-4f98-9b5e-4d775a4c561c","Type":"ContainerStarted","Data":"18de47f6118d9faf6e6dfe92a33834523ed70682bedcb1c91ddb46592c3e11ef"} Oct 31 12:52:40.917716 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:40.917584 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-622e03-7574857797-f4fdg" event={"ID":"fb6f239c-4a3d-4f98-9b5e-4d775a4c561c","Type":"ContainerStarted","Data":"316527ca7f4739546addbf4f9485ae462f44537b9433ee496261650d98a8cd03"} Oct 31 12:52:40.917716 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:40.917671 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-622e03-7574857797-f4fdg" Oct 31 12:52:40.938279 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:40.938228 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-622e03-55955d5bc9-nfrwx" podStartSLOduration=1.938213951 podStartE2EDuration="1.938213951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:52:40.937378652 +0000 UTC m=+1207.420501554" watchObservedRunningTime="2026-10-31 12:52:40.938213951 +0000 UTC m=+1207.421336852" Oct 31 12:52:40.959609 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:40.959554 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-622e03-7574857797-f4fdg" podStartSLOduration=1.959535013 podStartE2EDuration="1.959535013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:52:40.95823846 +0000 UTC m=+1207.441361363" watchObservedRunningTime="2026-10-31 12:52:40.959535013 +0000 UTC m=+1207.442657918" Oct 31 12:52:41.007239 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:41.007146 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:41.043057 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:41.042946 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e342df0-b108-4369-9c0a-0a2bf7b850bf-maas-api-tls\") pod \"maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8\" (UID: \"4e342df0-b108-4369-9c0a-0a2bf7b850bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8" Oct 31 12:52:41.045624 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:41.045594 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e342df0-b108-4369-9c0a-0a2bf7b850bf-maas-api-tls\") pod \"maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8\" (UID: \"4e342df0-b108-4369-9c0a-0a2bf7b850bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8" Oct 31 12:52:41.297026 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:41.296921 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8" Oct 31 12:52:41.466508 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:41.465599 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:41.605821 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:41.605790 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8"] Oct 31 12:52:41.607572 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:52:41.607544 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e342df0_b108_4369_9c0a_0a2bf7b850bf.slice/crio-610b81ae9ebcdddaff6d6f075ceacc497f0b221509adf87eb6986a82dcb99b3e WatchSource:0}: Error finding container 610b81ae9ebcdddaff6d6f075ceacc497f0b221509adf87eb6986a82dcb99b3e: Status 404 returned error can't find the container with id 610b81ae9ebcdddaff6d6f075ceacc497f0b221509adf87eb6986a82dcb99b3e Oct 31 12:52:41.680349 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:41.680316 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:41.924708 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:41.924617 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8" event={"ID":"4e342df0-b108-4369-9c0a-0a2bf7b850bf","Type":"ContainerStarted","Data":"610b81ae9ebcdddaff6d6f075ceacc497f0b221509adf87eb6986a82dcb99b3e"} Oct 31 12:52:42.259254 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:42.259216 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm"] Oct 31 12:52:42.265220 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:42.265196 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm" Oct 31 12:52:42.275177 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:42.275151 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-622e03-serving-cert\"" Oct 31 12:52:42.279939 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:42.279907 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm"] Oct 31 12:52:42.370255 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:42.370219 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85eedb21-9713-44f5-aaa7-6e8906b2df27-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm\" (UID: \"85eedb21-9713-44f5-aaa7-6e8906b2df27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm" Oct 31 12:52:42.370404 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:42.370285 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85eedb21-9713-44f5-aaa7-6e8906b2df27-maas-api-tls\") pod \"maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm\" (UID: \"85eedb21-9713-44f5-aaa7-6e8906b2df27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm" Oct 31 12:52:42.370551 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:42.370520 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85eedb21-9713-44f5-aaa7-6e8906b2df27-metrics-certs\") pod \"maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm\" (UID: \"85eedb21-9713-44f5-aaa7-6e8906b2df27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm" Oct 31 12:52:42.370604 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:42.370586 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t7sc\" (UniqueName: \"kubernetes.io/projected/85eedb21-9713-44f5-aaa7-6e8906b2df27-kube-api-access-2t7sc\") pod \"maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm\" (UID: \"85eedb21-9713-44f5-aaa7-6e8906b2df27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm" Oct 31 12:52:42.410689 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:42.410647 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:42.472089 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:42.472035 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85eedb21-9713-44f5-aaa7-6e8906b2df27-maas-api-tls\") pod \"maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm\" (UID: \"85eedb21-9713-44f5-aaa7-6e8906b2df27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm" Oct 31 12:52:42.472383 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:42.472355 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85eedb21-9713-44f5-aaa7-6e8906b2df27-metrics-certs\") pod \"maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm\" (UID: \"85eedb21-9713-44f5-aaa7-6e8906b2df27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm" Oct 31 12:52:42.472535 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:42.472405 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t7sc\" (UniqueName: \"kubernetes.io/projected/85eedb21-9713-44f5-aaa7-6e8906b2df27-kube-api-access-2t7sc\") pod \"maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm\" (UID: \"85eedb21-9713-44f5-aaa7-6e8906b2df27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm" Oct 31 12:52:42.472606 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:42.472538 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85eedb21-9713-44f5-aaa7-6e8906b2df27-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm\" (UID: \"85eedb21-9713-44f5-aaa7-6e8906b2df27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm" Oct 31 12:52:42.473354 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:42.473327 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85eedb21-9713-44f5-aaa7-6e8906b2df27-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm\" (UID: \"85eedb21-9713-44f5-aaa7-6e8906b2df27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm" Oct 31 12:52:42.475122 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:42.475093 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85eedb21-9713-44f5-aaa7-6e8906b2df27-maas-api-tls\") pod \"maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm\" (UID: \"85eedb21-9713-44f5-aaa7-6e8906b2df27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm" Oct 31 12:52:42.475307 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:42.475290 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85eedb21-9713-44f5-aaa7-6e8906b2df27-metrics-certs\") pod \"maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm\" (UID: \"85eedb21-9713-44f5-aaa7-6e8906b2df27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm" Oct 31 12:52:42.484807 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:42.484783 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t7sc\" (UniqueName: \"kubernetes.io/projected/85eedb21-9713-44f5-aaa7-6e8906b2df27-kube-api-access-2t7sc\") pod \"maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm\" (UID: \"85eedb21-9713-44f5-aaa7-6e8906b2df27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm" Oct 31 12:52:42.591475 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:42.591396 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm" Oct 31 12:52:42.669151 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:42.669115 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:42.873997 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:42.873957 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm"] Oct 31 12:52:42.875959 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:52:42.875929 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85eedb21_9713_44f5_aaa7_6e8906b2df27.slice/crio-a33fc55606119a78fcb5f0a606b3a58ed5c2275ef1496ff435203240a67ac3c1 WatchSource:0}: Error finding container a33fc55606119a78fcb5f0a606b3a58ed5c2275ef1496ff435203240a67ac3c1: Status 404 returned error can't find the container with id a33fc55606119a78fcb5f0a606b3a58ed5c2275ef1496ff435203240a67ac3c1 Oct 31 12:52:42.938277 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:42.938235 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8" event={"ID":"4e342df0-b108-4369-9c0a-0a2bf7b850bf","Type":"ContainerStarted","Data":"2cdde0a7b67dcebd70babffc79ecbb4e02a68fb2e5db44ca56500acc2352809e"} Oct 31 12:52:42.938474 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:42.938307 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8" Oct 31 12:52:42.939797 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:42.939760 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm" event={"ID":"85eedb21-9713-44f5-aaa7-6e8906b2df27","Type":"ContainerStarted","Data":"a33fc55606119a78fcb5f0a606b3a58ed5c2275ef1496ff435203240a67ac3c1"} Oct 31 12:52:42.962666 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:42.962624 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8" podStartSLOduration=2.549891581 podStartE2EDuration="2.962610435s" totalImagesPullingTime="412.718854ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:52:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:52:42.960619079 +0000 UTC m=+1209.443741981" watchObservedRunningTime="2026-10-31 12:52:42.962610435 +0000 UTC m=+1209.445733336" Oct 31 12:52:43.206053 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:43.206021 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:43.901623 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:43.901547 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-107955-fc749446-559d9" Oct 31 12:52:43.947161 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:43.947126 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm" event={"ID":"85eedb21-9713-44f5-aaa7-6e8906b2df27","Type":"ContainerStarted","Data":"26f5db959b416458d2a5907d9d29f851da68149a6c0465ca586f56fde54c97ad"} Oct 31 12:52:43.947341 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:43.947262 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm" Oct 31 12:52:43.951269 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:43.951235 6841 generic.go:356] "Generic (PLEG): container finished" podID="2975edb7-e9cc-4793-acac-baa30b53613c" containerID="0541c2e1998aca3a1d4d3ca605532a1e2e984bcfa0bd581f092d11f2fe0e4c3b" exitCode=0 Oct 31 12:52:43.952012 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:43.951970 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" event={"ID":"2975edb7-e9cc-4793-acac-baa30b53613c","Type":"ContainerDied","Data":"0541c2e1998aca3a1d4d3ca605532a1e2e984bcfa0bd581f092d11f2fe0e4c3b"} Oct 31 12:52:43.952359 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:43.952341 6841 scope.go:122] "RemoveContainer" containerID="0541c2e1998aca3a1d4d3ca605532a1e2e984bcfa0bd581f092d11f2fe0e4c3b" Oct 31 12:52:43.961408 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:43.961376 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:52:43.961532 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:43.961459 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:52:43.961532 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:43.961507 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:52:44.003726 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:44.003678 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm" podStartSLOduration=1.6215325900000002 podStartE2EDuration="2.003664012s" totalImagesPullingTime="382.131422ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:52:44.003317414 +0000 UTC m=+1210.486440312" watchObservedRunningTime="2026-10-31 12:52:44.003664012 +0000 UTC m=+1210.486786912" Oct 31 12:52:44.122631 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:44.122601 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:44.984882 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:44.980479 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" event={"ID":"2975edb7-e9cc-4793-acac-baa30b53613c","Type":"ContainerStarted","Data":"46cf4036f285105781d6f462426889bdd0ea24f5562afb22cbf3cf61a9276f95"} Oct 31 12:52:45.012916 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:45.012881 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:45.013360 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:45.013331 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 31 12:52:45.013473 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:45.013378 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" podUID="2975edb7-e9cc-4793-acac-baa30b53613c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 31 12:52:45.031385 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:45.031356 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:45.558008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:45.553532 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:45.875468 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:45.875389 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:46.013537 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:46.013498 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 31 12:52:46.013902 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:46.013550 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" podUID="2975edb7-e9cc-4793-acac-baa30b53613c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 31 12:52:46.636534 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:46.636497 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:46.838434 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:46.838406 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-107955-86f6b68d45-ztxhj" Oct 31 12:52:47.013946 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:47.013911 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 31 12:52:47.014411 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:47.013963 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" podUID="2975edb7-e9cc-4793-acac-baa30b53613c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 31 12:52:47.483746 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:47.483714 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:48.013727 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:48.013694 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 31 12:52:48.013893 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:48.013746 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" podUID="2975edb7-e9cc-4793-acac-baa30b53613c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 31 12:52:48.412075 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:48.411964 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:48.620037 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:48.620004 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:48.970791 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:48.970765 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8" Oct 31 12:52:49.013391 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:49.013360 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:49.014008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:49.013468 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 31 12:52:49.014136 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:49.014021 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" podUID="2975edb7-e9cc-4793-acac-baa30b53613c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 31 12:52:49.326492 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:49.326376 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:49.583672 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:49.583594 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:49.987517 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:49.987488 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm" Oct 31 12:52:50.013276 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:50.013243 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 31 12:52:50.013942 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:50.013299 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" podUID="2975edb7-e9cc-4793-acac-baa30b53613c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 31 12:52:50.043442 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:52:50.043403 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:52:50.384594 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:50.384559 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:51.018007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:51.017967 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:51.018395 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:51.018272 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:51.019087 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:51.019062 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 12:52:51.350802 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:51.350712 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:51.928134 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:51.928107 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-622e03-55955d5bc9-nfrwx" Oct 31 12:52:51.928305 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:51.928177 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-622e03-7574857797-f4fdg" Oct 31 12:52:52.296109 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:52.296071 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:53.135214 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:53.135181 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:54.121319 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:54.121285 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:54.955556 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:54.955519 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:56.001085 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:56.001041 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:56.214056 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:56.214017 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:56.708151 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:56.708113 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:56.928213 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:56.928171 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:57.134569 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:57.134479 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:57.933658 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:57.933617 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:58.138377 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:58.138341 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:58.534426 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:58.534373 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:58.747383 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:58.747340 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:52:58.965376 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:52:58.965325 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:00.142122 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:00.142085 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:01.030929 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:01.030895 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:02.019856 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:02.019819 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:02.834210 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:02.834173 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:03.674818 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:03.674772 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:03.889865 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:03.889832 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:04.539813 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:04.539775 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:05.309855 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:05.309822 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:05.753542 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:05.753507 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:06.130104 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:06.130019 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:07.209755 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:07.209715 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:08.006136 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:08.006092 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:08.705048 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:08.705016 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:09.509561 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:09.509524 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:10.396340 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:10.396305 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:10.651329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:10.651240 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:11.132180 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:11.132148 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:11.988548 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:11.988515 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:12.670469 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:12.669447 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:12.874538 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:12.874504 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:13.510925 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:13.510893 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:14.303283 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:14.303247 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:15.010673 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:15.010636 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:15.774734 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:15.774701 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:16.447717 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:16.447683 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:17.142481 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:17.142448 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:17.826106 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:17.826074 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:18.509919 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:18.509880 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:19.083806 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:19.083768 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:19.288422 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:19.288391 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:19.936090 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:19.936044 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:20.034893 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:53:20.034858 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:53:20.830337 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:20.830298 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:21.592424 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:21.592390 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:22.428477 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:22.428441 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:23.231521 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:23.231484 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:23.991294 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:23.991264 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:24.825088 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:24.825051 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:25.702575 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:25.702539 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:26.047676 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:26.047549 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:26.509284 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:26.509250 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:27.394492 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:27.394454 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:28.240201 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:28.240161 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:29.016847 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:29.016809 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:29.973620 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:29.973580 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:30.837733 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:30.837691 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:31.542649 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:31.542613 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:31.839817 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:31.839735 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:32.634689 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:32.634647 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:33.544336 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:33.544301 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:34.447910 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:34.447874 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:35.172214 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:35.172176 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:35.929954 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:35.929901 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:36.158933 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:36.158897 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:36.926626 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:36.926594 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:37.703367 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:37.703326 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:38.550807 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:38.550766 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:39.318227 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:39.318178 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:40.303910 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:40.303868 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:41.239239 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:41.239204 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:42.054468 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:42.054432 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:42.837116 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:42.837076 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:43.520852 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:43.520818 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn"] Oct 31 12:53:43.526255 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:43.526229 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" Oct 31 12:53:43.531183 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:43.531155 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-47bcd9-openshift-default-dockercfg-dwvfp\"" Oct 31 12:53:43.542382 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:43.542358 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn"] Oct 31 12:53:43.634470 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:43.634434 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:43.639927 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:43.639899 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3897f656-b3a9-4d66-b894-3663c88c253d-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn\" (UID: \"3897f656-b3a9-4d66-b894-3663c88c253d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" Oct 31 12:53:43.640084 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:43.639949 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3897f656-b3a9-4d66-b894-3663c88c253d-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn\" (UID: \"3897f656-b3a9-4d66-b894-3663c88c253d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" Oct 31 12:53:43.640084 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:43.639976 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3897f656-b3a9-4d66-b894-3663c88c253d-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn\" (UID: \"3897f656-b3a9-4d66-b894-3663c88c253d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" Oct 31 12:53:43.640162 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:43.640129 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3897f656-b3a9-4d66-b894-3663c88c253d-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn\" (UID: \"3897f656-b3a9-4d66-b894-3663c88c253d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" Oct 31 12:53:43.640211 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:43.640174 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3897f656-b3a9-4d66-b894-3663c88c253d-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn\" (UID: \"3897f656-b3a9-4d66-b894-3663c88c253d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" Oct 31 12:53:43.640248 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:43.640231 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3897f656-b3a9-4d66-b894-3663c88c253d-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn\" (UID: \"3897f656-b3a9-4d66-b894-3663c88c253d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" Oct 31 12:53:43.640296 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:43.640263 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t7f5\" (UniqueName: \"kubernetes.io/projected/3897f656-b3a9-4d66-b894-3663c88c253d-kube-api-access-8t7f5\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn\" (UID: \"3897f656-b3a9-4d66-b894-3663c88c253d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" Oct 31 12:53:43.640380 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:43.640361 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3897f656-b3a9-4d66-b894-3663c88c253d-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn\" (UID: \"3897f656-b3a9-4d66-b894-3663c88c253d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" Oct 31 12:53:43.640446 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:43.640429 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3897f656-b3a9-4d66-b894-3663c88c253d-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn\" (UID: \"3897f656-b3a9-4d66-b894-3663c88c253d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" Oct 31 12:53:43.741708 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:43.741666 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3897f656-b3a9-4d66-b894-3663c88c253d-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn\" (UID: \"3897f656-b3a9-4d66-b894-3663c88c253d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" Oct 31 12:53:43.741708 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:43.741714 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3897f656-b3a9-4d66-b894-3663c88c253d-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn\" (UID: \"3897f656-b3a9-4d66-b894-3663c88c253d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" Oct 31 12:53:43.741976 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:43.741747 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3897f656-b3a9-4d66-b894-3663c88c253d-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn\" (UID: \"3897f656-b3a9-4d66-b894-3663c88c253d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" Oct 31 12:53:43.741976 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:43.741787 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t7f5\" (UniqueName: \"kubernetes.io/projected/3897f656-b3a9-4d66-b894-3663c88c253d-kube-api-access-8t7f5\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn\" (UID: \"3897f656-b3a9-4d66-b894-3663c88c253d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" Oct 31 12:53:43.741976 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:43.741838 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3897f656-b3a9-4d66-b894-3663c88c253d-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn\" (UID: \"3897f656-b3a9-4d66-b894-3663c88c253d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" Oct 31 12:53:43.741976 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:43.741886 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3897f656-b3a9-4d66-b894-3663c88c253d-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn\" (UID: \"3897f656-b3a9-4d66-b894-3663c88c253d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" Oct 31 12:53:43.741976 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:43.741921 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3897f656-b3a9-4d66-b894-3663c88c253d-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn\" (UID: \"3897f656-b3a9-4d66-b894-3663c88c253d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" Oct 31 12:53:43.741976 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:43.741972 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3897f656-b3a9-4d66-b894-3663c88c253d-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn\" (UID: \"3897f656-b3a9-4d66-b894-3663c88c253d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" Oct 31 12:53:43.742340 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:43.742048 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3897f656-b3a9-4d66-b894-3663c88c253d-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn\" (UID: \"3897f656-b3a9-4d66-b894-3663c88c253d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" Oct 31 12:53:43.742340 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:43.742185 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3897f656-b3a9-4d66-b894-3663c88c253d-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn\" (UID: \"3897f656-b3a9-4d66-b894-3663c88c253d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" Oct 31 12:53:43.742340 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:43.742228 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3897f656-b3a9-4d66-b894-3663c88c253d-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn\" (UID: \"3897f656-b3a9-4d66-b894-3663c88c253d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" Oct 31 12:53:43.742340 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:43.742313 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3897f656-b3a9-4d66-b894-3663c88c253d-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn\" (UID: \"3897f656-b3a9-4d66-b894-3663c88c253d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" Oct 31 12:53:43.742527 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:43.742364 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3897f656-b3a9-4d66-b894-3663c88c253d-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn\" (UID: \"3897f656-b3a9-4d66-b894-3663c88c253d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" Oct 31 12:53:43.742693 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:43.742670 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3897f656-b3a9-4d66-b894-3663c88c253d-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn\" (UID: \"3897f656-b3a9-4d66-b894-3663c88c253d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" Oct 31 12:53:43.744494 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:43.744469 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3897f656-b3a9-4d66-b894-3663c88c253d-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn\" (UID: \"3897f656-b3a9-4d66-b894-3663c88c253d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" Oct 31 12:53:43.744626 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:43.744609 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3897f656-b3a9-4d66-b894-3663c88c253d-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn\" (UID: \"3897f656-b3a9-4d66-b894-3663c88c253d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" Oct 31 12:53:43.754751 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:43.754722 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3897f656-b3a9-4d66-b894-3663c88c253d-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn\" (UID: \"3897f656-b3a9-4d66-b894-3663c88c253d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" Oct 31 12:53:43.754938 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:43.754914 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t7f5\" (UniqueName: \"kubernetes.io/projected/3897f656-b3a9-4d66-b894-3663c88c253d-kube-api-access-8t7f5\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn\" (UID: \"3897f656-b3a9-4d66-b894-3663c88c253d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" Oct 31 12:53:43.840107 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:43.840027 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" Oct 31 12:53:44.121860 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.121831 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn"] Oct 31 12:53:44.123525 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:53:44.123498 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3897f656_b3a9_4d66_b894_3663c88c253d.slice/crio-0e6d9cbc2805f8ca0135a0384dad8b4131e839d303eb762b815b45436127f060 WatchSource:0}: Error finding container 0e6d9cbc2805f8ca0135a0384dad8b4131e839d303eb762b815b45436127f060: Status 404 returned error can't find the container with id 0e6d9cbc2805f8ca0135a0384dad8b4131e839d303eb762b815b45436127f060 Oct 31 12:53:44.125852 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.125822 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:53:44.125932 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.125886 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:53:44.125932 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.125923 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:53:44.325516 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.325480 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:44.339039 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.339012 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" event={"ID":"3897f656-b3a9-4d66-b894-3663c88c253d","Type":"ContainerStarted","Data":"c7c000fcad49d07c31892d965142e5219ad5168a4261310fbad6a414f7e3ed80"} Oct 31 12:53:44.339195 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.339062 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" event={"ID":"3897f656-b3a9-4d66-b894-3663c88c253d","Type":"ContainerStarted","Data":"0e6d9cbc2805f8ca0135a0384dad8b4131e839d303eb762b815b45436127f060"} Oct 31 12:53:44.367051 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.366998 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" podStartSLOduration=1.366968736 podStartE2EDuration="1.366968736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:44.364525153 +0000 UTC m=+1270.847648054" watchObservedRunningTime="2026-10-31 12:53:44.366968736 +0000 UTC m=+1270.850091636" Oct 31 12:53:44.552332 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.552298 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8"] Oct 31 12:53:44.556270 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.556250 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" Oct 31 12:53:44.560618 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.560597 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-107955-openshift-default-dockercfg-qq5mx\"" Oct 31 12:53:44.572049 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.572024 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8"] Oct 31 12:53:44.652163 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.652129 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a38e68ad-a968-4534-8135-8bb0721098f4-istio-data\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8\" (UID: \"a38e68ad-a968-4534-8135-8bb0721098f4\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" Oct 31 12:53:44.652163 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.652172 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38e68ad-a968-4534-8135-8bb0721098f4-workload-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8\" (UID: \"a38e68ad-a968-4534-8135-8bb0721098f4\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" Oct 31 12:53:44.652397 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.652194 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38e68ad-a968-4534-8135-8bb0721098f4-credential-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8\" (UID: \"a38e68ad-a968-4534-8135-8bb0721098f4\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" Oct 31 12:53:44.652397 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.652272 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a38e68ad-a968-4534-8135-8bb0721098f4-istio-envoy\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8\" (UID: \"a38e68ad-a968-4534-8135-8bb0721098f4\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" Oct 31 12:53:44.652397 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.652367 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fdrm\" (UniqueName: \"kubernetes.io/projected/a38e68ad-a968-4534-8135-8bb0721098f4-kube-api-access-6fdrm\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8\" (UID: \"a38e68ad-a968-4534-8135-8bb0721098f4\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" Oct 31 12:53:44.652502 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.652424 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a38e68ad-a968-4534-8135-8bb0721098f4-istiod-ca-cert\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8\" (UID: \"a38e68ad-a968-4534-8135-8bb0721098f4\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" Oct 31 12:53:44.652502 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.652449 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a38e68ad-a968-4534-8135-8bb0721098f4-istio-token\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8\" (UID: \"a38e68ad-a968-4534-8135-8bb0721098f4\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" Oct 31 12:53:44.652637 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.652611 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a38e68ad-a968-4534-8135-8bb0721098f4-workload-certs\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8\" (UID: \"a38e68ad-a968-4534-8135-8bb0721098f4\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" Oct 31 12:53:44.652669 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.652652 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a38e68ad-a968-4534-8135-8bb0721098f4-istio-podinfo\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8\" (UID: \"a38e68ad-a968-4534-8135-8bb0721098f4\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" Oct 31 12:53:44.754633 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.754591 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fdrm\" (UniqueName: \"kubernetes.io/projected/a38e68ad-a968-4534-8135-8bb0721098f4-kube-api-access-6fdrm\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8\" (UID: \"a38e68ad-a968-4534-8135-8bb0721098f4\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" Oct 31 12:53:44.757660 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.757630 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a38e68ad-a968-4534-8135-8bb0721098f4-istiod-ca-cert\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8\" (UID: \"a38e68ad-a968-4534-8135-8bb0721098f4\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" Oct 31 12:53:44.758326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.758290 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a38e68ad-a968-4534-8135-8bb0721098f4-istio-token\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8\" (UID: \"a38e68ad-a968-4534-8135-8bb0721098f4\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" Oct 31 12:53:44.758508 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.758485 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a38e68ad-a968-4534-8135-8bb0721098f4-workload-certs\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8\" (UID: \"a38e68ad-a968-4534-8135-8bb0721098f4\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" Oct 31 12:53:44.758670 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.758650 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a38e68ad-a968-4534-8135-8bb0721098f4-istio-podinfo\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8\" (UID: \"a38e68ad-a968-4534-8135-8bb0721098f4\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" Oct 31 12:53:44.758752 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.758725 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a38e68ad-a968-4534-8135-8bb0721098f4-istiod-ca-cert\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8\" (UID: \"a38e68ad-a968-4534-8135-8bb0721098f4\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" Oct 31 12:53:44.758811 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.758796 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a38e68ad-a968-4534-8135-8bb0721098f4-istio-data\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8\" (UID: \"a38e68ad-a968-4534-8135-8bb0721098f4\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" Oct 31 12:53:44.758853 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.758831 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38e68ad-a968-4534-8135-8bb0721098f4-workload-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8\" (UID: \"a38e68ad-a968-4534-8135-8bb0721098f4\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" Oct 31 12:53:44.758906 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.758862 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38e68ad-a968-4534-8135-8bb0721098f4-credential-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8\" (UID: \"a38e68ad-a968-4534-8135-8bb0721098f4\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" Oct 31 12:53:44.758906 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.758883 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a38e68ad-a968-4534-8135-8bb0721098f4-workload-certs\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8\" (UID: \"a38e68ad-a968-4534-8135-8bb0721098f4\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" Oct 31 12:53:44.758906 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.758900 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a38e68ad-a968-4534-8135-8bb0721098f4-istio-envoy\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8\" (UID: \"a38e68ad-a968-4534-8135-8bb0721098f4\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" Oct 31 12:53:44.759180 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.759161 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a38e68ad-a968-4534-8135-8bb0721098f4-istio-data\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8\" (UID: \"a38e68ad-a968-4534-8135-8bb0721098f4\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" Oct 31 12:53:44.762260 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.762227 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38e68ad-a968-4534-8135-8bb0721098f4-workload-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8\" (UID: \"a38e68ad-a968-4534-8135-8bb0721098f4\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" Oct 31 12:53:44.762260 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.762244 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38e68ad-a968-4534-8135-8bb0721098f4-credential-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8\" (UID: \"a38e68ad-a968-4534-8135-8bb0721098f4\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" Oct 31 12:53:44.762765 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.762740 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a38e68ad-a968-4534-8135-8bb0721098f4-istio-podinfo\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8\" (UID: \"a38e68ad-a968-4534-8135-8bb0721098f4\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" Oct 31 12:53:44.764627 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.764601 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a38e68ad-a968-4534-8135-8bb0721098f4-istio-envoy\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8\" (UID: \"a38e68ad-a968-4534-8135-8bb0721098f4\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" Oct 31 12:53:44.771286 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.771229 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a38e68ad-a968-4534-8135-8bb0721098f4-istio-token\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8\" (UID: \"a38e68ad-a968-4534-8135-8bb0721098f4\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" Oct 31 12:53:44.774527 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.774449 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fdrm\" (UniqueName: \"kubernetes.io/projected/a38e68ad-a968-4534-8135-8bb0721098f4-kube-api-access-6fdrm\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8\" (UID: \"a38e68ad-a968-4534-8135-8bb0721098f4\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" Oct 31 12:53:44.841110 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.840994 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" Oct 31 12:53:44.878023 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:44.877959 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" Oct 31 12:53:45.123578 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:45.123489 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:45.169013 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:45.168618 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8"] Oct 31 12:53:45.172103 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:53:45.172071 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda38e68ad_a968_4534_8135_8bb0721098f4.slice/crio-8e9d11a40212592c448a37f3b95bc1e50f05376f37a88826fa3778c1b1e99d96 WatchSource:0}: Error finding container 8e9d11a40212592c448a37f3b95bc1e50f05376f37a88826fa3778c1b1e99d96: Status 404 returned error can't find the container with id 8e9d11a40212592c448a37f3b95bc1e50f05376f37a88826fa3778c1b1e99d96 Oct 31 12:53:45.175226 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:45.175191 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:53:45.175320 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:45.175278 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:53:45.175360 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:45.175326 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:53:45.372471 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:45.372428 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" event={"ID":"a38e68ad-a968-4534-8135-8bb0721098f4","Type":"ContainerStarted","Data":"6d1a10ae7b1289038f925c85c13bb20d1d5cce3f1a0b16174aa4190367a80a64"} Oct 31 12:53:45.372648 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:45.372489 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" event={"ID":"a38e68ad-a968-4534-8135-8bb0721098f4","Type":"ContainerStarted","Data":"8e9d11a40212592c448a37f3b95bc1e50f05376f37a88826fa3778c1b1e99d96"} Oct 31 12:53:45.398428 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:45.398303 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" podStartSLOduration=1.398285085 podStartE2EDuration="1.398285085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:53:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:45.396681052 +0000 UTC m=+1271.879803986" watchObservedRunningTime="2026-10-31 12:53:45.398285085 +0000 UTC m=+1271.881407986" Oct 31 12:53:45.855064 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:45.847496 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 31 12:53:45.855064 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:45.847575 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" podUID="3897f656-b3a9-4d66-b894-3663c88c253d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 31 12:53:45.885005 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:45.884080 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" Oct 31 12:53:46.064004 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:46.063735 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:46.840680 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:46.840653 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 31 12:53:46.840910 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:46.840888 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" podUID="3897f656-b3a9-4d66-b894-3663c88c253d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 31 12:53:46.890006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:46.885780 6841 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 12:53:46.890006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:46.885843 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" podUID="a38e68ad-a968-4534-8135-8bb0721098f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 12:53:47.296205 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:47.296033 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:47.840671 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:47.840637 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 31 12:53:47.840858 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:47.840690 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" podUID="3897f656-b3a9-4d66-b894-3663c88c253d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 31 12:53:47.881158 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:47.881120 6841 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 12:53:47.881435 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:47.881393 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" podUID="a38e68ad-a968-4534-8135-8bb0721098f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 12:53:48.199099 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:48.199061 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:48.840915 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:48.840885 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 31 12:53:48.849242 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:48.849206 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" podUID="3897f656-b3a9-4d66-b894-3663c88c253d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 31 12:53:48.880556 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:48.880500 6841 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 12:53:48.880880 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:48.880827 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" podUID="a38e68ad-a968-4534-8135-8bb0721098f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 12:53:48.911779 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:48.911741 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:49.615127 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:49.615091 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:49.840629 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:49.840587 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 31 12:53:49.840808 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:49.840649 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" podUID="3897f656-b3a9-4d66-b894-3663c88c253d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 31 12:53:49.890401 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:49.890298 6841 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 12:53:49.890682 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:49.890635 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" podUID="a38e68ad-a968-4534-8135-8bb0721098f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 12:53:50.035065 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:53:50.035032 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:53:50.312480 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:50.312413 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:50.840644 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:50.840613 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 31 12:53:50.841211 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:50.840663 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" podUID="3897f656-b3a9-4d66-b894-3663c88c253d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 31 12:53:50.878757 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:50.878706 6841 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 12:53:50.878970 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:50.878775 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" podUID="a38e68ad-a968-4534-8135-8bb0721098f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 12:53:51.015767 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:51.015729 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:51.704099 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:51.704066 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:51.845621 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:51.845583 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" Oct 31 12:53:51.846105 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:51.846017 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" Oct 31 12:53:51.846769 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:51.846750 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" Oct 31 12:53:52.406647 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:52.406604 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:52.663499 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:52.663405 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" Oct 31 12:53:52.664048 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:52.664019 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" Oct 31 12:53:52.664609 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:52.664588 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" Oct 31 12:53:53.109230 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:53.109190 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:54.019917 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:54.019875 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:54.714699 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:54.714660 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:55.532794 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:55.532749 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:56.521437 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:56.521396 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:57.218815 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:57.218777 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:58.186715 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:58.186648 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:58.931609 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:58.931576 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:53:59.871090 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:53:59.871050 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:00.517428 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:00.517395 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:01.493426 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:01.493380 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:02.297020 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:02.296974 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:03.217011 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:03.216942 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:04.105465 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:04.105423 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:04.838949 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:04.838911 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:05.712561 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:05.712523 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:06.626376 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:06.626341 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:07.311122 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:07.311087 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:08.027271 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:08.027203 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:08.731919 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:08.731882 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:09.560263 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:09.560213 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:10.304052 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:10.304016 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:11.010507 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:11.010472 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:11.715252 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:11.715219 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:12.463036 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:12.462947 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:12.664720 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:12.664684 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:12.968326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:12.968278 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:13.627258 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:13.627224 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:14.888355 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:14.888301 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:16.207627 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:16.207591 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:16.924494 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:16.924458 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:17.697445 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:17.697407 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:18.571364 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:18.571325 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:19.165218 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.165179 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k"] Oct 31 12:54:19.169485 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.169455 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" Oct 31 12:54:19.174969 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.174949 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-622e03-openshift-default-dockercfg-br2cc\"" Oct 31 12:54:19.183711 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.183686 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k"] Oct 31 12:54:19.194219 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.194196 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68406df2-8453-4762-9398-4019221157aa-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-k266k\" (UID: \"68406df2-8453-4762-9398-4019221157aa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" Oct 31 12:54:19.194375 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.194245 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdvrf\" (UniqueName: \"kubernetes.io/projected/68406df2-8453-4762-9398-4019221157aa-kube-api-access-zdvrf\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-k266k\" (UID: \"68406df2-8453-4762-9398-4019221157aa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" Oct 31 12:54:19.194375 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.194301 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68406df2-8453-4762-9398-4019221157aa-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-k266k\" (UID: \"68406df2-8453-4762-9398-4019221157aa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" Oct 31 12:54:19.194375 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.194338 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68406df2-8453-4762-9398-4019221157aa-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-k266k\" (UID: \"68406df2-8453-4762-9398-4019221157aa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" Oct 31 12:54:19.194567 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.194403 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68406df2-8453-4762-9398-4019221157aa-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-k266k\" (UID: \"68406df2-8453-4762-9398-4019221157aa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" Oct 31 12:54:19.194567 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.194466 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68406df2-8453-4762-9398-4019221157aa-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-k266k\" (UID: \"68406df2-8453-4762-9398-4019221157aa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" Oct 31 12:54:19.194567 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.194504 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68406df2-8453-4762-9398-4019221157aa-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-k266k\" (UID: \"68406df2-8453-4762-9398-4019221157aa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" Oct 31 12:54:19.194567 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.194546 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68406df2-8453-4762-9398-4019221157aa-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-k266k\" (UID: \"68406df2-8453-4762-9398-4019221157aa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" Oct 31 12:54:19.194713 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.194642 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68406df2-8453-4762-9398-4019221157aa-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-k266k\" (UID: \"68406df2-8453-4762-9398-4019221157aa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" Oct 31 12:54:19.295250 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.295215 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68406df2-8453-4762-9398-4019221157aa-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-k266k\" (UID: \"68406df2-8453-4762-9398-4019221157aa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" Oct 31 12:54:19.295421 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.295257 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68406df2-8453-4762-9398-4019221157aa-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-k266k\" (UID: \"68406df2-8453-4762-9398-4019221157aa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" Oct 31 12:54:19.295421 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.295282 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68406df2-8453-4762-9398-4019221157aa-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-k266k\" (UID: \"68406df2-8453-4762-9398-4019221157aa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" Oct 31 12:54:19.295421 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.295346 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68406df2-8453-4762-9398-4019221157aa-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-k266k\" (UID: \"68406df2-8453-4762-9398-4019221157aa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" Oct 31 12:54:19.295421 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.295385 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68406df2-8453-4762-9398-4019221157aa-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-k266k\" (UID: \"68406df2-8453-4762-9398-4019221157aa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" Oct 31 12:54:19.295421 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.295419 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdvrf\" (UniqueName: \"kubernetes.io/projected/68406df2-8453-4762-9398-4019221157aa-kube-api-access-zdvrf\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-k266k\" (UID: \"68406df2-8453-4762-9398-4019221157aa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" Oct 31 12:54:19.295662 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.295453 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68406df2-8453-4762-9398-4019221157aa-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-k266k\" (UID: \"68406df2-8453-4762-9398-4019221157aa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" Oct 31 12:54:19.295662 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.295488 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68406df2-8453-4762-9398-4019221157aa-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-k266k\" (UID: \"68406df2-8453-4762-9398-4019221157aa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" Oct 31 12:54:19.295662 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.295530 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68406df2-8453-4762-9398-4019221157aa-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-k266k\" (UID: \"68406df2-8453-4762-9398-4019221157aa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" Oct 31 12:54:19.296069 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.295921 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68406df2-8453-4762-9398-4019221157aa-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-k266k\" (UID: \"68406df2-8453-4762-9398-4019221157aa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" Oct 31 12:54:19.296069 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.296001 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68406df2-8453-4762-9398-4019221157aa-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-k266k\" (UID: \"68406df2-8453-4762-9398-4019221157aa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" Oct 31 12:54:19.296069 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.296060 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68406df2-8453-4762-9398-4019221157aa-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-k266k\" (UID: \"68406df2-8453-4762-9398-4019221157aa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" Oct 31 12:54:19.296298 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.296060 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68406df2-8453-4762-9398-4019221157aa-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-k266k\" (UID: \"68406df2-8453-4762-9398-4019221157aa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" Oct 31 12:54:19.296474 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.296447 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68406df2-8453-4762-9398-4019221157aa-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-k266k\" (UID: \"68406df2-8453-4762-9398-4019221157aa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" Oct 31 12:54:19.297750 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.297731 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68406df2-8453-4762-9398-4019221157aa-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-k266k\" (UID: \"68406df2-8453-4762-9398-4019221157aa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" Oct 31 12:54:19.297878 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.297862 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68406df2-8453-4762-9398-4019221157aa-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-k266k\" (UID: \"68406df2-8453-4762-9398-4019221157aa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" Oct 31 12:54:19.319099 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.319073 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68406df2-8453-4762-9398-4019221157aa-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-k266k\" (UID: \"68406df2-8453-4762-9398-4019221157aa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" Oct 31 12:54:19.319230 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.319108 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdvrf\" (UniqueName: \"kubernetes.io/projected/68406df2-8453-4762-9398-4019221157aa-kube-api-access-zdvrf\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-k266k\" (UID: \"68406df2-8453-4762-9398-4019221157aa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" Oct 31 12:54:19.431699 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.431622 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:19.486010 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.485954 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" Oct 31 12:54:19.769192 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.769161 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k"] Oct 31 12:54:19.770526 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:54:19.770499 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68406df2_8453_4762_9398_4019221157aa.slice/crio-69e12c96397c6077e2777c163195d92df1a8305d72a5e373f0f47f480d311ad4 WatchSource:0}: Error finding container 69e12c96397c6077e2777c163195d92df1a8305d72a5e373f0f47f480d311ad4: Status 404 returned error can't find the container with id 69e12c96397c6077e2777c163195d92df1a8305d72a5e373f0f47f480d311ad4 Oct 31 12:54:19.772884 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.772851 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:54:19.773003 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.772922 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:54:19.773003 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:19.772966 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:54:20.070055 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:54:20.069313 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:54:20.529029 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:20.528470 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:20.736006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:20.735281 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" event={"ID":"68406df2-8453-4762-9398-4019221157aa","Type":"ContainerStarted","Data":"0caecc427a63417b3ec0852e445fbd548b4baa6ff3b985a71d639814be844a17"} Oct 31 12:54:20.736006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:20.735350 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" event={"ID":"68406df2-8453-4762-9398-4019221157aa","Type":"ContainerStarted","Data":"69e12c96397c6077e2777c163195d92df1a8305d72a5e373f0f47f480d311ad4"} Oct 31 12:54:20.793062 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:20.786976 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" podStartSLOduration=1.7869537210000002 podStartE2EDuration="1.786953721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:54:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:54:20.768436267 +0000 UTC m=+1307.251559169" watchObservedRunningTime="2026-10-31 12:54:20.786953721 +0000 UTC m=+1307.270076623" Oct 31 12:54:21.427042 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:21.426804 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:21.494007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:21.490738 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" Oct 31 12:54:21.494007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:21.490870 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 12:54:21.494007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:21.490913 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" podUID="68406df2-8453-4762-9398-4019221157aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 12:54:22.487864 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:22.487389 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 12:54:22.487864 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:22.487452 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" podUID="68406df2-8453-4762-9398-4019221157aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 12:54:22.503118 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:22.503084 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:23.332857 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:23.332822 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:23.488497 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:23.488450 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 12:54:23.489152 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:23.489096 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" podUID="68406df2-8453-4762-9398-4019221157aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 12:54:24.073850 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:24.073806 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:24.487361 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:24.487324 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 12:54:24.487535 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:24.487383 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" podUID="68406df2-8453-4762-9398-4019221157aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 12:54:24.811590 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:24.811491 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:25.488236 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:25.487653 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 12:54:25.488236 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:25.487724 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" podUID="68406df2-8453-4762-9398-4019221157aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 12:54:25.517367 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:25.517332 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:26.220013 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:26.219959 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:26.486815 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:26.486736 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 12:54:26.486815 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:26.486798 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" podUID="68406df2-8453-4762-9398-4019221157aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 12:54:26.914398 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:26.914310 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:27.491613 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:27.491583 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" Oct 31 12:54:27.492112 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:27.491938 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" Oct 31 12:54:27.492722 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:27.492700 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" Oct 31 12:54:27.609599 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:27.609565 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:28.462825 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:28.462784 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:29.390928 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:29.390891 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:30.895487 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:30.895452 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:31.265113 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:31.265067 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-ghnhx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:54:31.265402 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:31.265370 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" podUID="66f79621-82da-431c-8c50-12eb405fc80e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:54:32.284118 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:32.284083 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:32.751614 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:32.751583 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm"] Oct 31 12:54:32.755594 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:32.755572 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" Oct 31 12:54:32.768175 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:32.768149 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-3dd1e5-openshift-default-dockercfg-ktcdl\"" Oct 31 12:54:32.769675 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:32.769649 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm"] Oct 31 12:54:32.794915 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:32.794875 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-istio-podinfo\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm\" (UID: \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" Oct 31 12:54:32.795080 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:32.794922 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-workload-certs\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm\" (UID: \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" Oct 31 12:54:32.795080 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:32.795062 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-istiod-ca-cert\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm\" (UID: \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" Oct 31 12:54:32.795279 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:32.795115 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-workload-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm\" (UID: \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" Oct 31 12:54:32.795279 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:32.795208 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-istio-data\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm\" (UID: \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" Oct 31 12:54:32.795279 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:32.795245 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-credential-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm\" (UID: \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" Oct 31 12:54:32.795441 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:32.795331 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgzb6\" (UniqueName: \"kubernetes.io/projected/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-kube-api-access-jgzb6\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm\" (UID: \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" Oct 31 12:54:32.795441 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:32.795377 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-istio-envoy\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm\" (UID: \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" Oct 31 12:54:32.795441 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:32.795400 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-istio-token\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm\" (UID: \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" Oct 31 12:54:32.896596 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:32.896564 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-workload-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm\" (UID: \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" Oct 31 12:54:32.896744 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:32.896615 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-istio-data\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm\" (UID: \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" Oct 31 12:54:32.896744 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:32.896726 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-credential-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm\" (UID: \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" Oct 31 12:54:32.896828 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:32.896810 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgzb6\" (UniqueName: \"kubernetes.io/projected/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-kube-api-access-jgzb6\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm\" (UID: \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" Oct 31 12:54:32.896872 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:32.896856 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-istio-envoy\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm\" (UID: \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" Oct 31 12:54:32.897032 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:32.897011 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-workload-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm\" (UID: \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" Oct 31 12:54:32.897092 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:32.897020 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-istio-token\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm\" (UID: \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" Oct 31 12:54:32.897092 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:32.897037 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-istio-data\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm\" (UID: \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" Oct 31 12:54:32.897200 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:32.897110 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-istio-podinfo\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm\" (UID: \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" Oct 31 12:54:32.897200 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:32.897113 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-credential-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm\" (UID: \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" Oct 31 12:54:32.897200 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:32.897130 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-workload-certs\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm\" (UID: \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" Oct 31 12:54:32.897354 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:32.897216 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-istiod-ca-cert\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm\" (UID: \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" Oct 31 12:54:32.897467 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:32.897434 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-workload-certs\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm\" (UID: \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" Oct 31 12:54:32.897815 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:32.897798 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-istiod-ca-cert\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm\" (UID: \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" Oct 31 12:54:32.899139 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:32.899117 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-istio-envoy\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm\" (UID: \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" Oct 31 12:54:32.899456 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:32.899440 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-istio-podinfo\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm\" (UID: \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" Oct 31 12:54:32.909662 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:32.909637 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgzb6\" (UniqueName: \"kubernetes.io/projected/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-kube-api-access-jgzb6\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm\" (UID: \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" Oct 31 12:54:32.909963 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:32.909943 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-istio-token\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm\" (UID: \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" Oct 31 12:54:33.072099 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:33.072009 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" Oct 31 12:54:33.148124 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:33.148092 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:33.466087 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:33.466039 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm"] Oct 31 12:54:33.468443 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:54:33.468413 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod164b7cbb_c80d_4709_b5d0_5bdba8070ce6.slice/crio-16cd9edd182bf7edd61880575bfca8e8c0df01f58cb843e4c8456328447dfcf2 WatchSource:0}: Error finding container 16cd9edd182bf7edd61880575bfca8e8c0df01f58cb843e4c8456328447dfcf2: Status 404 returned error can't find the container with id 16cd9edd182bf7edd61880575bfca8e8c0df01f58cb843e4c8456328447dfcf2 Oct 31 12:54:33.471617 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:33.471578 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:54:33.471719 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:33.471654 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:54:33.471719 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:33.471696 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:54:34.044547 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:34.044507 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" event={"ID":"164b7cbb-c80d-4709-b5d0-5bdba8070ce6","Type":"ContainerStarted","Data":"c9a43971a23e2288e79ffc3c02714ba9bc0cc61090695b688fd66c15be4a6c98"} Oct 31 12:54:34.044722 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:34.044560 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" event={"ID":"164b7cbb-c80d-4709-b5d0-5bdba8070ce6","Type":"ContainerStarted","Data":"16cd9edd182bf7edd61880575bfca8e8c0df01f58cb843e4c8456328447dfcf2"} Oct 31 12:54:34.070903 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:34.070856 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" podStartSLOduration=2.070842929 podStartE2EDuration="2.070842929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:54:34.07006831 +0000 UTC m=+1320.553191212" watchObservedRunningTime="2026-10-31 12:54:34.070842929 +0000 UTC m=+1320.553965829" Oct 31 12:54:34.073129 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:34.073110 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" Oct 31 12:54:34.074795 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:34.074772 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 12:54:34.074879 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:34.074818 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" podUID="164b7cbb-c80d-4709-b5d0-5bdba8070ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 12:54:34.290016 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:34.289277 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:34.517355 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:34.517314 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:35.073378 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:35.073343 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 12:54:35.073552 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:35.073391 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" podUID="164b7cbb-c80d-4709-b5d0-5bdba8070ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 12:54:35.164006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:35.160865 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:36.042009 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:36.029971 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:36.077006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:36.076393 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 12:54:36.077006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:36.076451 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" podUID="164b7cbb-c80d-4709-b5d0-5bdba8070ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 12:54:36.266039 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:36.264488 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:36.977965 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:36.977834 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:37.072583 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:37.072541 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 12:54:37.073114 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:37.072605 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" podUID="164b7cbb-c80d-4709-b5d0-5bdba8070ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 12:54:38.073348 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:38.073313 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 12:54:38.073751 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:38.073365 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" podUID="164b7cbb-c80d-4709-b5d0-5bdba8070ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 12:54:38.090915 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:38.090888 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:38.910826 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:38.910793 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:39.073191 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:39.073155 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 12:54:39.073364 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:39.073209 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" podUID="164b7cbb-c80d-4709-b5d0-5bdba8070ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 12:54:39.371101 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:39.371074 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:39.636031 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:39.635910 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:39.841043 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:39.840999 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:40.521647 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:40.521610 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:40.986010 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:40.985968 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" Oct 31 12:54:40.986268 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:40.986243 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" Oct 31 12:54:40.987257 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:40.987238 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" Oct 31 12:54:41.234192 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:41.234160 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:42.080255 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:42.080218 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:42.900519 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:42.900476 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:43.829669 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:43.829634 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:44.731213 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:44.731161 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:45.738615 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:45.738566 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:46.230289 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:46.230255 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-ghnhx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:54:46.230612 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:46.230565 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" podUID="66f79621-82da-431c-8c50-12eb405fc80e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:54:46.442702 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:46.442665 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:46.818020 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:46.814452 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:47.529858 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:47.529824 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:48.419791 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:48.419749 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:49.221893 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:49.221852 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:50.015331 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:50.015259 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:50.039518 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:54:50.039488 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:54:50.923131 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:50.923087 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:51.944112 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:51.944060 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:52.823464 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:52.823425 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:53.832561 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:53.832527 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:54.720107 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:54.720052 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:55.419850 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:55.419800 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:56.225908 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:56.225868 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:57.122063 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:57.122028 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:57.652431 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:57.652391 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:57.996607 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:57.996570 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:58.717697 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:58.717654 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:54:59.523823 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:54:59.523782 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:00.415457 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:00.415421 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:01.444526 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:01.444495 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-57fc876657-4b2dx"] Oct 31 12:55:01.449782 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:01.449758 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-57fc876657-4b2dx"] Oct 31 12:55:01.449926 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:01.449851 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57fc876657-4b2dx" Oct 31 12:55:01.453992 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:01.453950 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 31 12:55:01.460375 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:01.460354 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 31 12:55:01.460674 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:01.460639 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 31 12:55:01.460787 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:01.460678 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 31 12:55:01.460787 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:01.460705 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-vms8t\"" Oct 31 12:55:01.460787 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:01.460733 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-c59bbce24cdf\"" Oct 31 12:55:01.476313 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:01.476287 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:01.535576 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:01.535542 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c-routing-overlay-volume\") pod \"payload-processing-external-model-57fc876657-4b2dx\" (UID: \"77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c\") " pod="models-as-a-service/payload-processing-external-model-57fc876657-4b2dx" Oct 31 12:55:01.535777 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:01.535610 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c-provider-credentials\") pod \"payload-processing-external-model-57fc876657-4b2dx\" (UID: \"77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c\") " pod="models-as-a-service/payload-processing-external-model-57fc876657-4b2dx" Oct 31 12:55:01.535777 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:01.535709 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c-plugins-config-volume\") pod \"payload-processing-external-model-57fc876657-4b2dx\" (UID: \"77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c\") " pod="models-as-a-service/payload-processing-external-model-57fc876657-4b2dx" Oct 31 12:55:01.535906 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:01.535885 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh6fs\" (UniqueName: \"kubernetes.io/projected/77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c-kube-api-access-lh6fs\") pod \"payload-processing-external-model-57fc876657-4b2dx\" (UID: \"77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c\") " pod="models-as-a-service/payload-processing-external-model-57fc876657-4b2dx" Oct 31 12:55:01.636672 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:01.636638 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c-provider-credentials\") pod \"payload-processing-external-model-57fc876657-4b2dx\" (UID: \"77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c\") " pod="models-as-a-service/payload-processing-external-model-57fc876657-4b2dx" Oct 31 12:55:01.636672 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:01.636678 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c-plugins-config-volume\") pod \"payload-processing-external-model-57fc876657-4b2dx\" (UID: \"77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c\") " pod="models-as-a-service/payload-processing-external-model-57fc876657-4b2dx" Oct 31 12:55:01.636968 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:01.636759 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh6fs\" (UniqueName: \"kubernetes.io/projected/77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c-kube-api-access-lh6fs\") pod \"payload-processing-external-model-57fc876657-4b2dx\" (UID: \"77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c\") " pod="models-as-a-service/payload-processing-external-model-57fc876657-4b2dx" Oct 31 12:55:01.636968 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:01.636809 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c-routing-overlay-volume\") pod \"payload-processing-external-model-57fc876657-4b2dx\" (UID: \"77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c\") " pod="models-as-a-service/payload-processing-external-model-57fc876657-4b2dx" Oct 31 12:55:01.637794 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:01.637766 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c-plugins-config-volume\") pod \"payload-processing-external-model-57fc876657-4b2dx\" (UID: \"77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c\") " pod="models-as-a-service/payload-processing-external-model-57fc876657-4b2dx" Oct 31 12:55:01.637937 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:01.637918 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c-routing-overlay-volume\") pod \"payload-processing-external-model-57fc876657-4b2dx\" (UID: \"77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c\") " pod="models-as-a-service/payload-processing-external-model-57fc876657-4b2dx" Oct 31 12:55:01.639430 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:01.639407 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c-provider-credentials\") pod \"payload-processing-external-model-57fc876657-4b2dx\" (UID: \"77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c\") " pod="models-as-a-service/payload-processing-external-model-57fc876657-4b2dx" Oct 31 12:55:01.649392 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:01.649370 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh6fs\" (UniqueName: \"kubernetes.io/projected/77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c-kube-api-access-lh6fs\") pod \"payload-processing-external-model-57fc876657-4b2dx\" (UID: \"77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c\") " pod="models-as-a-service/payload-processing-external-model-57fc876657-4b2dx" Oct 31 12:55:01.767546 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:01.767466 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57fc876657-4b2dx" Oct 31 12:55:02.050014 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:02.044307 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-57fc876657-4b2dx"] Oct 31 12:55:02.054127 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:55:02.054096 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77fbaa7d_34e2_4d69_9fdd_b08fca36ad8c.slice/crio-f2b29809be15b4c526d1899218da8d5cfb54215fb1d79ea14e2e344648380524 WatchSource:0}: Error finding container f2b29809be15b4c526d1899218da8d5cfb54215fb1d79ea14e2e344648380524: Status 404 returned error can't find the container with id f2b29809be15b4c526d1899218da8d5cfb54215fb1d79ea14e2e344648380524 Oct 31 12:55:02.240480 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:02.240446 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:02.347142 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:02.347026 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57fc876657-4b2dx" event={"ID":"77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c","Type":"ContainerStarted","Data":"e10768efa72ff369d05959f3cf0fdc2d8d87ef92f9f6996214d75f78713ec756"} Oct 31 12:55:02.347471 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:02.347447 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57fc876657-4b2dx" event={"ID":"77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c","Type":"ContainerStarted","Data":"f2b29809be15b4c526d1899218da8d5cfb54215fb1d79ea14e2e344648380524"} Oct 31 12:55:02.348521 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:02.348484 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-57fc876657-4b2dx" Oct 31 12:55:02.412874 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:02.412817 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-57fc876657-4b2dx" podStartSLOduration=1.4127969949999999 podStartE2EDuration="1.412796995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:55:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:55:02.411113647 +0000 UTC m=+1348.894236547" watchObservedRunningTime="2026-10-31 12:55:02.412796995 +0000 UTC m=+1348.895919896" Oct 31 12:55:02.576003 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:02.572597 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:03.378012 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:03.377960 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:04.115596 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:04.115559 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:04.914621 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:04.914580 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:05.621219 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:05.621180 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:06.416853 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:06.416811 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:07.127407 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:07.127349 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:07.400053 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:07.398957 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:07.901116 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:07.901083 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:08.619365 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:08.619321 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:09.334271 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:09.334238 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:10.038864 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:10.038831 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:10.816652 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:10.816598 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:11.513893 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:11.513859 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:12.231302 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:12.231261 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:13.015707 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:13.015659 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:13.758007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:13.757494 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6"] Oct 31 12:55:13.764006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:13.763316 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" Oct 31 12:55:13.788440 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:13.763201 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6"] Oct 31 12:55:13.857579 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:13.857427 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6\" (UID: \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" Oct 31 12:55:13.857792 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:13.857591 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6\" (UID: \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" Oct 31 12:55:13.857792 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:13.857666 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6\" (UID: \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" Oct 31 12:55:13.857792 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:13.857712 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92tzv\" (UniqueName: \"kubernetes.io/projected/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-kube-api-access-92tzv\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6\" (UID: \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" Oct 31 12:55:13.857792 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:13.857745 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6\" (UID: \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" Oct 31 12:55:13.858178 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:13.857797 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6\" (UID: \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" Oct 31 12:55:13.859566 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:13.859353 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6\" (UID: \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" Oct 31 12:55:13.859566 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:13.859510 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6\" (UID: \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" Oct 31 12:55:13.859756 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:13.859589 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6\" (UID: \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" Oct 31 12:55:13.974009 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:13.972777 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6\" (UID: \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" Oct 31 12:55:13.974009 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:13.972865 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6\" (UID: \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" Oct 31 12:55:13.974009 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:13.972910 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92tzv\" (UniqueName: \"kubernetes.io/projected/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-kube-api-access-92tzv\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6\" (UID: \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" Oct 31 12:55:13.974009 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:13.972940 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6\" (UID: \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" Oct 31 12:55:13.974009 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:13.973004 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6\" (UID: \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" Oct 31 12:55:13.974009 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:13.973061 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6\" (UID: \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" Oct 31 12:55:13.974009 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:13.973137 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6\" (UID: \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" Oct 31 12:55:13.974009 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:13.973192 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6\" (UID: \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" Oct 31 12:55:13.974009 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:13.973230 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6\" (UID: \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" Oct 31 12:55:13.974009 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:13.973709 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6\" (UID: \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" Oct 31 12:55:13.974009 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:13.973999 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6\" (UID: \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" Oct 31 12:55:13.988021 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:13.985359 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6\" (UID: \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" Oct 31 12:55:13.988021 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:13.986396 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6\" (UID: \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" Oct 31 12:55:13.988021 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:13.986858 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6\" (UID: \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" Oct 31 12:55:14.006003 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:14.003412 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6\" (UID: \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" Oct 31 12:55:14.026426 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:14.021312 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6\" (UID: \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" Oct 31 12:55:14.026426 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:14.023183 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6\" (UID: \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" Oct 31 12:55:14.026426 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:14.025922 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:14.026426 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:14.026277 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92tzv\" (UniqueName: \"kubernetes.io/projected/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-kube-api-access-92tzv\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6\" (UID: \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" Oct 31 12:55:14.096470 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:14.096166 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" Oct 31 12:55:14.375381 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:14.375290 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-57fc876657-4b2dx" Oct 31 12:55:15.090415 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:15.090326 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:15.158032 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:15.157966 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6"] Oct 31 12:55:15.287418 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:15.286257 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:55:15.287418 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:15.286350 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:55:15.287418 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:15.286395 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:55:15.651899 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:15.632284 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" event={"ID":"3e30e333-c7fd-4aaa-9541-3a1ef29189a4","Type":"ContainerStarted","Data":"1158b55b93e76e20caa4fc7949166c62ef6013a1f74677505f9ceec92b76811e"} Oct 31 12:55:15.651899 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:15.632353 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" event={"ID":"3e30e333-c7fd-4aaa-9541-3a1ef29189a4","Type":"ContainerStarted","Data":"f8867c4653a4b8fd6a0f11ea5c7296f2c00416fce6d9d4626afb99742f0220c8"} Oct 31 12:55:15.710049 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:15.706698 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" podStartSLOduration=2.706678026 podStartE2EDuration="2.706678026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:55:15.698448959 +0000 UTC m=+1362.181571860" watchObservedRunningTime="2026-10-31 12:55:15.706678026 +0000 UTC m=+1362.189800927" Oct 31 12:55:16.112195 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:16.112142 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" Oct 31 12:55:16.117113 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:16.117081 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 12:55:16.117324 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:16.117297 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" podUID="3e30e333-c7fd-4aaa-9541-3a1ef29189a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 12:55:16.223127 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:16.223086 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-ghnhx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:55:16.223395 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:16.223365 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" podUID="66f79621-82da-431c-8c50-12eb405fc80e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:16.449831 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:16.449799 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:17.117437 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:17.117396 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 12:55:17.118449 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:17.118032 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" podUID="3e30e333-c7fd-4aaa-9541-3a1ef29189a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 12:55:17.358009 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:17.357955 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:18.098086 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:18.098044 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 12:55:18.098246 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:18.098100 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" podUID="3e30e333-c7fd-4aaa-9541-3a1ef29189a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 12:55:18.187019 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:18.185997 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:19.107008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:19.101474 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 12:55:19.107008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:19.101537 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" podUID="3e30e333-c7fd-4aaa-9541-3a1ef29189a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 12:55:19.135833 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:19.135797 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:19.923003 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:19.919852 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:20.034731 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:55:20.034697 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:55:20.097254 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:20.097221 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 12:55:20.097418 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:20.097269 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" podUID="3e30e333-c7fd-4aaa-9541-3a1ef29189a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 12:55:20.505005 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:20.501012 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:20.830945 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:20.830864 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:21.097238 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:21.097162 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 12:55:21.097238 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:21.097212 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" podUID="3e30e333-c7fd-4aaa-9541-3a1ef29189a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 12:55:21.606358 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:21.606257 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:22.110243 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:22.110205 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 12:55:22.110838 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:22.110800 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" podUID="3e30e333-c7fd-4aaa-9541-3a1ef29189a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 12:55:22.409690 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:22.409599 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:23.097738 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:23.097701 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 12:55:23.097905 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:23.097768 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" podUID="3e30e333-c7fd-4aaa-9541-3a1ef29189a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 12:55:23.209865 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:23.209824 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:24.115870 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:24.115600 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" Oct 31 12:55:24.116791 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:24.116772 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" Oct 31 12:55:24.119045 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:24.119029 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" Oct 31 12:55:24.160326 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:24.160292 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:25.137912 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:25.137878 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:26.105941 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:26.105901 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:27.148917 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:27.148882 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:28.893109 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:28.893073 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:29.099005 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:29.098025 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:29.295631 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:29.295593 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:29.912790 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:29.912752 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:30.541131 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:30.541095 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:30.749892 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:30.743638 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:30.951647 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:30.951596 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:31.032502 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:31.032468 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6cb6bcb954-xjt8m"] Oct 31 12:55:31.037439 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:31.037418 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cb6bcb954-xjt8m" Oct 31 12:55:31.078441 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:31.078410 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cb6bcb954-xjt8m"] Oct 31 12:55:31.090069 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:31.090044 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-d026e517442e\"" Oct 31 12:55:31.090718 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:31.090699 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1e51dd5d4038\"" Oct 31 12:55:31.144485 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:31.144456 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7rxk\" (UniqueName: \"kubernetes.io/projected/19906921-8b1a-48ef-a75e-391211820c68-kube-api-access-v7rxk\") pod \"payload-processing-external-model-6cb6bcb954-xjt8m\" (UID: \"19906921-8b1a-48ef-a75e-391211820c68\") " pod="models-as-a-service/payload-processing-external-model-6cb6bcb954-xjt8m" Oct 31 12:55:31.144660 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:31.144518 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19906921-8b1a-48ef-a75e-391211820c68-plugins-config-volume\") pod \"payload-processing-external-model-6cb6bcb954-xjt8m\" (UID: \"19906921-8b1a-48ef-a75e-391211820c68\") " pod="models-as-a-service/payload-processing-external-model-6cb6bcb954-xjt8m" Oct 31 12:55:31.144660 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:31.144636 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19906921-8b1a-48ef-a75e-391211820c68-provider-credentials\") pod \"payload-processing-external-model-6cb6bcb954-xjt8m\" (UID: \"19906921-8b1a-48ef-a75e-391211820c68\") " pod="models-as-a-service/payload-processing-external-model-6cb6bcb954-xjt8m" Oct 31 12:55:31.144807 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:31.144790 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19906921-8b1a-48ef-a75e-391211820c68-routing-overlay-volume\") pod \"payload-processing-external-model-6cb6bcb954-xjt8m\" (UID: \"19906921-8b1a-48ef-a75e-391211820c68\") " pod="models-as-a-service/payload-processing-external-model-6cb6bcb954-xjt8m" Oct 31 12:55:31.246095 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:31.246016 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19906921-8b1a-48ef-a75e-391211820c68-routing-overlay-volume\") pod \"payload-processing-external-model-6cb6bcb954-xjt8m\" (UID: \"19906921-8b1a-48ef-a75e-391211820c68\") " pod="models-as-a-service/payload-processing-external-model-6cb6bcb954-xjt8m" Oct 31 12:55:31.246095 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:31.246092 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7rxk\" (UniqueName: \"kubernetes.io/projected/19906921-8b1a-48ef-a75e-391211820c68-kube-api-access-v7rxk\") pod \"payload-processing-external-model-6cb6bcb954-xjt8m\" (UID: \"19906921-8b1a-48ef-a75e-391211820c68\") " pod="models-as-a-service/payload-processing-external-model-6cb6bcb954-xjt8m" Oct 31 12:55:31.246269 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:31.246138 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19906921-8b1a-48ef-a75e-391211820c68-plugins-config-volume\") pod \"payload-processing-external-model-6cb6bcb954-xjt8m\" (UID: \"19906921-8b1a-48ef-a75e-391211820c68\") " pod="models-as-a-service/payload-processing-external-model-6cb6bcb954-xjt8m" Oct 31 12:55:31.246269 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:31.246163 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19906921-8b1a-48ef-a75e-391211820c68-provider-credentials\") pod \"payload-processing-external-model-6cb6bcb954-xjt8m\" (UID: \"19906921-8b1a-48ef-a75e-391211820c68\") " pod="models-as-a-service/payload-processing-external-model-6cb6bcb954-xjt8m" Oct 31 12:55:31.246865 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:31.246838 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19906921-8b1a-48ef-a75e-391211820c68-routing-overlay-volume\") pod \"payload-processing-external-model-6cb6bcb954-xjt8m\" (UID: \"19906921-8b1a-48ef-a75e-391211820c68\") " pod="models-as-a-service/payload-processing-external-model-6cb6bcb954-xjt8m" Oct 31 12:55:31.246966 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:31.246947 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19906921-8b1a-48ef-a75e-391211820c68-plugins-config-volume\") pod \"payload-processing-external-model-6cb6bcb954-xjt8m\" (UID: \"19906921-8b1a-48ef-a75e-391211820c68\") " pod="models-as-a-service/payload-processing-external-model-6cb6bcb954-xjt8m" Oct 31 12:55:31.248694 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:31.248672 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19906921-8b1a-48ef-a75e-391211820c68-provider-credentials\") pod \"payload-processing-external-model-6cb6bcb954-xjt8m\" (UID: \"19906921-8b1a-48ef-a75e-391211820c68\") " pod="models-as-a-service/payload-processing-external-model-6cb6bcb954-xjt8m" Oct 31 12:55:31.258783 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:31.258758 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7rxk\" (UniqueName: \"kubernetes.io/projected/19906921-8b1a-48ef-a75e-391211820c68-kube-api-access-v7rxk\") pod \"payload-processing-external-model-6cb6bcb954-xjt8m\" (UID: \"19906921-8b1a-48ef-a75e-391211820c68\") " pod="models-as-a-service/payload-processing-external-model-6cb6bcb954-xjt8m" Oct 31 12:55:31.350157 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:31.350122 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cb6bcb954-xjt8m" Oct 31 12:55:31.631216 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:31.631190 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cb6bcb954-xjt8m"] Oct 31 12:55:31.632866 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:55:31.632844 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19906921_8b1a_48ef_a75e_391211820c68.slice/crio-91bcbac138b3a650f12ef34dd3ed66796384b8e0fec2d491439b872af52eede6 WatchSource:0}: Error finding container 91bcbac138b3a650f12ef34dd3ed66796384b8e0fec2d491439b872af52eede6: Status 404 returned error can't find the container with id 91bcbac138b3a650f12ef34dd3ed66796384b8e0fec2d491439b872af52eede6 Oct 31 12:55:31.828612 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:31.828580 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cb6bcb954-xjt8m" event={"ID":"19906921-8b1a-48ef-a75e-391211820c68","Type":"ContainerStarted","Data":"e0181a640ddf2ddb2c08b14afb19f86a37571f172485656a838ded694c304f7f"} Oct 31 12:55:31.828799 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:31.828627 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cb6bcb954-xjt8m" event={"ID":"19906921-8b1a-48ef-a75e-391211820c68","Type":"ContainerStarted","Data":"91bcbac138b3a650f12ef34dd3ed66796384b8e0fec2d491439b872af52eede6"} Oct 31 12:55:31.828799 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:31.828713 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6cb6bcb954-xjt8m" Oct 31 12:55:31.848808 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:31.848756 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6cb6bcb954-xjt8m" podStartSLOduration=0.848738966 podStartE2EDuration="848.738966ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:55:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:55:31.847396902 +0000 UTC m=+1378.330519803" watchObservedRunningTime="2026-10-31 12:55:31.848738966 +0000 UTC m=+1378.331861868" Oct 31 12:55:31.914342 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:31.914241 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:32.287008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:32.286033 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:32.895452 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:32.895408 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:33.798331 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:33.797208 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:34.595513 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:34.595464 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:35.460521 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:35.460488 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:36.055285 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:36.055244 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:36.261444 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:36.261410 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:36.982610 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:36.982559 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb"] Oct 31 12:55:36.994125 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:36.991026 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" Oct 31 12:55:36.999899 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:36.999869 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb"] Oct 31 12:55:37.034879 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.034843 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91727fdb-0e14-4b48-b74f-05bfd90c0e17-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb\" (UID: \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" Oct 31 12:55:37.034879 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.034881 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91727fdb-0e14-4b48-b74f-05bfd90c0e17-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb\" (UID: \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" Oct 31 12:55:37.035129 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.034929 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91727fdb-0e14-4b48-b74f-05bfd90c0e17-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb\" (UID: \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" Oct 31 12:55:37.035129 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.034975 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91727fdb-0e14-4b48-b74f-05bfd90c0e17-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb\" (UID: \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" Oct 31 12:55:37.035129 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.035048 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91727fdb-0e14-4b48-b74f-05bfd90c0e17-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb\" (UID: \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" Oct 31 12:55:37.035129 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.035074 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91727fdb-0e14-4b48-b74f-05bfd90c0e17-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb\" (UID: \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" Oct 31 12:55:37.035129 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.035099 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hnmd\" (UniqueName: \"kubernetes.io/projected/91727fdb-0e14-4b48-b74f-05bfd90c0e17-kube-api-access-9hnmd\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb\" (UID: \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" Oct 31 12:55:37.035321 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.035168 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91727fdb-0e14-4b48-b74f-05bfd90c0e17-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb\" (UID: \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" Oct 31 12:55:37.035321 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.035213 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91727fdb-0e14-4b48-b74f-05bfd90c0e17-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb\" (UID: \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" Oct 31 12:55:37.035741 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.035706 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:37.136216 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.136175 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91727fdb-0e14-4b48-b74f-05bfd90c0e17-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb\" (UID: \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" Oct 31 12:55:37.136391 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.136247 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91727fdb-0e14-4b48-b74f-05bfd90c0e17-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb\" (UID: \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" Oct 31 12:55:37.136391 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.136313 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91727fdb-0e14-4b48-b74f-05bfd90c0e17-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb\" (UID: \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" Oct 31 12:55:37.136391 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.136348 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91727fdb-0e14-4b48-b74f-05bfd90c0e17-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb\" (UID: \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" Oct 31 12:55:37.136391 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.136371 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hnmd\" (UniqueName: \"kubernetes.io/projected/91727fdb-0e14-4b48-b74f-05bfd90c0e17-kube-api-access-9hnmd\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb\" (UID: \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" Oct 31 12:55:37.136609 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.136447 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91727fdb-0e14-4b48-b74f-05bfd90c0e17-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb\" (UID: \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" Oct 31 12:55:37.136609 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.136514 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91727fdb-0e14-4b48-b74f-05bfd90c0e17-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb\" (UID: \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" Oct 31 12:55:37.136609 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.136556 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91727fdb-0e14-4b48-b74f-05bfd90c0e17-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb\" (UID: \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" Oct 31 12:55:37.136609 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.136582 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91727fdb-0e14-4b48-b74f-05bfd90c0e17-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb\" (UID: \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" Oct 31 12:55:37.136877 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.136851 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91727fdb-0e14-4b48-b74f-05bfd90c0e17-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb\" (UID: \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" Oct 31 12:55:37.136950 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.136878 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91727fdb-0e14-4b48-b74f-05bfd90c0e17-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb\" (UID: \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" Oct 31 12:55:37.137124 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.137103 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91727fdb-0e14-4b48-b74f-05bfd90c0e17-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb\" (UID: \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" Oct 31 12:55:37.137194 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.137131 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91727fdb-0e14-4b48-b74f-05bfd90c0e17-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb\" (UID: \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" Oct 31 12:55:37.137194 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.137143 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91727fdb-0e14-4b48-b74f-05bfd90c0e17-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb\" (UID: \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" Oct 31 12:55:37.138652 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.138631 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91727fdb-0e14-4b48-b74f-05bfd90c0e17-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb\" (UID: \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" Oct 31 12:55:37.139228 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.139208 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91727fdb-0e14-4b48-b74f-05bfd90c0e17-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb\" (UID: \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" Oct 31 12:55:37.149815 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.149792 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hnmd\" (UniqueName: \"kubernetes.io/projected/91727fdb-0e14-4b48-b74f-05bfd90c0e17-kube-api-access-9hnmd\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb\" (UID: \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" Oct 31 12:55:37.150215 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.150198 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91727fdb-0e14-4b48-b74f-05bfd90c0e17-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb\" (UID: \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" Oct 31 12:55:37.336602 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.336520 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" Oct 31 12:55:37.619210 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.619186 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb"] Oct 31 12:55:37.620822 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:55:37.620793 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91727fdb_0e14_4b48_b74f_05bfd90c0e17.slice/crio-739b95588ee20fa27026e5fdb3cb578989aea57dd308363b61844ba02e830e31 WatchSource:0}: Error finding container 739b95588ee20fa27026e5fdb3cb578989aea57dd308363b61844ba02e830e31: Status 404 returned error can't find the container with id 739b95588ee20fa27026e5fdb3cb578989aea57dd308363b61844ba02e830e31 Oct 31 12:55:37.623408 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.623373 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:55:37.623497 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.623443 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:55:37.623497 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.623477 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:55:37.914193 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.913804 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" event={"ID":"91727fdb-0e14-4b48-b74f-05bfd90c0e17","Type":"ContainerStarted","Data":"7fcac52694ff14c2a80efaf7248ea045280cf904060fdc34e1ad18454948a7af"} Oct 31 12:55:37.914193 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.913876 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" event={"ID":"91727fdb-0e14-4b48-b74f-05bfd90c0e17","Type":"ContainerStarted","Data":"739b95588ee20fa27026e5fdb3cb578989aea57dd308363b61844ba02e830e31"} Oct 31 12:55:37.936862 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.936819 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:37.949681 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:37.949617 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" podStartSLOduration=1.949600487 podStartE2EDuration="1.949600487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:55:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:55:37.946403818 +0000 UTC m=+1384.429526718" watchObservedRunningTime="2026-10-31 12:55:37.949600487 +0000 UTC m=+1384.432723392" Oct 31 12:55:38.337340 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:38.337301 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" Oct 31 12:55:38.338691 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:38.338669 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 12:55:38.338766 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:38.338714 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" podUID="91727fdb-0e14-4b48-b74f-05bfd90c0e17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 12:55:38.752083 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:38.752034 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:39.231683 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:39.231639 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:39.337098 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:39.337061 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 12:55:39.337274 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:39.337113 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" podUID="91727fdb-0e14-4b48-b74f-05bfd90c0e17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 12:55:39.665561 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:39.665470 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:40.337284 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:40.337236 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 12:55:40.337492 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:40.337315 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" podUID="91727fdb-0e14-4b48-b74f-05bfd90c0e17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 12:55:40.735004 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:40.733408 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:41.339018 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:41.338361 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 12:55:41.339018 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:41.338423 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" podUID="91727fdb-0e14-4b48-b74f-05bfd90c0e17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 12:55:42.339015 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:42.338944 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 12:55:42.339641 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:42.339040 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" podUID="91727fdb-0e14-4b48-b74f-05bfd90c0e17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 12:55:42.551646 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:42.551606 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:42.901222 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:42.901189 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6cb6bcb954-xjt8m" Oct 31 12:55:43.030010 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:43.010752 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-57fc876657-4b2dx"] Oct 31 12:55:43.349136 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:43.349104 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 12:55:43.349757 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:43.349724 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" podUID="91727fdb-0e14-4b48-b74f-05bfd90c0e17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 12:55:43.851010 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:43.818052 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk"] Oct 31 12:55:43.919118 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:43.894302 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" Oct 31 12:55:44.034008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:44.029619 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f4b3219-9e15-407f-9715-9865e01ea8f7-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk\" (UID: \"4f4b3219-9e15-407f-9715-9865e01ea8f7\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" Oct 31 12:55:44.034008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:44.029697 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f4b3219-9e15-407f-9715-9865e01ea8f7-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk\" (UID: \"4f4b3219-9e15-407f-9715-9865e01ea8f7\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" Oct 31 12:55:44.034008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:44.029766 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f4b3219-9e15-407f-9715-9865e01ea8f7-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk\" (UID: \"4f4b3219-9e15-407f-9715-9865e01ea8f7\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" Oct 31 12:55:44.034008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:44.029847 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f4b3219-9e15-407f-9715-9865e01ea8f7-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk\" (UID: \"4f4b3219-9e15-407f-9715-9865e01ea8f7\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" Oct 31 12:55:44.034008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:44.029876 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f4b3219-9e15-407f-9715-9865e01ea8f7-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk\" (UID: \"4f4b3219-9e15-407f-9715-9865e01ea8f7\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" Oct 31 12:55:44.034008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:44.029998 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f4b3219-9e15-407f-9715-9865e01ea8f7-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk\" (UID: \"4f4b3219-9e15-407f-9715-9865e01ea8f7\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" Oct 31 12:55:44.034008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:44.030043 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f4b3219-9e15-407f-9715-9865e01ea8f7-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk\" (UID: \"4f4b3219-9e15-407f-9715-9865e01ea8f7\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" Oct 31 12:55:44.034008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:44.030111 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f4b3219-9e15-407f-9715-9865e01ea8f7-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk\" (UID: \"4f4b3219-9e15-407f-9715-9865e01ea8f7\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" Oct 31 12:55:44.034008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:44.030146 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xcf5\" (UniqueName: \"kubernetes.io/projected/4f4b3219-9e15-407f-9715-9865e01ea8f7-kube-api-access-8xcf5\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk\" (UID: \"4f4b3219-9e15-407f-9715-9865e01ea8f7\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" Oct 31 12:55:44.163060 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:44.134833 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f4b3219-9e15-407f-9715-9865e01ea8f7-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk\" (UID: \"4f4b3219-9e15-407f-9715-9865e01ea8f7\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" Oct 31 12:55:44.163060 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:44.134886 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f4b3219-9e15-407f-9715-9865e01ea8f7-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk\" (UID: \"4f4b3219-9e15-407f-9715-9865e01ea8f7\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" Oct 31 12:55:44.163060 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:44.134966 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f4b3219-9e15-407f-9715-9865e01ea8f7-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk\" (UID: \"4f4b3219-9e15-407f-9715-9865e01ea8f7\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" Oct 31 12:55:44.163060 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:44.135023 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f4b3219-9e15-407f-9715-9865e01ea8f7-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk\" (UID: \"4f4b3219-9e15-407f-9715-9865e01ea8f7\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" Oct 31 12:55:44.163060 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:44.135078 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f4b3219-9e15-407f-9715-9865e01ea8f7-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk\" (UID: \"4f4b3219-9e15-407f-9715-9865e01ea8f7\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" Oct 31 12:55:44.163060 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:44.135106 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xcf5\" (UniqueName: \"kubernetes.io/projected/4f4b3219-9e15-407f-9715-9865e01ea8f7-kube-api-access-8xcf5\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk\" (UID: \"4f4b3219-9e15-407f-9715-9865e01ea8f7\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" Oct 31 12:55:44.163060 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:44.135153 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f4b3219-9e15-407f-9715-9865e01ea8f7-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk\" (UID: \"4f4b3219-9e15-407f-9715-9865e01ea8f7\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" Oct 31 12:55:44.163060 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:44.135209 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f4b3219-9e15-407f-9715-9865e01ea8f7-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk\" (UID: \"4f4b3219-9e15-407f-9715-9865e01ea8f7\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" Oct 31 12:55:44.163060 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:44.135270 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f4b3219-9e15-407f-9715-9865e01ea8f7-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk\" (UID: \"4f4b3219-9e15-407f-9715-9865e01ea8f7\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" Oct 31 12:55:44.163060 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:44.136122 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f4b3219-9e15-407f-9715-9865e01ea8f7-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk\" (UID: \"4f4b3219-9e15-407f-9715-9865e01ea8f7\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" Oct 31 12:55:44.163060 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:44.136426 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f4b3219-9e15-407f-9715-9865e01ea8f7-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk\" (UID: \"4f4b3219-9e15-407f-9715-9865e01ea8f7\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" Oct 31 12:55:44.195761 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:44.195722 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f4b3219-9e15-407f-9715-9865e01ea8f7-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk\" (UID: \"4f4b3219-9e15-407f-9715-9865e01ea8f7\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" Oct 31 12:55:44.196368 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:44.196343 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f4b3219-9e15-407f-9715-9865e01ea8f7-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk\" (UID: \"4f4b3219-9e15-407f-9715-9865e01ea8f7\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" Oct 31 12:55:44.213654 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:44.213623 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f4b3219-9e15-407f-9715-9865e01ea8f7-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk\" (UID: \"4f4b3219-9e15-407f-9715-9865e01ea8f7\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" Oct 31 12:55:44.226185 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:44.225132 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f4b3219-9e15-407f-9715-9865e01ea8f7-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk\" (UID: \"4f4b3219-9e15-407f-9715-9865e01ea8f7\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" Oct 31 12:55:44.232087 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:44.226534 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f4b3219-9e15-407f-9715-9865e01ea8f7-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk\" (UID: \"4f4b3219-9e15-407f-9715-9865e01ea8f7\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" Oct 31 12:55:44.250006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:44.244093 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk"] Oct 31 12:55:44.337105 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:44.335361 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f4b3219-9e15-407f-9715-9865e01ea8f7-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk\" (UID: \"4f4b3219-9e15-407f-9715-9865e01ea8f7\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" Oct 31 12:55:44.385373 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:44.383038 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xcf5\" (UniqueName: \"kubernetes.io/projected/4f4b3219-9e15-407f-9715-9865e01ea8f7-kube-api-access-8xcf5\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk\" (UID: \"4f4b3219-9e15-407f-9715-9865e01ea8f7\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" Oct 31 12:55:44.385373 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:44.383368 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 12:55:44.385373 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:44.383416 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" podUID="91727fdb-0e14-4b48-b74f-05bfd90c0e17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 12:55:44.640007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:44.623024 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:44.657053 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:44.656383 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" Oct 31 12:55:45.346204 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:45.338371 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 12:55:45.346204 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:45.338435 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" podUID="91727fdb-0e14-4b48-b74f-05bfd90c0e17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 12:55:45.473825 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:45.473674 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:45.531750 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:45.531270 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk"] Oct 31 12:55:45.533050 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:45.532973 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:55:45.533257 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:45.533233 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:55:45.533400 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:45.533381 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:55:46.144039 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:46.141030 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:46.337316 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:46.337284 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 12:55:46.337492 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:46.337334 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" podUID="91727fdb-0e14-4b48-b74f-05bfd90c0e17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 12:55:46.487579 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:46.487540 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" event={"ID":"4f4b3219-9e15-407f-9715-9865e01ea8f7","Type":"ContainerStarted","Data":"d7e43ad29f8df3b5b72f3717a607d985c30d9adffae3f992bd0b244f420e3c3f"} Oct 31 12:55:46.488130 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:46.487622 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" event={"ID":"4f4b3219-9e15-407f-9715-9865e01ea8f7","Type":"ContainerStarted","Data":"3672deae979ec0109af66537f6e47895efd7862044dc9eca09116cc916c385a3"} Oct 31 12:55:46.512819 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:46.512761 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" podStartSLOduration=3.512743283 podStartE2EDuration="3.512743283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:55:46.511258992 +0000 UTC m=+1392.994381896" watchObservedRunningTime="2026-10-31 12:55:46.512743283 +0000 UTC m=+1392.995866185" Oct 31 12:55:46.668006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:46.667045 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" Oct 31 12:55:46.668400 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:46.668368 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 31 12:55:46.668604 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:46.668578 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" podUID="4f4b3219-9e15-407f-9715-9865e01ea8f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 31 12:55:47.007606 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:47.005503 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:47.677006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:47.676380 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 31 12:55:47.677006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:47.676447 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" podUID="4f4b3219-9e15-407f-9715-9865e01ea8f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 31 12:55:47.876222 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:47.835166 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:48.011413 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:48.011374 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-57fc876657-4b2dx" podUID="77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c" containerName="payload-processing" containerID="cri-o://e10768efa72ff369d05959f3cf0fdc2d8d87ef92f9f6996214d75f78713ec756" gracePeriod=25 Oct 31 12:55:48.382204 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:48.381466 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:55:48.382204 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:48.381513 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" podUID="91727fdb-0e14-4b48-b74f-05bfd90c0e17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:48.383568 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:48.383528 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57fc876657-4b2dx" Oct 31 12:55:48.424698 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:48.424663 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c-plugins-config-volume\") pod \"77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c\" (UID: \"77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c\") " Oct 31 12:55:48.424881 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:48.424715 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c-routing-overlay-volume\") pod \"77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c\" (UID: \"77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c\") " Oct 31 12:55:48.424881 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:48.424765 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c-kube-api-access-lh6fs\" (UniqueName: \"kubernetes.io/projected/77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c-kube-api-access-lh6fs\") pod \"77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c\" (UID: \"77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c\") " Oct 31 12:55:48.424881 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:48.424807 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c-provider-credentials\") pod \"77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c\" (UID: \"77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c\") " Oct 31 12:55:48.425107 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:48.425065 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c-plugins-config-volume" pod "77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c" (UID: "77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:55:48.425423 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:48.425310 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:48.425929 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:48.425900 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c-routing-overlay-volume" pod "77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c" (UID: "77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:55:48.427725 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:48.427702 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c-kube-api-access-lh6fs" pod "77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c" (UID: "77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c"). InnerVolumeSpecName "kube-api-access-lh6fs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:55:48.427725 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:48.427713 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c-provider-credentials" pod "77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c" (UID: "77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:55:48.483756 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:48.483725 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" Oct 31 12:55:48.484752 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:48.484403 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" Oct 31 12:55:48.485224 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:48.485168 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" Oct 31 12:55:48.502108 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:48.502085 6841 generic.go:356] "Generic (PLEG): container finished" podID="77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c" containerID="e10768efa72ff369d05959f3cf0fdc2d8d87ef92f9f6996214d75f78713ec756" exitCode=0 Oct 31 12:55:48.502222 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:48.502133 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57fc876657-4b2dx" Oct 31 12:55:48.502222 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:48.502173 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57fc876657-4b2dx" event={"ID":"77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c","Type":"ContainerDied","Data":"e10768efa72ff369d05959f3cf0fdc2d8d87ef92f9f6996214d75f78713ec756"} Oct 31 12:55:48.502347 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:48.502222 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57fc876657-4b2dx" event={"ID":"77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c","Type":"ContainerDied","Data":"f2b29809be15b4c526d1899218da8d5cfb54215fb1d79ea14e2e344648380524"} Oct 31 12:55:48.502347 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:48.502247 6841 scope.go:122] "RemoveContainer" containerID="e10768efa72ff369d05959f3cf0fdc2d8d87ef92f9f6996214d75f78713ec756" Oct 31 12:55:48.514948 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:48.514925 6841 scope.go:122] "RemoveContainer" containerID="e10768efa72ff369d05959f3cf0fdc2d8d87ef92f9f6996214d75f78713ec756" Oct 31 12:55:48.515241 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:55:48.515209 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e10768efa72ff369d05959f3cf0fdc2d8d87ef92f9f6996214d75f78713ec756\": container with ID starting with e10768efa72ff369d05959f3cf0fdc2d8d87ef92f9f6996214d75f78713ec756 not found: ID does not exist" containerID="e10768efa72ff369d05959f3cf0fdc2d8d87ef92f9f6996214d75f78713ec756" Oct 31 12:55:48.515349 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:48.515249 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e10768efa72ff369d05959f3cf0fdc2d8d87ef92f9f6996214d75f78713ec756"} err="failed to get container status \"e10768efa72ff369d05959f3cf0fdc2d8d87ef92f9f6996214d75f78713ec756\": rpc error: code = NotFound desc = could not find container \"e10768efa72ff369d05959f3cf0fdc2d8d87ef92f9f6996214d75f78713ec756\": container with ID starting with e10768efa72ff369d05959f3cf0fdc2d8d87ef92f9f6996214d75f78713ec756 not found: ID does not exist" Oct 31 12:55:48.526515 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:48.526494 6841 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c-routing-overlay-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:48.526590 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:48.526516 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh6fs\" (UniqueName: \"kubernetes.io/projected/77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c-kube-api-access-lh6fs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:48.526590 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:48.526527 6841 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c-provider-credentials\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:48.582630 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:48.582469 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-57fc876657-4b2dx"] Oct 31 12:55:48.586645 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:48.586615 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-57fc876657-4b2dx"] Oct 31 12:55:48.644218 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:48.644028 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:48.674906 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:48.674870 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 31 12:55:48.675200 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:48.675171 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" podUID="4f4b3219-9e15-407f-9715-9865e01ea8f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 31 12:55:49.606009 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:49.594560 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:49.661006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:49.655272 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c" path="/var/lib/kubelet/pods/77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c/volumes" Oct 31 12:55:49.665414 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:49.665386 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 31 12:55:49.665619 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:49.665595 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" podUID="4f4b3219-9e15-407f-9715-9865e01ea8f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 31 12:55:50.052508 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:55:50.052446 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:55:50.326054 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:50.324513 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:50.666536 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:50.666443 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 31 12:55:50.666536 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:50.666512 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" podUID="4f4b3219-9e15-407f-9715-9865e01ea8f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 31 12:55:51.118358 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:51.118288 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:51.663662 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:51.663618 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 31 12:55:51.663869 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:51.663672 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" podUID="4f4b3219-9e15-407f-9715-9865e01ea8f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 31 12:55:51.822484 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:51.822438 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:52.514665 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:52.514398 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:52.667486 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:52.667456 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" Oct 31 12:55:52.667962 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:52.667932 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" Oct 31 12:55:52.668583 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:52.668565 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" Oct 31 12:55:53.375849 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:53.374709 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:56.132007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:56.131694 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:57.023700 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:57.019846 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:57.071635 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:57.025113 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-6lxfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:55:57.132187 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:57.132135 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" podUID="d0ad44f3-de70-4b31-8d39-ceeed24efd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:57.140123 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:57.140090 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fk2tx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:55:57.140380 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:57.140348 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" podUID="aa7e0a48-9dc5-4735-b1ae-cafd28ff982f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:57.921006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:57.914275 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:58.419006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:58.418435 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-ssz2q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:55:58.419006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:58.418513 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" podUID="47a4ec22-fe77-4a7b-a66a-5debadbf5800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:58.610006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:58.608504 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:58.808561 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:58.808527 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:55:59.505405 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:55:59.505363 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:00.599780 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:00.599733 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:01.313386 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:01.313347 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:02.014084 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:02.014043 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:02.792935 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:02.792848 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:02.878282 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:02.878253 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97"] Oct 31 12:56:02.878834 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:02.878820 6841 cpu_manager.go:414] "containerMap: removing container" podUID="77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c" containerName="payload-processing" Oct 31 12:56:02.878884 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:02.878835 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c" containerName="payload-processing" Oct 31 12:56:02.878938 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:02.878929 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="77fbaa7d-34e2-4d69-9fdd-b08fca36ad8c" containerName="payload-processing" Oct 31 12:56:02.882480 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:02.882461 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" Oct 31 12:56:02.897819 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:02.897791 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97"] Oct 31 12:56:02.960209 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:02.960178 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rd8f\" (UniqueName: \"kubernetes.io/projected/cb204311-3184-4407-bea7-a759f629ec3e-kube-api-access-6rd8f\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97\" (UID: \"cb204311-3184-4407-bea7-a759f629ec3e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" Oct 31 12:56:02.960372 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:02.960237 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb204311-3184-4407-bea7-a759f629ec3e-istio-envoy\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97\" (UID: \"cb204311-3184-4407-bea7-a759f629ec3e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" Oct 31 12:56:02.960445 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:02.960372 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb204311-3184-4407-bea7-a759f629ec3e-istiod-ca-cert\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97\" (UID: \"cb204311-3184-4407-bea7-a759f629ec3e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" Oct 31 12:56:02.960497 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:02.960478 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb204311-3184-4407-bea7-a759f629ec3e-credential-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97\" (UID: \"cb204311-3184-4407-bea7-a759f629ec3e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" Oct 31 12:56:02.960587 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:02.960569 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb204311-3184-4407-bea7-a759f629ec3e-istio-podinfo\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97\" (UID: \"cb204311-3184-4407-bea7-a759f629ec3e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" Oct 31 12:56:02.960624 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:02.960608 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb204311-3184-4407-bea7-a759f629ec3e-istio-data\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97\" (UID: \"cb204311-3184-4407-bea7-a759f629ec3e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" Oct 31 12:56:02.960709 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:02.960694 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb204311-3184-4407-bea7-a759f629ec3e-istio-token\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97\" (UID: \"cb204311-3184-4407-bea7-a759f629ec3e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" Oct 31 12:56:02.960770 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:02.960750 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb204311-3184-4407-bea7-a759f629ec3e-workload-certs\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97\" (UID: \"cb204311-3184-4407-bea7-a759f629ec3e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" Oct 31 12:56:02.960825 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:02.960794 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb204311-3184-4407-bea7-a759f629ec3e-workload-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97\" (UID: \"cb204311-3184-4407-bea7-a759f629ec3e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" Oct 31 12:56:03.062158 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:03.062084 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb204311-3184-4407-bea7-a759f629ec3e-istio-envoy\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97\" (UID: \"cb204311-3184-4407-bea7-a759f629ec3e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" Oct 31 12:56:03.062158 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:03.062136 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb204311-3184-4407-bea7-a759f629ec3e-istiod-ca-cert\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97\" (UID: \"cb204311-3184-4407-bea7-a759f629ec3e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" Oct 31 12:56:03.062653 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:03.062175 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb204311-3184-4407-bea7-a759f629ec3e-credential-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97\" (UID: \"cb204311-3184-4407-bea7-a759f629ec3e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" Oct 31 12:56:03.062653 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:03.062223 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb204311-3184-4407-bea7-a759f629ec3e-istio-podinfo\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97\" (UID: \"cb204311-3184-4407-bea7-a759f629ec3e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" Oct 31 12:56:03.062653 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:03.062249 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb204311-3184-4407-bea7-a759f629ec3e-istio-data\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97\" (UID: \"cb204311-3184-4407-bea7-a759f629ec3e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" Oct 31 12:56:03.062653 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:03.062316 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb204311-3184-4407-bea7-a759f629ec3e-istio-token\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97\" (UID: \"cb204311-3184-4407-bea7-a759f629ec3e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" Oct 31 12:56:03.062653 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:03.062343 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb204311-3184-4407-bea7-a759f629ec3e-workload-certs\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97\" (UID: \"cb204311-3184-4407-bea7-a759f629ec3e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" Oct 31 12:56:03.062653 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:03.062376 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb204311-3184-4407-bea7-a759f629ec3e-workload-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97\" (UID: \"cb204311-3184-4407-bea7-a759f629ec3e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" Oct 31 12:56:03.062653 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:03.062408 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rd8f\" (UniqueName: \"kubernetes.io/projected/cb204311-3184-4407-bea7-a759f629ec3e-kube-api-access-6rd8f\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97\" (UID: \"cb204311-3184-4407-bea7-a759f629ec3e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" Oct 31 12:56:03.062653 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:03.062609 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb204311-3184-4407-bea7-a759f629ec3e-credential-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97\" (UID: \"cb204311-3184-4407-bea7-a759f629ec3e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" Oct 31 12:56:03.062962 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:03.062706 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb204311-3184-4407-bea7-a759f629ec3e-istio-data\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97\" (UID: \"cb204311-3184-4407-bea7-a759f629ec3e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" Oct 31 12:56:03.062962 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:03.062747 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb204311-3184-4407-bea7-a759f629ec3e-workload-certs\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97\" (UID: \"cb204311-3184-4407-bea7-a759f629ec3e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" Oct 31 12:56:03.062962 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:03.062869 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb204311-3184-4407-bea7-a759f629ec3e-workload-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97\" (UID: \"cb204311-3184-4407-bea7-a759f629ec3e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" Oct 31 12:56:03.063100 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:03.063010 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb204311-3184-4407-bea7-a759f629ec3e-istiod-ca-cert\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97\" (UID: \"cb204311-3184-4407-bea7-a759f629ec3e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" Oct 31 12:56:03.064530 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:03.064508 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb204311-3184-4407-bea7-a759f629ec3e-istio-envoy\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97\" (UID: \"cb204311-3184-4407-bea7-a759f629ec3e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" Oct 31 12:56:03.064772 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:03.064755 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb204311-3184-4407-bea7-a759f629ec3e-istio-podinfo\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97\" (UID: \"cb204311-3184-4407-bea7-a759f629ec3e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" Oct 31 12:56:03.107729 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:03.107697 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rd8f\" (UniqueName: \"kubernetes.io/projected/cb204311-3184-4407-bea7-a759f629ec3e-kube-api-access-6rd8f\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97\" (UID: \"cb204311-3184-4407-bea7-a759f629ec3e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" Oct 31 12:56:03.107875 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:03.107851 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb204311-3184-4407-bea7-a759f629ec3e-istio-token\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97\" (UID: \"cb204311-3184-4407-bea7-a759f629ec3e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" Oct 31 12:56:03.195862 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:03.195832 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" Oct 31 12:56:03.482378 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:03.482355 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97"] Oct 31 12:56:03.483393 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:56:03.483368 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb204311_3184_4407_bea7_a759f629ec3e.slice/crio-4d1487d4dd623c13010d08afd4850035d93f87eee5237cf482c606964801318c WatchSource:0}: Error finding container 4d1487d4dd623c13010d08afd4850035d93f87eee5237cf482c606964801318c: Status 404 returned error can't find the container with id 4d1487d4dd623c13010d08afd4850035d93f87eee5237cf482c606964801318c Oct 31 12:56:03.485862 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:03.485827 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:56:03.485996 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:03.485891 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:56:03.485996 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:03.485920 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:56:03.728025 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:03.723514 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:04.171395 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:04.171354 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" event={"ID":"cb204311-3184-4407-bea7-a759f629ec3e","Type":"ContainerStarted","Data":"51bbbf9626a1223e286fd1e7afe46de522c4eee1f293ac840987e5d0b64a75e4"} Oct 31 12:56:04.171395 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:04.171402 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" event={"ID":"cb204311-3184-4407-bea7-a759f629ec3e","Type":"ContainerStarted","Data":"4d1487d4dd623c13010d08afd4850035d93f87eee5237cf482c606964801318c"} Oct 31 12:56:04.196239 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:04.196204 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" Oct 31 12:56:04.198027 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:04.197995 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 12:56:04.198132 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:04.198053 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" podUID="cb204311-3184-4407-bea7-a759f629ec3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 12:56:04.198399 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:04.198362 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" podStartSLOduration=2.198350145 podStartE2EDuration="2.198350145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:56:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:56:04.194812452 +0000 UTC m=+1410.677935355" watchObservedRunningTime="2026-10-31 12:56:04.198350145 +0000 UTC m=+1410.681473046" Oct 31 12:56:04.522191 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:04.520061 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:05.196793 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:05.196763 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 12:56:05.197216 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:05.196813 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" podUID="cb204311-3184-4407-bea7-a759f629ec3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 12:56:05.316374 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:05.316337 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:06.010110 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:06.010068 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:06.196265 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:06.196233 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 12:56:06.196463 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:06.196287 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" podUID="cb204311-3184-4407-bea7-a759f629ec3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 12:56:06.819006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:06.818540 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:07.012576 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.012543 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58"] Oct 31 12:56:07.017775 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.017747 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" Oct 31 12:56:07.030876 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.030848 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58"] Oct 31 12:56:07.077102 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.075803 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/189ebc59-a986-4e40-a9b1-25a350c5d397-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58\" (UID: \"189ebc59-a986-4e40-a9b1-25a350c5d397\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" Oct 31 12:56:07.077102 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.075877 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jp4x\" (UniqueName: \"kubernetes.io/projected/189ebc59-a986-4e40-a9b1-25a350c5d397-kube-api-access-2jp4x\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58\" (UID: \"189ebc59-a986-4e40-a9b1-25a350c5d397\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" Oct 31 12:56:07.077102 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.075926 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/189ebc59-a986-4e40-a9b1-25a350c5d397-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58\" (UID: \"189ebc59-a986-4e40-a9b1-25a350c5d397\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" Oct 31 12:56:07.077102 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.076043 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/189ebc59-a986-4e40-a9b1-25a350c5d397-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58\" (UID: \"189ebc59-a986-4e40-a9b1-25a350c5d397\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" Oct 31 12:56:07.077102 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.076071 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/189ebc59-a986-4e40-a9b1-25a350c5d397-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58\" (UID: \"189ebc59-a986-4e40-a9b1-25a350c5d397\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" Oct 31 12:56:07.077102 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.076122 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/189ebc59-a986-4e40-a9b1-25a350c5d397-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58\" (UID: \"189ebc59-a986-4e40-a9b1-25a350c5d397\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" Oct 31 12:56:07.077102 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.076145 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/189ebc59-a986-4e40-a9b1-25a350c5d397-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58\" (UID: \"189ebc59-a986-4e40-a9b1-25a350c5d397\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" Oct 31 12:56:07.077102 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.076192 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/189ebc59-a986-4e40-a9b1-25a350c5d397-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58\" (UID: \"189ebc59-a986-4e40-a9b1-25a350c5d397\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" Oct 31 12:56:07.077102 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.076260 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/189ebc59-a986-4e40-a9b1-25a350c5d397-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58\" (UID: \"189ebc59-a986-4e40-a9b1-25a350c5d397\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" Oct 31 12:56:07.081072 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.081039 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt"] Oct 31 12:56:07.086110 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.086079 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" Oct 31 12:56:07.087174 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.087152 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt"] Oct 31 12:56:07.177424 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.177384 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/954541ff-08a9-4c5e-ab44-a9f6148e0315-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt\" (UID: \"954541ff-08a9-4c5e-ab44-a9f6148e0315\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" Oct 31 12:56:07.177424 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.177426 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/189ebc59-a986-4e40-a9b1-25a350c5d397-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58\" (UID: \"189ebc59-a986-4e40-a9b1-25a350c5d397\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" Oct 31 12:56:07.177621 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.177449 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/954541ff-08a9-4c5e-ab44-a9f6148e0315-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt\" (UID: \"954541ff-08a9-4c5e-ab44-a9f6148e0315\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" Oct 31 12:56:07.177655 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.177626 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/954541ff-08a9-4c5e-ab44-a9f6148e0315-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt\" (UID: \"954541ff-08a9-4c5e-ab44-a9f6148e0315\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" Oct 31 12:56:07.177694 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.177665 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/954541ff-08a9-4c5e-ab44-a9f6148e0315-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt\" (UID: \"954541ff-08a9-4c5e-ab44-a9f6148e0315\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" Oct 31 12:56:07.177737 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.177695 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/189ebc59-a986-4e40-a9b1-25a350c5d397-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58\" (UID: \"189ebc59-a986-4e40-a9b1-25a350c5d397\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" Oct 31 12:56:07.177860 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.177840 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/189ebc59-a986-4e40-a9b1-25a350c5d397-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58\" (UID: \"189ebc59-a986-4e40-a9b1-25a350c5d397\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" Oct 31 12:56:07.177955 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.177915 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/954541ff-08a9-4c5e-ab44-a9f6148e0315-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt\" (UID: \"954541ff-08a9-4c5e-ab44-a9f6148e0315\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" Oct 31 12:56:07.178059 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.178032 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/954541ff-08a9-4c5e-ab44-a9f6148e0315-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt\" (UID: \"954541ff-08a9-4c5e-ab44-a9f6148e0315\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" Oct 31 12:56:07.178121 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.178042 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/189ebc59-a986-4e40-a9b1-25a350c5d397-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58\" (UID: \"189ebc59-a986-4e40-a9b1-25a350c5d397\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" Oct 31 12:56:07.178121 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.178105 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/189ebc59-a986-4e40-a9b1-25a350c5d397-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58\" (UID: \"189ebc59-a986-4e40-a9b1-25a350c5d397\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" Oct 31 12:56:07.178260 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.178241 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jp4x\" (UniqueName: \"kubernetes.io/projected/189ebc59-a986-4e40-a9b1-25a350c5d397-kube-api-access-2jp4x\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58\" (UID: \"189ebc59-a986-4e40-a9b1-25a350c5d397\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" Oct 31 12:56:07.178338 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.178314 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/954541ff-08a9-4c5e-ab44-a9f6148e0315-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt\" (UID: \"954541ff-08a9-4c5e-ab44-a9f6148e0315\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" Oct 31 12:56:07.178395 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.178374 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/189ebc59-a986-4e40-a9b1-25a350c5d397-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58\" (UID: \"189ebc59-a986-4e40-a9b1-25a350c5d397\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" Oct 31 12:56:07.178486 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.178469 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm6js\" (UniqueName: \"kubernetes.io/projected/954541ff-08a9-4c5e-ab44-a9f6148e0315-kube-api-access-hm6js\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt\" (UID: \"954541ff-08a9-4c5e-ab44-a9f6148e0315\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" Oct 31 12:56:07.178676 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.178637 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/189ebc59-a986-4e40-a9b1-25a350c5d397-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58\" (UID: \"189ebc59-a986-4e40-a9b1-25a350c5d397\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" Oct 31 12:56:07.178747 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.178694 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/189ebc59-a986-4e40-a9b1-25a350c5d397-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58\" (UID: \"189ebc59-a986-4e40-a9b1-25a350c5d397\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" Oct 31 12:56:07.178747 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.178696 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/189ebc59-a986-4e40-a9b1-25a350c5d397-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58\" (UID: \"189ebc59-a986-4e40-a9b1-25a350c5d397\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" Oct 31 12:56:07.178747 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.178727 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/954541ff-08a9-4c5e-ab44-a9f6148e0315-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt\" (UID: \"954541ff-08a9-4c5e-ab44-a9f6148e0315\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" Oct 31 12:56:07.178907 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.178807 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/189ebc59-a986-4e40-a9b1-25a350c5d397-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58\" (UID: \"189ebc59-a986-4e40-a9b1-25a350c5d397\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" Oct 31 12:56:07.178907 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.178842 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/189ebc59-a986-4e40-a9b1-25a350c5d397-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58\" (UID: \"189ebc59-a986-4e40-a9b1-25a350c5d397\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" Oct 31 12:56:07.179141 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.179122 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/189ebc59-a986-4e40-a9b1-25a350c5d397-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58\" (UID: \"189ebc59-a986-4e40-a9b1-25a350c5d397\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" Oct 31 12:56:07.179420 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.179403 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/189ebc59-a986-4e40-a9b1-25a350c5d397-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58\" (UID: \"189ebc59-a986-4e40-a9b1-25a350c5d397\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" Oct 31 12:56:07.180616 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.180582 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/189ebc59-a986-4e40-a9b1-25a350c5d397-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58\" (UID: \"189ebc59-a986-4e40-a9b1-25a350c5d397\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" Oct 31 12:56:07.180927 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.180908 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/189ebc59-a986-4e40-a9b1-25a350c5d397-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58\" (UID: \"189ebc59-a986-4e40-a9b1-25a350c5d397\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" Oct 31 12:56:07.190828 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.190802 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/189ebc59-a986-4e40-a9b1-25a350c5d397-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58\" (UID: \"189ebc59-a986-4e40-a9b1-25a350c5d397\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" Oct 31 12:56:07.190828 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.190809 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jp4x\" (UniqueName: \"kubernetes.io/projected/189ebc59-a986-4e40-a9b1-25a350c5d397-kube-api-access-2jp4x\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58\" (UID: \"189ebc59-a986-4e40-a9b1-25a350c5d397\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" Oct 31 12:56:07.196733 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.196709 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 12:56:07.196828 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.196766 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" podUID="cb204311-3184-4407-bea7-a759f629ec3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 12:56:07.280554 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.280518 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/954541ff-08a9-4c5e-ab44-a9f6148e0315-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt\" (UID: \"954541ff-08a9-4c5e-ab44-a9f6148e0315\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" Oct 31 12:56:07.280695 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.280570 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm6js\" (UniqueName: \"kubernetes.io/projected/954541ff-08a9-4c5e-ab44-a9f6148e0315-kube-api-access-hm6js\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt\" (UID: \"954541ff-08a9-4c5e-ab44-a9f6148e0315\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" Oct 31 12:56:07.280695 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.280618 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/954541ff-08a9-4c5e-ab44-a9f6148e0315-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt\" (UID: \"954541ff-08a9-4c5e-ab44-a9f6148e0315\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" Oct 31 12:56:07.280695 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.280659 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/954541ff-08a9-4c5e-ab44-a9f6148e0315-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt\" (UID: \"954541ff-08a9-4c5e-ab44-a9f6148e0315\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" Oct 31 12:56:07.280695 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.280689 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/954541ff-08a9-4c5e-ab44-a9f6148e0315-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt\" (UID: \"954541ff-08a9-4c5e-ab44-a9f6148e0315\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" Oct 31 12:56:07.280895 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.280728 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/954541ff-08a9-4c5e-ab44-a9f6148e0315-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt\" (UID: \"954541ff-08a9-4c5e-ab44-a9f6148e0315\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" Oct 31 12:56:07.280895 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.280753 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/954541ff-08a9-4c5e-ab44-a9f6148e0315-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt\" (UID: \"954541ff-08a9-4c5e-ab44-a9f6148e0315\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" Oct 31 12:56:07.280895 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.280825 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/954541ff-08a9-4c5e-ab44-a9f6148e0315-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt\" (UID: \"954541ff-08a9-4c5e-ab44-a9f6148e0315\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" Oct 31 12:56:07.280895 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.280853 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/954541ff-08a9-4c5e-ab44-a9f6148e0315-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt\" (UID: \"954541ff-08a9-4c5e-ab44-a9f6148e0315\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" Oct 31 12:56:07.281196 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.281126 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/954541ff-08a9-4c5e-ab44-a9f6148e0315-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt\" (UID: \"954541ff-08a9-4c5e-ab44-a9f6148e0315\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" Oct 31 12:56:07.281254 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.281204 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/954541ff-08a9-4c5e-ab44-a9f6148e0315-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt\" (UID: \"954541ff-08a9-4c5e-ab44-a9f6148e0315\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" Oct 31 12:56:07.281306 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.281256 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/954541ff-08a9-4c5e-ab44-a9f6148e0315-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt\" (UID: \"954541ff-08a9-4c5e-ab44-a9f6148e0315\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" Oct 31 12:56:07.281369 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.281350 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/954541ff-08a9-4c5e-ab44-a9f6148e0315-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt\" (UID: \"954541ff-08a9-4c5e-ab44-a9f6148e0315\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" Oct 31 12:56:07.281436 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.281414 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/954541ff-08a9-4c5e-ab44-a9f6148e0315-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt\" (UID: \"954541ff-08a9-4c5e-ab44-a9f6148e0315\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" Oct 31 12:56:07.283095 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.283074 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/954541ff-08a9-4c5e-ab44-a9f6148e0315-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt\" (UID: \"954541ff-08a9-4c5e-ab44-a9f6148e0315\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" Oct 31 12:56:07.283312 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.283294 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/954541ff-08a9-4c5e-ab44-a9f6148e0315-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt\" (UID: \"954541ff-08a9-4c5e-ab44-a9f6148e0315\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" Oct 31 12:56:07.300834 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.300801 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/954541ff-08a9-4c5e-ab44-a9f6148e0315-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt\" (UID: \"954541ff-08a9-4c5e-ab44-a9f6148e0315\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" Oct 31 12:56:07.301031 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.301011 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm6js\" (UniqueName: \"kubernetes.io/projected/954541ff-08a9-4c5e-ab44-a9f6148e0315-kube-api-access-hm6js\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt\" (UID: \"954541ff-08a9-4c5e-ab44-a9f6148e0315\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" Oct 31 12:56:07.332927 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.332867 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" Oct 31 12:56:07.406145 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.406112 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" Oct 31 12:56:07.667009 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.663605 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58"] Oct 31 12:56:07.703680 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:56:07.703641 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod189ebc59_a986_4e40_a9b1_25a350c5d397.slice/crio-4f0cd0617f5cc38d1e5c4a6e7b1b6bd41518368bf2e862ebea853e5bc034e1cf WatchSource:0}: Error finding container 4f0cd0617f5cc38d1e5c4a6e7b1b6bd41518368bf2e862ebea853e5bc034e1cf: Status 404 returned error can't find the container with id 4f0cd0617f5cc38d1e5c4a6e7b1b6bd41518368bf2e862ebea853e5bc034e1cf Oct 31 12:56:07.738006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.737672 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:07.750244 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.745390 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:56:07.750244 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.745483 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:56:07.750244 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.745529 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:56:07.782883 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.782839 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt"] Oct 31 12:56:07.784132 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:56:07.784103 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod954541ff_08a9_4c5e_ab44_a9f6148e0315.slice/crio-439031760aa5fc56aae4ebe09d90bb242ad54ac6fdd5f179211289538164e6e3 WatchSource:0}: Error finding container 439031760aa5fc56aae4ebe09d90bb242ad54ac6fdd5f179211289538164e6e3: Status 404 returned error can't find the container with id 439031760aa5fc56aae4ebe09d90bb242ad54ac6fdd5f179211289538164e6e3 Oct 31 12:56:07.787162 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.787130 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:56:07.787235 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.787196 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:56:07.787235 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.787225 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:56:07.939813 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:07.939723 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:08.196885 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:08.196280 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 12:56:08.196885 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:08.196337 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" podUID="cb204311-3184-4407-bea7-a759f629ec3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 12:56:08.196885 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:08.196468 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" event={"ID":"954541ff-08a9-4c5e-ab44-a9f6148e0315","Type":"ContainerStarted","Data":"d7f476fd9acbeb79829c34771a4377a5a1bb38f36f9eb595096f4005328356c0"} Oct 31 12:56:08.196885 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:08.196525 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" event={"ID":"954541ff-08a9-4c5e-ab44-a9f6148e0315","Type":"ContainerStarted","Data":"439031760aa5fc56aae4ebe09d90bb242ad54ac6fdd5f179211289538164e6e3"} Oct 31 12:56:08.199425 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:08.199389 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" event={"ID":"189ebc59-a986-4e40-a9b1-25a350c5d397","Type":"ContainerStarted","Data":"646855c7cb04a308807921ad8a07dff98776c8bd427c21a9b7cbaf234565d69a"} Oct 31 12:56:08.199561 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:08.199451 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" event={"ID":"189ebc59-a986-4e40-a9b1-25a350c5d397","Type":"ContainerStarted","Data":"4f0cd0617f5cc38d1e5c4a6e7b1b6bd41518368bf2e862ebea853e5bc034e1cf"} Oct 31 12:56:08.293619 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:08.292853 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" podStartSLOduration=1.2928337380000001 podStartE2EDuration="1.292833738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:56:08.290683787 +0000 UTC m=+1414.773806730" watchObservedRunningTime="2026-10-31 12:56:08.292833738 +0000 UTC m=+1414.775956641" Oct 31 12:56:08.343011 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:08.336669 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" Oct 31 12:56:08.343011 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:08.336837 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 12:56:08.343011 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:08.336879 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" podUID="189ebc59-a986-4e40-a9b1-25a350c5d397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 12:56:08.347005 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:08.346693 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" podStartSLOduration=2.346673626 podStartE2EDuration="2.346673626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:56:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:56:08.343476395 +0000 UTC m=+1414.826599321" watchObservedRunningTime="2026-10-31 12:56:08.346673626 +0000 UTC m=+1414.829796528" Oct 31 12:56:08.413836 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:08.409316 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" Oct 31 12:56:08.414230 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:08.414202 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 12:56:08.414400 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:08.414373 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" podUID="954541ff-08a9-4c5e-ab44-a9f6148e0315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 12:56:08.783008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:08.781961 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:09.198006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:09.196369 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 12:56:09.198006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:09.196430 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" podUID="cb204311-3184-4407-bea7-a759f629ec3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 12:56:09.335007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:09.334385 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 12:56:09.335007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:09.334449 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" podUID="189ebc59-a986-4e40-a9b1-25a350c5d397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 12:56:09.426008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:09.417730 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 12:56:09.426008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:09.417794 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" podUID="954541ff-08a9-4c5e-ab44-a9f6148e0315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 12:56:10.204008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:10.203459 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 12:56:10.204008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:10.203524 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" podUID="cb204311-3184-4407-bea7-a759f629ec3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 12:56:10.350008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:10.335355 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 12:56:10.350008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:10.335419 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" podUID="189ebc59-a986-4e40-a9b1-25a350c5d397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 12:56:10.410508 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:10.410469 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 12:56:10.410888 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:10.410770 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" podUID="954541ff-08a9-4c5e-ab44-a9f6148e0315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 12:56:11.069150 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:11.051816 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:11.270565 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:11.270528 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 12:56:11.271165 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:11.271127 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" podUID="cb204311-3184-4407-bea7-a759f629ec3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 12:56:11.271682 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:11.271663 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-57fc876657-qh5lt"] Oct 31 12:56:11.440701 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:11.410161 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 12:56:11.440701 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:11.410226 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" podUID="954541ff-08a9-4c5e-ab44-a9f6148e0315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 12:56:11.440701 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:11.410355 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 12:56:11.440701 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:11.410381 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" podUID="189ebc59-a986-4e40-a9b1-25a350c5d397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 12:56:11.494752 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:11.494711 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:11.763010 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:11.762126 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-57fc876657-qh5lt"] Oct 31 12:56:11.766634 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:11.764124 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57fc876657-qh5lt" Oct 31 12:56:11.992482 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:11.992424 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/959d8796-94ca-4486-bea1-b011522535ad-provider-credentials\") pod \"payload-processing-external-model-57fc876657-qh5lt\" (UID: \"959d8796-94ca-4486-bea1-b011522535ad\") " pod="models-as-a-service/payload-processing-external-model-57fc876657-qh5lt" Oct 31 12:56:12.020060 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:11.992948 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/959d8796-94ca-4486-bea1-b011522535ad-routing-overlay-volume\") pod \"payload-processing-external-model-57fc876657-qh5lt\" (UID: \"959d8796-94ca-4486-bea1-b011522535ad\") " pod="models-as-a-service/payload-processing-external-model-57fc876657-qh5lt" Oct 31 12:56:12.073016 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:12.072383 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/959d8796-94ca-4486-bea1-b011522535ad-plugins-config-volume\") pod \"payload-processing-external-model-57fc876657-qh5lt\" (UID: \"959d8796-94ca-4486-bea1-b011522535ad\") " pod="models-as-a-service/payload-processing-external-model-57fc876657-qh5lt" Oct 31 12:56:12.073016 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:12.072533 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcphl\" (UniqueName: \"kubernetes.io/projected/959d8796-94ca-4486-bea1-b011522535ad-kube-api-access-pcphl\") pod \"payload-processing-external-model-57fc876657-qh5lt\" (UID: \"959d8796-94ca-4486-bea1-b011522535ad\") " pod="models-as-a-service/payload-processing-external-model-57fc876657-qh5lt" Oct 31 12:56:12.119007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:12.114129 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:56:12.119007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:12.114212 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" podUID="164b7cbb-c80d-4709-b5d0-5bdba8070ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:12.180118 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:12.176030 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/959d8796-94ca-4486-bea1-b011522535ad-routing-overlay-volume\") pod \"payload-processing-external-model-57fc876657-qh5lt\" (UID: \"959d8796-94ca-4486-bea1-b011522535ad\") " pod="models-as-a-service/payload-processing-external-model-57fc876657-qh5lt" Oct 31 12:56:12.180118 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:12.176106 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/959d8796-94ca-4486-bea1-b011522535ad-plugins-config-volume\") pod \"payload-processing-external-model-57fc876657-qh5lt\" (UID: \"959d8796-94ca-4486-bea1-b011522535ad\") " pod="models-as-a-service/payload-processing-external-model-57fc876657-qh5lt" Oct 31 12:56:12.180118 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:12.176204 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcphl\" (UniqueName: \"kubernetes.io/projected/959d8796-94ca-4486-bea1-b011522535ad-kube-api-access-pcphl\") pod \"payload-processing-external-model-57fc876657-qh5lt\" (UID: \"959d8796-94ca-4486-bea1-b011522535ad\") " pod="models-as-a-service/payload-processing-external-model-57fc876657-qh5lt" Oct 31 12:56:12.180118 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:12.176328 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/959d8796-94ca-4486-bea1-b011522535ad-provider-credentials\") pod \"payload-processing-external-model-57fc876657-qh5lt\" (UID: \"959d8796-94ca-4486-bea1-b011522535ad\") " pod="models-as-a-service/payload-processing-external-model-57fc876657-qh5lt" Oct 31 12:56:12.196286 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:12.196249 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 12:56:12.196521 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:12.196492 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" podUID="cb204311-3184-4407-bea7-a759f629ec3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 12:56:12.197061 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:12.196867 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/959d8796-94ca-4486-bea1-b011522535ad-plugins-config-volume\") pod \"payload-processing-external-model-57fc876657-qh5lt\" (UID: \"959d8796-94ca-4486-bea1-b011522535ad\") " pod="models-as-a-service/payload-processing-external-model-57fc876657-qh5lt" Oct 31 12:56:12.225038 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:12.212744 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/959d8796-94ca-4486-bea1-b011522535ad-provider-credentials\") pod \"payload-processing-external-model-57fc876657-qh5lt\" (UID: \"959d8796-94ca-4486-bea1-b011522535ad\") " pod="models-as-a-service/payload-processing-external-model-57fc876657-qh5lt" Oct 31 12:56:12.225282 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:12.219625 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/959d8796-94ca-4486-bea1-b011522535ad-routing-overlay-volume\") pod \"payload-processing-external-model-57fc876657-qh5lt\" (UID: \"959d8796-94ca-4486-bea1-b011522535ad\") " pod="models-as-a-service/payload-processing-external-model-57fc876657-qh5lt" Oct 31 12:56:12.258824 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:12.258787 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcphl\" (UniqueName: \"kubernetes.io/projected/959d8796-94ca-4486-bea1-b011522535ad-kube-api-access-pcphl\") pod \"payload-processing-external-model-57fc876657-qh5lt\" (UID: \"959d8796-94ca-4486-bea1-b011522535ad\") " pod="models-as-a-service/payload-processing-external-model-57fc876657-qh5lt" Oct 31 12:56:12.335436 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:12.335338 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 12:56:12.336157 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:12.336114 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" podUID="189ebc59-a986-4e40-a9b1-25a350c5d397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 12:56:12.369846 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:12.369810 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:12.400314 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:12.400269 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57fc876657-qh5lt" Oct 31 12:56:12.607071 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:12.557405 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 12:56:12.627242 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:12.627202 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" podUID="954541ff-08a9-4c5e-ab44-a9f6148e0315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 12:56:13.139627 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:13.139594 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-57fc876657-qh5lt"] Oct 31 12:56:13.145173 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:56:13.145111 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod959d8796_94ca_4486_bea1_b011522535ad.slice/crio-3de0863ea6e41d8de25565bf1c6d8322f8d4509152defba85cbcd7999b86c569 WatchSource:0}: Error finding container 3de0863ea6e41d8de25565bf1c6d8322f8d4509152defba85cbcd7999b86c569: Status 404 returned error can't find the container with id 3de0863ea6e41d8de25565bf1c6d8322f8d4509152defba85cbcd7999b86c569 Oct 31 12:56:13.199176 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:13.199034 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 12:56:13.199176 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:13.199103 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" podUID="cb204311-3184-4407-bea7-a759f629ec3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 12:56:13.352711 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:13.352611 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:13.392859 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:13.391304 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 12:56:13.392859 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:13.391365 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" podUID="189ebc59-a986-4e40-a9b1-25a350c5d397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 12:56:13.406603 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:13.406476 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 12:56:13.406603 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:13.406536 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" podUID="954541ff-08a9-4c5e-ab44-a9f6148e0315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 12:56:13.690007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:13.689031 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57fc876657-qh5lt" event={"ID":"959d8796-94ca-4486-bea1-b011522535ad","Type":"ContainerStarted","Data":"d2f79e800caf96eeb450f21f4d5c94f75fd278277b0d2eb09aa1b43f0221e7c8"} Oct 31 12:56:13.690007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:13.689119 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57fc876657-qh5lt" event={"ID":"959d8796-94ca-4486-bea1-b011522535ad","Type":"ContainerStarted","Data":"3de0863ea6e41d8de25565bf1c6d8322f8d4509152defba85cbcd7999b86c569"} Oct 31 12:56:13.705839 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:13.705797 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-57fc876657-qh5lt" Oct 31 12:56:13.729836 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:13.729727 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-57fc876657-qh5lt" podStartSLOduration=2.729712804 podStartE2EDuration="2.729712804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:56:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:56:13.728535565 +0000 UTC m=+1420.211658466" watchObservedRunningTime="2026-10-31 12:56:13.729712804 +0000 UTC m=+1420.212835704" Oct 31 12:56:13.834961 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:13.834926 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t"] Oct 31 12:56:13.845187 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:13.845157 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t"] Oct 31 12:56:13.845531 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:13.845465 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" Oct 31 12:56:13.939797 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:13.939770 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61c4f511-5495-4b14-ae0f-b019d826ae7d-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t\" (UID: \"61c4f511-5495-4b14-ae0f-b019d826ae7d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" Oct 31 12:56:13.940442 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:13.940026 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61c4f511-5495-4b14-ae0f-b019d826ae7d-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t\" (UID: \"61c4f511-5495-4b14-ae0f-b019d826ae7d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" Oct 31 12:56:13.940442 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:13.940093 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c4f511-5495-4b14-ae0f-b019d826ae7d-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t\" (UID: \"61c4f511-5495-4b14-ae0f-b019d826ae7d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" Oct 31 12:56:13.940442 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:13.940162 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61c4f511-5495-4b14-ae0f-b019d826ae7d-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t\" (UID: \"61c4f511-5495-4b14-ae0f-b019d826ae7d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" Oct 31 12:56:13.940442 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:13.940203 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6fgg\" (UniqueName: \"kubernetes.io/projected/61c4f511-5495-4b14-ae0f-b019d826ae7d-kube-api-access-s6fgg\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t\" (UID: \"61c4f511-5495-4b14-ae0f-b019d826ae7d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" Oct 31 12:56:13.940442 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:13.940238 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61c4f511-5495-4b14-ae0f-b019d826ae7d-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t\" (UID: \"61c4f511-5495-4b14-ae0f-b019d826ae7d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" Oct 31 12:56:13.940442 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:13.940268 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c4f511-5495-4b14-ae0f-b019d826ae7d-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t\" (UID: \"61c4f511-5495-4b14-ae0f-b019d826ae7d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" Oct 31 12:56:13.940442 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:13.940291 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61c4f511-5495-4b14-ae0f-b019d826ae7d-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t\" (UID: \"61c4f511-5495-4b14-ae0f-b019d826ae7d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" Oct 31 12:56:13.940442 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:13.940311 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61c4f511-5495-4b14-ae0f-b019d826ae7d-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t\" (UID: \"61c4f511-5495-4b14-ae0f-b019d826ae7d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" Oct 31 12:56:14.041838 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:14.041744 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c4f511-5495-4b14-ae0f-b019d826ae7d-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t\" (UID: \"61c4f511-5495-4b14-ae0f-b019d826ae7d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" Oct 31 12:56:14.042526 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:14.042502 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61c4f511-5495-4b14-ae0f-b019d826ae7d-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t\" (UID: \"61c4f511-5495-4b14-ae0f-b019d826ae7d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" Oct 31 12:56:14.043030 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:14.043008 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61c4f511-5495-4b14-ae0f-b019d826ae7d-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t\" (UID: \"61c4f511-5495-4b14-ae0f-b019d826ae7d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" Oct 31 12:56:14.043951 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:14.043929 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61c4f511-5495-4b14-ae0f-b019d826ae7d-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t\" (UID: \"61c4f511-5495-4b14-ae0f-b019d826ae7d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" Oct 31 12:56:14.044439 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:14.044419 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61c4f511-5495-4b14-ae0f-b019d826ae7d-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t\" (UID: \"61c4f511-5495-4b14-ae0f-b019d826ae7d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" Oct 31 12:56:14.044970 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:14.044942 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c4f511-5495-4b14-ae0f-b019d826ae7d-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t\" (UID: \"61c4f511-5495-4b14-ae0f-b019d826ae7d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" Oct 31 12:56:14.045514 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:14.045495 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61c4f511-5495-4b14-ae0f-b019d826ae7d-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t\" (UID: \"61c4f511-5495-4b14-ae0f-b019d826ae7d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" Oct 31 12:56:14.046336 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:14.046315 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6fgg\" (UniqueName: \"kubernetes.io/projected/61c4f511-5495-4b14-ae0f-b019d826ae7d-kube-api-access-s6fgg\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t\" (UID: \"61c4f511-5495-4b14-ae0f-b019d826ae7d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" Oct 31 12:56:14.046778 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:14.046758 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61c4f511-5495-4b14-ae0f-b019d826ae7d-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t\" (UID: \"61c4f511-5495-4b14-ae0f-b019d826ae7d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" Oct 31 12:56:14.047314 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:14.045362 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c4f511-5495-4b14-ae0f-b019d826ae7d-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t\" (UID: \"61c4f511-5495-4b14-ae0f-b019d826ae7d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" Oct 31 12:56:14.047456 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:14.042936 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61c4f511-5495-4b14-ae0f-b019d826ae7d-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t\" (UID: \"61c4f511-5495-4b14-ae0f-b019d826ae7d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" Oct 31 12:56:14.047564 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:14.042439 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c4f511-5495-4b14-ae0f-b019d826ae7d-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t\" (UID: \"61c4f511-5495-4b14-ae0f-b019d826ae7d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" Oct 31 12:56:14.047661 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:14.043783 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61c4f511-5495-4b14-ae0f-b019d826ae7d-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t\" (UID: \"61c4f511-5495-4b14-ae0f-b019d826ae7d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" Oct 31 12:56:14.047758 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:14.047158 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61c4f511-5495-4b14-ae0f-b019d826ae7d-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t\" (UID: \"61c4f511-5495-4b14-ae0f-b019d826ae7d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" Oct 31 12:56:14.050660 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:14.050616 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61c4f511-5495-4b14-ae0f-b019d826ae7d-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t\" (UID: \"61c4f511-5495-4b14-ae0f-b019d826ae7d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" Oct 31 12:56:14.083661 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:14.083589 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61c4f511-5495-4b14-ae0f-b019d826ae7d-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t\" (UID: \"61c4f511-5495-4b14-ae0f-b019d826ae7d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" Oct 31 12:56:14.107653 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:14.106947 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61c4f511-5495-4b14-ae0f-b019d826ae7d-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t\" (UID: \"61c4f511-5495-4b14-ae0f-b019d826ae7d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" Oct 31 12:56:14.107653 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:14.107589 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6fgg\" (UniqueName: \"kubernetes.io/projected/61c4f511-5495-4b14-ae0f-b019d826ae7d-kube-api-access-s6fgg\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t\" (UID: \"61c4f511-5495-4b14-ae0f-b019d826ae7d\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" Oct 31 12:56:14.229862 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:14.229827 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" Oct 31 12:56:14.240387 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:14.240349 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:14.374059 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:14.373108 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 12:56:14.374059 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:14.373167 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" podUID="189ebc59-a986-4e40-a9b1-25a350c5d397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 12:56:14.408007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:14.407760 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 12:56:14.408007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:14.407821 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" podUID="954541ff-08a9-4c5e-ab44-a9f6148e0315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 12:56:14.463171 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:14.453962 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:14.877061 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:14.873517 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t"] Oct 31 12:56:14.923267 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:56:14.923227 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61c4f511_5495_4b14_ae0f_b019d826ae7d.slice/crio-f3f6eb6b6390e5279d9b6406bc11718642d48378a098cb62a6d00bbf38c21e64 WatchSource:0}: Error finding container f3f6eb6b6390e5279d9b6406bc11718642d48378a098cb62a6d00bbf38c21e64: Status 404 returned error can't find the container with id f3f6eb6b6390e5279d9b6406bc11718642d48378a098cb62a6d00bbf38c21e64 Oct 31 12:56:14.926835 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:14.926790 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:56:14.927043 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:14.926892 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:56:14.927043 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:14.926939 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:56:15.126104 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:15.126070 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:15.250005 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:15.242163 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:56:15.250005 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:15.242217 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" podUID="cb204311-3184-4407-bea7-a759f629ec3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:15.333812 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:15.333760 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 12:56:15.334019 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:15.333824 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" podUID="189ebc59-a986-4e40-a9b1-25a350c5d397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 12:56:15.414064 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:15.413384 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 12:56:15.414064 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:15.413445 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" podUID="954541ff-08a9-4c5e-ab44-a9f6148e0315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 12:56:15.777008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:15.772878 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" event={"ID":"61c4f511-5495-4b14-ae0f-b019d826ae7d","Type":"ContainerStarted","Data":"025092551b0a473f1f1640e39ca889d10f706727677fd5a0d827846cf3f7c547"} Oct 31 12:56:15.786514 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:15.786467 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" event={"ID":"61c4f511-5495-4b14-ae0f-b019d826ae7d","Type":"ContainerStarted","Data":"f3f6eb6b6390e5279d9b6406bc11718642d48378a098cb62a6d00bbf38c21e64"} Oct 31 12:56:15.829032 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:15.828072 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" podStartSLOduration=2.828049857 podStartE2EDuration="2.828049857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:56:15.824402482 +0000 UTC m=+1422.307525396" watchObservedRunningTime="2026-10-31 12:56:15.828049857 +0000 UTC m=+1422.311172759" Oct 31 12:56:15.928404 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:15.928365 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" Oct 31 12:56:15.930278 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:15.930138 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" Oct 31 12:56:15.960400 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:15.960370 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" Oct 31 12:56:16.092325 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:16.078702 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:16.230582 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:16.230551 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" Oct 31 12:56:16.231752 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:16.231728 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 12:56:16.231957 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:16.231936 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" podUID="61c4f511-5495-4b14-ae0f-b019d826ae7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 12:56:16.334239 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:16.334206 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 12:56:16.334386 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:16.334262 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" podUID="189ebc59-a986-4e40-a9b1-25a350c5d397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 12:56:16.407395 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:16.407309 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 12:56:16.407395 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:16.407386 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" podUID="954541ff-08a9-4c5e-ab44-a9f6148e0315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 12:56:17.026012 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:17.018820 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:17.231188 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:17.231142 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 12:56:17.231369 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:17.231212 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" podUID="61c4f511-5495-4b14-ae0f-b019d826ae7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 12:56:17.334584 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:17.334182 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 12:56:17.334584 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:17.334240 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" podUID="189ebc59-a986-4e40-a9b1-25a350c5d397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 12:56:17.406377 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:17.406338 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 12:56:17.406582 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:17.406386 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" podUID="954541ff-08a9-4c5e-ab44-a9f6148e0315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 12:56:18.049583 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.049544 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:18.096008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.095022 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x"] Oct 31 12:56:18.113231 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.113194 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x"] Oct 31 12:56:18.113422 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.113328 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" Oct 31 12:56:18.230524 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.230491 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 12:56:18.230713 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.230544 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" podUID="61c4f511-5495-4b14-ae0f-b019d826ae7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 12:56:18.292773 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.292724 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-istio-podinfo\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x\" (UID: \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" Oct 31 12:56:18.292964 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.292800 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trqtq\" (UniqueName: \"kubernetes.io/projected/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-kube-api-access-trqtq\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x\" (UID: \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" Oct 31 12:56:18.292964 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.292865 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-workload-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x\" (UID: \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" Oct 31 12:56:18.292964 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.292936 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-istiod-ca-cert\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x\" (UID: \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" Oct 31 12:56:18.293100 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.292994 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-istio-token\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x\" (UID: \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" Oct 31 12:56:18.293100 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.293044 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-workload-certs\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x\" (UID: \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" Oct 31 12:56:18.293179 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.293103 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-istio-envoy\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x\" (UID: \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" Oct 31 12:56:18.293224 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.293208 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-istio-data\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x\" (UID: \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" Oct 31 12:56:18.293275 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.293258 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-credential-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x\" (UID: \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" Oct 31 12:56:18.394794 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.394692 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-credential-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x\" (UID: \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" Oct 31 12:56:18.394969 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.394811 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-istio-podinfo\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x\" (UID: \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" Oct 31 12:56:18.394969 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.394838 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trqtq\" (UniqueName: \"kubernetes.io/projected/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-kube-api-access-trqtq\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x\" (UID: \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" Oct 31 12:56:18.394969 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.394892 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-workload-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x\" (UID: \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" Oct 31 12:56:18.394969 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.394956 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-istiod-ca-cert\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x\" (UID: \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" Oct 31 12:56:18.395237 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.395011 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-istio-token\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x\" (UID: \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" Oct 31 12:56:18.395237 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.395060 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-workload-certs\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x\" (UID: \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" Oct 31 12:56:18.395237 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.395119 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-istio-envoy\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x\" (UID: \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" Oct 31 12:56:18.395237 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.395186 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-credential-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x\" (UID: \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" Oct 31 12:56:18.395237 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.395220 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-istio-data\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x\" (UID: \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" Oct 31 12:56:18.395671 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.395645 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-istio-data\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x\" (UID: \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" Oct 31 12:56:18.396053 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.396027 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-workload-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x\" (UID: \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" Oct 31 12:56:18.396171 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.396152 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-istiod-ca-cert\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x\" (UID: \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" Oct 31 12:56:18.396271 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.396172 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-workload-certs\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x\" (UID: \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" Oct 31 12:56:18.398768 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.398739 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-istio-podinfo\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x\" (UID: \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" Oct 31 12:56:18.399242 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.399217 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-istio-envoy\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x\" (UID: \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" Oct 31 12:56:18.406624 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.406595 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 12:56:18.406725 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.406647 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" podUID="954541ff-08a9-4c5e-ab44-a9f6148e0315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 12:56:18.408832 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.408783 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-istio-token\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x\" (UID: \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" Oct 31 12:56:18.409708 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.409684 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trqtq\" (UniqueName: \"kubernetes.io/projected/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-kube-api-access-trqtq\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x\" (UID: \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" Oct 31 12:56:18.436705 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.436494 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" Oct 31 12:56:18.813733 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.813664 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x"] Oct 31 12:56:18.886136 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.860070 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:18.958047 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.957967 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:56:18.958204 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.958162 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:56:18.958290 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.958262 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:56:18.987612 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:18.987347 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" event={"ID":"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec","Type":"ContainerStarted","Data":"9764160cf47e4c545e8a35672d118cf5d506052711a4c0d21466b99e1c81d124"} Oct 31 12:56:19.067092 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:19.064315 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" Oct 31 12:56:19.082096 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:19.075431 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" Oct 31 12:56:19.082096 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:19.079434 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" Oct 31 12:56:19.232195 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:19.232154 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 12:56:19.232378 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:19.232213 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" podUID="61c4f511-5495-4b14-ae0f-b019d826ae7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 12:56:19.406679 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:19.406605 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 12:56:19.406679 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:19.406667 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" podUID="954541ff-08a9-4c5e-ab44-a9f6148e0315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 12:56:19.734411 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:19.734382 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:20.024630 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:20.024537 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" event={"ID":"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec","Type":"ContainerStarted","Data":"5fde765dd6300f6c8058f87b1315c8ad4bea25894bfd978e266363ada660fd71"} Oct 31 12:56:20.073007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:20.063663 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" podStartSLOduration=2.063644265 podStartE2EDuration="2.063644265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:56:20.056658002 +0000 UTC m=+1426.539780904" watchObservedRunningTime="2026-10-31 12:56:20.063644265 +0000 UTC m=+1426.546767167" Oct 31 12:56:20.088261 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:56:20.088218 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:56:20.231066 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:20.231034 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 12:56:20.231231 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:20.231092 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" podUID="61c4f511-5495-4b14-ae0f-b019d826ae7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 12:56:20.411666 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:20.411581 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" Oct 31 12:56:20.411970 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:20.411944 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" Oct 31 12:56:20.412699 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:20.412678 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" Oct 31 12:56:20.436671 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:20.436606 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" Oct 31 12:56:20.438343 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:20.438317 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 12:56:20.438474 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:20.438376 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" podUID="2b7734fb-74f8-4eed-bd5f-415d1c6e51ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 12:56:20.663063 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:20.662155 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:21.230304 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:21.230266 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 12:56:21.230733 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:21.230316 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" podUID="61c4f511-5495-4b14-ae0f-b019d826ae7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 12:56:21.438870 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:21.438388 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 12:56:21.438870 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:21.438449 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" podUID="2b7734fb-74f8-4eed-bd5f-415d1c6e51ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 12:56:21.918577 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:21.918532 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:22.230577 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:22.230537 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 12:56:22.231254 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:22.231219 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" podUID="61c4f511-5495-4b14-ae0f-b019d826ae7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 12:56:22.440493 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:22.440459 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 12:56:22.440822 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:22.440774 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" podUID="2b7734fb-74f8-4eed-bd5f-415d1c6e51ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 12:56:23.338909 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:23.309258 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 12:56:23.338909 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:23.309333 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" podUID="61c4f511-5495-4b14-ae0f-b019d826ae7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 12:56:23.433294 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:23.366041 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cb6bcb954-xjt8m"] Oct 31 12:56:23.434663 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:23.402504 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:23.502403 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:23.502358 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 12:56:23.502683 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:23.502636 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" podUID="2b7734fb-74f8-4eed-bd5f-415d1c6e51ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 12:56:23.734063 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:23.723400 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:24.056061 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:24.031559 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-57fc876657-qh5lt"] Oct 31 12:56:24.163319 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:24.163283 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-57fc876657-qh5lt" Oct 31 12:56:24.233279 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:24.232728 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 12:56:24.233279 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:24.232804 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" podUID="61c4f511-5495-4b14-ae0f-b019d826ae7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 12:56:24.444070 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:24.443197 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 12:56:24.444070 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:24.443254 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" podUID="2b7734fb-74f8-4eed-bd5f-415d1c6e51ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 12:56:24.938032 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:24.937921 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:25.247008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:25.244416 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 12:56:25.247008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:25.244481 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" podUID="61c4f511-5495-4b14-ae0f-b019d826ae7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 12:56:25.441017 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:25.438652 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 12:56:25.441017 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:25.438712 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" podUID="2b7734fb-74f8-4eed-bd5f-415d1c6e51ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 12:56:25.918106 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:25.918067 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:26.319028 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:26.318975 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" Oct 31 12:56:26.319843 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:26.319799 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" Oct 31 12:56:26.320526 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:26.320499 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" Oct 31 12:56:26.437434 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:26.437397 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 12:56:26.437622 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:26.437501 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" podUID="2b7734fb-74f8-4eed-bd5f-415d1c6e51ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 12:56:27.011371 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:27.010543 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:27.437044 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:27.436955 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 12:56:27.437198 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:27.437041 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" podUID="2b7734fb-74f8-4eed-bd5f-415d1c6e51ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 12:56:28.017389 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:28.017349 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:28.435422 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:28.435334 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6cb6bcb954-xjt8m" podUID="19906921-8b1a-48ef-a75e-391211820c68" containerName="payload-processing" containerID="cri-o://e0181a640ddf2ddb2c08b14afb19f86a37571f172485656a838ded694c304f7f" gracePeriod=25 Oct 31 12:56:28.437787 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:28.437751 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 12:56:28.437922 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:28.437811 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" podUID="2b7734fb-74f8-4eed-bd5f-415d1c6e51ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 12:56:28.716282 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:28.716256 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cb6bcb954-xjt8m" Oct 31 12:56:28.815813 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:28.815773 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19906921-8b1a-48ef-a75e-391211820c68-kube-api-access-v7rxk\" (UniqueName: \"kubernetes.io/projected/19906921-8b1a-48ef-a75e-391211820c68-kube-api-access-v7rxk\") pod \"19906921-8b1a-48ef-a75e-391211820c68\" (UID: \"19906921-8b1a-48ef-a75e-391211820c68\") " Oct 31 12:56:28.816022 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:28.815880 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19906921-8b1a-48ef-a75e-391211820c68-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19906921-8b1a-48ef-a75e-391211820c68-routing-overlay-volume\") pod \"19906921-8b1a-48ef-a75e-391211820c68\" (UID: \"19906921-8b1a-48ef-a75e-391211820c68\") " Oct 31 12:56:28.816022 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:28.815937 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19906921-8b1a-48ef-a75e-391211820c68-provider-credentials\" (UniqueName: \"kubernetes.io/projected/19906921-8b1a-48ef-a75e-391211820c68-provider-credentials\") pod \"19906921-8b1a-48ef-a75e-391211820c68\" (UID: \"19906921-8b1a-48ef-a75e-391211820c68\") " Oct 31 12:56:28.816736 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:28.816192 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19906921-8b1a-48ef-a75e-391211820c68-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19906921-8b1a-48ef-a75e-391211820c68-plugins-config-volume\") pod \"19906921-8b1a-48ef-a75e-391211820c68\" (UID: \"19906921-8b1a-48ef-a75e-391211820c68\") " Oct 31 12:56:28.816736 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:28.816508 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19906921-8b1a-48ef-a75e-391211820c68-routing-overlay-volume" pod "19906921-8b1a-48ef-a75e-391211820c68" (UID: "19906921-8b1a-48ef-a75e-391211820c68"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:56:28.816955 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:28.816925 6841 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19906921-8b1a-48ef-a75e-391211820c68-routing-overlay-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:28.817270 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:28.817240 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19906921-8b1a-48ef-a75e-391211820c68-plugins-config-volume" pod "19906921-8b1a-48ef-a75e-391211820c68" (UID: "19906921-8b1a-48ef-a75e-391211820c68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:56:28.818507 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:28.818468 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19906921-8b1a-48ef-a75e-391211820c68-kube-api-access-v7rxk" pod "19906921-8b1a-48ef-a75e-391211820c68" (UID: "19906921-8b1a-48ef-a75e-391211820c68"). InnerVolumeSpecName "kube-api-access-v7rxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:56:28.818939 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:28.818911 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19906921-8b1a-48ef-a75e-391211820c68-provider-credentials" pod "19906921-8b1a-48ef-a75e-391211820c68" (UID: "19906921-8b1a-48ef-a75e-391211820c68"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:56:28.918586 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:28.918547 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7rxk\" (UniqueName: \"kubernetes.io/projected/19906921-8b1a-48ef-a75e-391211820c68-kube-api-access-v7rxk\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:28.918586 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:28.918574 6841 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19906921-8b1a-48ef-a75e-391211820c68-provider-credentials\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:28.918586 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:28.918586 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19906921-8b1a-48ef-a75e-391211820c68-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:29.031072 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:29.030741 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:29.063913 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:29.063858 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-57fc876657-qh5lt" podUID="959d8796-94ca-4486-bea1-b011522535ad" containerName="payload-processing" containerID="cri-o://d2f79e800caf96eeb450f21f4d5c94f75fd278277b0d2eb09aa1b43f0221e7c8" gracePeriod=25 Oct 31 12:56:29.333720 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:29.333694 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57fc876657-qh5lt" Oct 31 12:56:29.424515 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:29.424479 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/959d8796-94ca-4486-bea1-b011522535ad-kube-api-access-pcphl\" (UniqueName: \"kubernetes.io/projected/959d8796-94ca-4486-bea1-b011522535ad-kube-api-access-pcphl\") pod \"959d8796-94ca-4486-bea1-b011522535ad\" (UID: \"959d8796-94ca-4486-bea1-b011522535ad\") " Oct 31 12:56:29.424684 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:29.424607 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/959d8796-94ca-4486-bea1-b011522535ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/959d8796-94ca-4486-bea1-b011522535ad-plugins-config-volume\") pod \"959d8796-94ca-4486-bea1-b011522535ad\" (UID: \"959d8796-94ca-4486-bea1-b011522535ad\") " Oct 31 12:56:29.424684 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:29.424652 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/959d8796-94ca-4486-bea1-b011522535ad-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/959d8796-94ca-4486-bea1-b011522535ad-routing-overlay-volume\") pod \"959d8796-94ca-4486-bea1-b011522535ad\" (UID: \"959d8796-94ca-4486-bea1-b011522535ad\") " Oct 31 12:56:29.424803 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:29.424786 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/959d8796-94ca-4486-bea1-b011522535ad-provider-credentials\" (UniqueName: \"kubernetes.io/projected/959d8796-94ca-4486-bea1-b011522535ad-provider-credentials\") pod \"959d8796-94ca-4486-bea1-b011522535ad\" (UID: \"959d8796-94ca-4486-bea1-b011522535ad\") " Oct 31 12:56:29.425147 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:29.425119 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/959d8796-94ca-4486-bea1-b011522535ad-plugins-config-volume" pod "959d8796-94ca-4486-bea1-b011522535ad" (UID: "959d8796-94ca-4486-bea1-b011522535ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:56:29.425304 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:29.425144 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/959d8796-94ca-4486-bea1-b011522535ad-routing-overlay-volume" pod "959d8796-94ca-4486-bea1-b011522535ad" (UID: "959d8796-94ca-4486-bea1-b011522535ad"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:56:29.426728 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:29.426704 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/959d8796-94ca-4486-bea1-b011522535ad-kube-api-access-pcphl" pod "959d8796-94ca-4486-bea1-b011522535ad" (UID: "959d8796-94ca-4486-bea1-b011522535ad"). InnerVolumeSpecName "kube-api-access-pcphl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:56:29.426812 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:29.426782 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/959d8796-94ca-4486-bea1-b011522535ad-provider-credentials" pod "959d8796-94ca-4486-bea1-b011522535ad" (UID: "959d8796-94ca-4486-bea1-b011522535ad"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:56:29.493382 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:29.493357 6841 generic.go:356] "Generic (PLEG): container finished" podID="19906921-8b1a-48ef-a75e-391211820c68" containerID="e0181a640ddf2ddb2c08b14afb19f86a37571f172485656a838ded694c304f7f" exitCode=0 Oct 31 12:56:29.493522 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:29.493448 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cb6bcb954-xjt8m" Oct 31 12:56:29.493595 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:29.493447 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cb6bcb954-xjt8m" event={"ID":"19906921-8b1a-48ef-a75e-391211820c68","Type":"ContainerDied","Data":"e0181a640ddf2ddb2c08b14afb19f86a37571f172485656a838ded694c304f7f"} Oct 31 12:56:29.493667 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:29.493601 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cb6bcb954-xjt8m" event={"ID":"19906921-8b1a-48ef-a75e-391211820c68","Type":"ContainerDied","Data":"91bcbac138b3a650f12ef34dd3ed66796384b8e0fec2d491439b872af52eede6"} Oct 31 12:56:29.493667 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:29.493633 6841 scope.go:122] "RemoveContainer" containerID="e0181a640ddf2ddb2c08b14afb19f86a37571f172485656a838ded694c304f7f" Oct 31 12:56:29.494972 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:29.494930 6841 generic.go:356] "Generic (PLEG): container finished" podID="959d8796-94ca-4486-bea1-b011522535ad" containerID="d2f79e800caf96eeb450f21f4d5c94f75fd278277b0d2eb09aa1b43f0221e7c8" exitCode=0 Oct 31 12:56:29.495076 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:29.494999 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57fc876657-qh5lt" Oct 31 12:56:29.495076 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:29.495042 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57fc876657-qh5lt" event={"ID":"959d8796-94ca-4486-bea1-b011522535ad","Type":"ContainerDied","Data":"d2f79e800caf96eeb450f21f4d5c94f75fd278277b0d2eb09aa1b43f0221e7c8"} Oct 31 12:56:29.495235 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:29.495087 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57fc876657-qh5lt" event={"ID":"959d8796-94ca-4486-bea1-b011522535ad","Type":"ContainerDied","Data":"3de0863ea6e41d8de25565bf1c6d8322f8d4509152defba85cbcd7999b86c569"} Oct 31 12:56:29.505508 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:29.505493 6841 scope.go:122] "RemoveContainer" containerID="e0181a640ddf2ddb2c08b14afb19f86a37571f172485656a838ded694c304f7f" Oct 31 12:56:29.505881 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:56:29.505769 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0181a640ddf2ddb2c08b14afb19f86a37571f172485656a838ded694c304f7f\": container with ID starting with e0181a640ddf2ddb2c08b14afb19f86a37571f172485656a838ded694c304f7f not found: ID does not exist" containerID="e0181a640ddf2ddb2c08b14afb19f86a37571f172485656a838ded694c304f7f" Oct 31 12:56:29.505881 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:29.505805 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0181a640ddf2ddb2c08b14afb19f86a37571f172485656a838ded694c304f7f"} err="failed to get container status \"e0181a640ddf2ddb2c08b14afb19f86a37571f172485656a838ded694c304f7f\": rpc error: code = NotFound desc = could not find container \"e0181a640ddf2ddb2c08b14afb19f86a37571f172485656a838ded694c304f7f\": container with ID starting with e0181a640ddf2ddb2c08b14afb19f86a37571f172485656a838ded694c304f7f not found: ID does not exist" Oct 31 12:56:29.505881 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:29.505829 6841 scope.go:122] "RemoveContainer" containerID="d2f79e800caf96eeb450f21f4d5c94f75fd278277b0d2eb09aa1b43f0221e7c8" Oct 31 12:56:29.515621 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:29.515599 6841 scope.go:122] "RemoveContainer" containerID="d2f79e800caf96eeb450f21f4d5c94f75fd278277b0d2eb09aa1b43f0221e7c8" Oct 31 12:56:29.515890 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:56:29.515870 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2f79e800caf96eeb450f21f4d5c94f75fd278277b0d2eb09aa1b43f0221e7c8\": container with ID starting with d2f79e800caf96eeb450f21f4d5c94f75fd278277b0d2eb09aa1b43f0221e7c8 not found: ID does not exist" containerID="d2f79e800caf96eeb450f21f4d5c94f75fd278277b0d2eb09aa1b43f0221e7c8" Oct 31 12:56:29.515958 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:29.515902 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2f79e800caf96eeb450f21f4d5c94f75fd278277b0d2eb09aa1b43f0221e7c8"} err="failed to get container status \"d2f79e800caf96eeb450f21f4d5c94f75fd278277b0d2eb09aa1b43f0221e7c8\": rpc error: code = NotFound desc = could not find container \"d2f79e800caf96eeb450f21f4d5c94f75fd278277b0d2eb09aa1b43f0221e7c8\": container with ID starting with d2f79e800caf96eeb450f21f4d5c94f75fd278277b0d2eb09aa1b43f0221e7c8 not found: ID does not exist" Oct 31 12:56:29.525163 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:29.525139 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cb6bcb954-xjt8m"] Oct 31 12:56:29.527089 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:29.527064 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/959d8796-94ca-4486-bea1-b011522535ad-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:29.527151 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:29.527097 6841 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/959d8796-94ca-4486-bea1-b011522535ad-routing-overlay-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:29.527151 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:29.527114 6841 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/959d8796-94ca-4486-bea1-b011522535ad-provider-credentials\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:29.527151 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:29.527128 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcphl\" (UniqueName: \"kubernetes.io/projected/959d8796-94ca-4486-bea1-b011522535ad-kube-api-access-pcphl\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:29.537757 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:29.537737 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cb6bcb954-xjt8m"] Oct 31 12:56:29.548924 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:29.548899 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-57fc876657-qh5lt"] Oct 31 12:56:29.553227 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:29.553207 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-57fc876657-qh5lt"] Oct 31 12:56:29.644343 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:29.644269 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19906921-8b1a-48ef-a75e-391211820c68" path="/var/lib/kubelet/pods/19906921-8b1a-48ef-a75e-391211820c68/volumes" Oct 31 12:56:29.644668 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:29.644655 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="959d8796-94ca-4486-bea1-b011522535ad" path="/var/lib/kubelet/pods/959d8796-94ca-4486-bea1-b011522535ad/volumes" Oct 31 12:56:29.840955 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:29.840260 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:30.395862 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:30.395827 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" Oct 31 12:56:30.396604 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:30.396535 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" Oct 31 12:56:30.397500 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:30.397468 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" Oct 31 12:56:30.722805 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:30.722523 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:31.421952 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:31.421919 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:32.132766 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:32.132730 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:32.825146 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:32.825108 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:33.524163 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:33.524125 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:33.940008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:33.939375 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:34.362162 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:34.362120 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:35.220433 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:35.220397 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:36.117715 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:36.117680 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:36.371234 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:36.371153 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:37.024931 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:37.024895 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:38.290008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:38.287302 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:41.669362 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:41.669282 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:41.878422 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:41.878367 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:41.904234 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:41.891489 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-6lxfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:56:41.904545 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:41.904497 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" podUID="d0ad44f3-de70-4b31-8d39-ceeed24efd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:42.087139 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:42.087101 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fk2tx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:56:42.087456 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:42.087423 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" podUID="aa7e0a48-9dc5-4735-b1ae-cafd28ff982f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:42.095052 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:42.095012 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:56:42.095314 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:42.095285 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" podUID="164b7cbb-c80d-4709-b5d0-5bdba8070ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:42.467390 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:42.464526 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:56:42.467390 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:42.464651 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" podUID="61c4f511-5495-4b14-ae0f-b019d826ae7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:43.432098 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:43.431602 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-ssz2q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:56:43.432098 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:43.431681 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" podUID="47a4ec22-fe77-4a7b-a66a-5debadbf5800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:43.503412 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:43.503319 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:45.211922 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:45.211852 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:45.905618 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:45.905530 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:46.625001 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:46.624048 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:47.414317 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:47.414280 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:47.981562 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:47.981523 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh"] Oct 31 12:56:47.982333 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:47.982303 6841 cpu_manager.go:414] "containerMap: removing container" podUID="19906921-8b1a-48ef-a75e-391211820c68" containerName="payload-processing" Oct 31 12:56:47.982333 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:47.982324 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19906921-8b1a-48ef-a75e-391211820c68" containerName="payload-processing" Oct 31 12:56:47.982502 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:47.982350 6841 cpu_manager.go:414] "containerMap: removing container" podUID="959d8796-94ca-4486-bea1-b011522535ad" containerName="payload-processing" Oct 31 12:56:47.982502 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:47.982358 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="959d8796-94ca-4486-bea1-b011522535ad" containerName="payload-processing" Oct 31 12:56:47.982600 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:47.982519 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="19906921-8b1a-48ef-a75e-391211820c68" containerName="payload-processing" Oct 31 12:56:47.982600 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:47.982535 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="959d8796-94ca-4486-bea1-b011522535ad" containerName="payload-processing" Oct 31 12:56:47.988434 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:47.988405 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" Oct 31 12:56:48.001209 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:47.999210 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d9c49d7-4979-4d5c-b94e-d477215dd938-workload-certs\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh\" (UID: \"1d9c49d7-4979-4d5c-b94e-d477215dd938\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" Oct 31 12:56:48.001209 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:47.999254 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d9c49d7-4979-4d5c-b94e-d477215dd938-istio-data\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh\" (UID: \"1d9c49d7-4979-4d5c-b94e-d477215dd938\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" Oct 31 12:56:48.001209 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:47.999289 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d9c49d7-4979-4d5c-b94e-d477215dd938-istio-token\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh\" (UID: \"1d9c49d7-4979-4d5c-b94e-d477215dd938\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" Oct 31 12:56:48.001209 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:47.999313 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9c49d7-4979-4d5c-b94e-d477215dd938-workload-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh\" (UID: \"1d9c49d7-4979-4d5c-b94e-d477215dd938\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" Oct 31 12:56:48.001209 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:47.999362 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d9c49d7-4979-4d5c-b94e-d477215dd938-istio-podinfo\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh\" (UID: \"1d9c49d7-4979-4d5c-b94e-d477215dd938\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" Oct 31 12:56:48.001209 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:47.999452 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ggqf\" (UniqueName: \"kubernetes.io/projected/1d9c49d7-4979-4d5c-b94e-d477215dd938-kube-api-access-9ggqf\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh\" (UID: \"1d9c49d7-4979-4d5c-b94e-d477215dd938\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" Oct 31 12:56:48.001209 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:47.999484 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d9c49d7-4979-4d5c-b94e-d477215dd938-istio-envoy\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh\" (UID: \"1d9c49d7-4979-4d5c-b94e-d477215dd938\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" Oct 31 12:56:48.001209 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:47.999526 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d9c49d7-4979-4d5c-b94e-d477215dd938-istiod-ca-cert\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh\" (UID: \"1d9c49d7-4979-4d5c-b94e-d477215dd938\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" Oct 31 12:56:48.001209 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:47.999615 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9c49d7-4979-4d5c-b94e-d477215dd938-credential-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh\" (UID: \"1d9c49d7-4979-4d5c-b94e-d477215dd938\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" Oct 31 12:56:48.006974 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:48.006937 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh"] Oct 31 12:56:48.100345 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:48.100308 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d9c49d7-4979-4d5c-b94e-d477215dd938-workload-certs\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh\" (UID: \"1d9c49d7-4979-4d5c-b94e-d477215dd938\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" Oct 31 12:56:48.100345 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:48.100348 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d9c49d7-4979-4d5c-b94e-d477215dd938-istio-data\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh\" (UID: \"1d9c49d7-4979-4d5c-b94e-d477215dd938\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" Oct 31 12:56:48.100598 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:48.100370 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d9c49d7-4979-4d5c-b94e-d477215dd938-istio-token\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh\" (UID: \"1d9c49d7-4979-4d5c-b94e-d477215dd938\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" Oct 31 12:56:48.100598 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:48.100386 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9c49d7-4979-4d5c-b94e-d477215dd938-workload-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh\" (UID: \"1d9c49d7-4979-4d5c-b94e-d477215dd938\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" Oct 31 12:56:48.100598 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:48.100427 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d9c49d7-4979-4d5c-b94e-d477215dd938-istio-podinfo\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh\" (UID: \"1d9c49d7-4979-4d5c-b94e-d477215dd938\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" Oct 31 12:56:48.100598 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:48.100501 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ggqf\" (UniqueName: \"kubernetes.io/projected/1d9c49d7-4979-4d5c-b94e-d477215dd938-kube-api-access-9ggqf\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh\" (UID: \"1d9c49d7-4979-4d5c-b94e-d477215dd938\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" Oct 31 12:56:48.100598 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:48.100536 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d9c49d7-4979-4d5c-b94e-d477215dd938-istio-envoy\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh\" (UID: \"1d9c49d7-4979-4d5c-b94e-d477215dd938\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" Oct 31 12:56:48.100598 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:48.100566 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d9c49d7-4979-4d5c-b94e-d477215dd938-istiod-ca-cert\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh\" (UID: \"1d9c49d7-4979-4d5c-b94e-d477215dd938\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" Oct 31 12:56:48.100886 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:48.100651 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9c49d7-4979-4d5c-b94e-d477215dd938-credential-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh\" (UID: \"1d9c49d7-4979-4d5c-b94e-d477215dd938\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" Oct 31 12:56:48.100886 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:48.100765 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d9c49d7-4979-4d5c-b94e-d477215dd938-workload-certs\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh\" (UID: \"1d9c49d7-4979-4d5c-b94e-d477215dd938\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" Oct 31 12:56:48.100886 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:48.100827 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d9c49d7-4979-4d5c-b94e-d477215dd938-istio-data\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh\" (UID: \"1d9c49d7-4979-4d5c-b94e-d477215dd938\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" Oct 31 12:56:48.101010 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:48.100886 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9c49d7-4979-4d5c-b94e-d477215dd938-workload-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh\" (UID: \"1d9c49d7-4979-4d5c-b94e-d477215dd938\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" Oct 31 12:56:48.101090 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:48.101072 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9c49d7-4979-4d5c-b94e-d477215dd938-credential-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh\" (UID: \"1d9c49d7-4979-4d5c-b94e-d477215dd938\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" Oct 31 12:56:48.101295 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:48.101256 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d9c49d7-4979-4d5c-b94e-d477215dd938-istiod-ca-cert\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh\" (UID: \"1d9c49d7-4979-4d5c-b94e-d477215dd938\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" Oct 31 12:56:48.103094 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:48.103068 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d9c49d7-4979-4d5c-b94e-d477215dd938-istio-podinfo\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh\" (UID: \"1d9c49d7-4979-4d5c-b94e-d477215dd938\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" Oct 31 12:56:48.103372 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:48.103345 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d9c49d7-4979-4d5c-b94e-d477215dd938-istio-envoy\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh\" (UID: \"1d9c49d7-4979-4d5c-b94e-d477215dd938\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" Oct 31 12:56:48.112638 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:48.112614 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d9c49d7-4979-4d5c-b94e-d477215dd938-istio-token\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh\" (UID: \"1d9c49d7-4979-4d5c-b94e-d477215dd938\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" Oct 31 12:56:48.112733 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:48.112654 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ggqf\" (UniqueName: \"kubernetes.io/projected/1d9c49d7-4979-4d5c-b94e-d477215dd938-kube-api-access-9ggqf\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh\" (UID: \"1d9c49d7-4979-4d5c-b94e-d477215dd938\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" Oct 31 12:56:48.235069 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:48.234766 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:48.308659 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:48.308629 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" Oct 31 12:56:48.594389 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:48.594356 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh"] Oct 31 12:56:48.595415 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:56:48.595385 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d9c49d7_4979_4d5c_b94e_d477215dd938.slice/crio-753e5965c1bad3a5a337dc59137fff4417d033e5bdd9346462628d4351997f62 WatchSource:0}: Error finding container 753e5965c1bad3a5a337dc59137fff4417d033e5bdd9346462628d4351997f62: Status 404 returned error can't find the container with id 753e5965c1bad3a5a337dc59137fff4417d033e5bdd9346462628d4351997f62 Oct 31 12:56:48.597773 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:48.597740 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:56:48.597842 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:48.597807 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:56:48.597842 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:48.597836 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:56:49.133754 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:49.133718 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:49.227072 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:49.226976 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" event={"ID":"1d9c49d7-4979-4d5c-b94e-d477215dd938","Type":"ContainerStarted","Data":"892f364691264adda080aa0dba36fe773fbd1c42c7a07167ecf3f3b675cbd2c7"} Oct 31 12:56:49.227237 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:49.227118 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" event={"ID":"1d9c49d7-4979-4d5c-b94e-d477215dd938","Type":"ContainerStarted","Data":"753e5965c1bad3a5a337dc59137fff4417d033e5bdd9346462628d4351997f62"} Oct 31 12:56:49.266339 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:49.265855 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" podStartSLOduration=2.265831192 podStartE2EDuration="2.265831192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:56:49.251637829 +0000 UTC m=+1455.734760731" watchObservedRunningTime="2026-10-31 12:56:49.265831192 +0000 UTC m=+1455.748954093" Oct 31 12:56:49.308818 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:49.308788 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" Oct 31 12:56:49.310464 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:49.310432 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 12:56:49.310610 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:49.310504 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" podUID="1d9c49d7-4979-4d5c-b94e-d477215dd938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 12:56:50.041031 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:50.040967 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:50.331054 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:56:50.324589 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:56:50.331054 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:50.325110 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 12:56:50.331054 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:50.325160 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" podUID="1d9c49d7-4979-4d5c-b94e-d477215dd938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 12:56:50.414008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:50.402639 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:50.923338 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:50.905203 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd"] Oct 31 12:56:50.956009 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:50.924190 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:50.995643 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:50.975932 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:56:51.075741 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:51.034222 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c4167730-openshift-default-dockercfg-cr68m\"" Oct 31 12:56:51.118815 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:51.096038 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd"] Oct 31 12:56:51.118815 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:51.111364 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-workload-socket\") pod \"e2e-mt-c4167730-openshift-default-578df84889-42vvd\" (UID: \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\") " pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:56:51.118815 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:51.111455 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-credential-socket\") pod \"e2e-mt-c4167730-openshift-default-578df84889-42vvd\" (UID: \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\") " pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:56:51.118815 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:51.111532 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-istio-envoy\") pod \"e2e-mt-c4167730-openshift-default-578df84889-42vvd\" (UID: \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\") " pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:56:51.118815 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:51.111572 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-istio-podinfo\") pod \"e2e-mt-c4167730-openshift-default-578df84889-42vvd\" (UID: \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\") " pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:56:51.118815 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:51.111650 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-workload-certs\") pod \"e2e-mt-c4167730-openshift-default-578df84889-42vvd\" (UID: \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\") " pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:56:51.118815 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:51.111683 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-istio-token\") pod \"e2e-mt-c4167730-openshift-default-578df84889-42vvd\" (UID: \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\") " pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:56:51.118815 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:51.111772 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z66l7\" (UniqueName: \"kubernetes.io/projected/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-kube-api-access-z66l7\") pod \"e2e-mt-c4167730-openshift-default-578df84889-42vvd\" (UID: \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\") " pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:56:51.118815 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:51.111884 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-istio-data\") pod \"e2e-mt-c4167730-openshift-default-578df84889-42vvd\" (UID: \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\") " pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:56:51.118815 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:51.111938 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-istiod-ca-cert\") pod \"e2e-mt-c4167730-openshift-default-578df84889-42vvd\" (UID: \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\") " pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:56:51.226006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:51.213099 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-workload-certs\") pod \"e2e-mt-c4167730-openshift-default-578df84889-42vvd\" (UID: \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\") " pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:56:51.231050 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:51.231017 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-istio-token\") pod \"e2e-mt-c4167730-openshift-default-578df84889-42vvd\" (UID: \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\") " pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:56:51.231740 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:51.231708 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z66l7\" (UniqueName: \"kubernetes.io/projected/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-kube-api-access-z66l7\") pod \"e2e-mt-c4167730-openshift-default-578df84889-42vvd\" (UID: \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\") " pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:56:51.237004 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:51.232012 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-istio-data\") pod \"e2e-mt-c4167730-openshift-default-578df84889-42vvd\" (UID: \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\") " pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:56:51.237232 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:51.237209 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-istiod-ca-cert\") pod \"e2e-mt-c4167730-openshift-default-578df84889-42vvd\" (UID: \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\") " pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:56:51.242157 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:51.242135 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-workload-socket\") pod \"e2e-mt-c4167730-openshift-default-578df84889-42vvd\" (UID: \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\") " pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:56:51.242377 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:51.242358 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-credential-socket\") pod \"e2e-mt-c4167730-openshift-default-578df84889-42vvd\" (UID: \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\") " pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:56:51.242558 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:51.242541 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-istio-envoy\") pod \"e2e-mt-c4167730-openshift-default-578df84889-42vvd\" (UID: \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\") " pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:56:51.242693 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:51.242676 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-istio-podinfo\") pod \"e2e-mt-c4167730-openshift-default-578df84889-42vvd\" (UID: \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\") " pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:56:51.257612 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:51.227431 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-workload-certs\") pod \"e2e-mt-c4167730-openshift-default-578df84889-42vvd\" (UID: \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\") " pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:56:51.257612 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:51.234703 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-istio-data\") pod \"e2e-mt-c4167730-openshift-default-578df84889-42vvd\" (UID: \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\") " pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:56:51.257612 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:51.252540 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-workload-socket\") pod \"e2e-mt-c4167730-openshift-default-578df84889-42vvd\" (UID: \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\") " pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:56:51.260189 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:51.260164 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-istiod-ca-cert\") pod \"e2e-mt-c4167730-openshift-default-578df84889-42vvd\" (UID: \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\") " pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:56:51.284021 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:51.279898 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-istio-envoy\") pod \"e2e-mt-c4167730-openshift-default-578df84889-42vvd\" (UID: \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\") " pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:56:51.284021 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:51.280406 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-credential-socket\") pod \"e2e-mt-c4167730-openshift-default-578df84889-42vvd\" (UID: \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\") " pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:56:51.338019 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:51.337568 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-istio-token\") pod \"e2e-mt-c4167730-openshift-default-578df84889-42vvd\" (UID: \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\") " pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:56:51.338512 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:51.338254 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z66l7\" (UniqueName: \"kubernetes.io/projected/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-kube-api-access-z66l7\") pod \"e2e-mt-c4167730-openshift-default-578df84889-42vvd\" (UID: \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\") " pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:56:51.339146 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:51.338691 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 12:56:51.339146 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:51.338727 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-istio-podinfo\") pod \"e2e-mt-c4167730-openshift-default-578df84889-42vvd\" (UID: \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\") " pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:56:51.339146 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:51.338747 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" podUID="1d9c49d7-4979-4d5c-b94e-d477215dd938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 12:56:51.540093 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:51.531354 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:51.576570 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:51.575304 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:56:51.989075 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:51.989033 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:52.321104 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:52.320642 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 12:56:52.321104 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:52.320706 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" podUID="1d9c49d7-4979-4d5c-b94e-d477215dd938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 12:56:52.899014 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:52.821621 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd"] Oct 31 12:56:53.031967 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:53.031302 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:56:53.031967 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:53.031407 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:56:53.031967 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:53.031451 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:56:53.346008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:53.319429 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 12:56:53.388116 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:53.387959 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" podUID="1d9c49d7-4979-4d5c-b94e-d477215dd938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 12:56:53.559435 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:53.559348 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" event={"ID":"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216","Type":"ContainerStarted","Data":"ba04af26eae0d1aa9e9401e9f2b86125419446d5b05c5cb721163591b2e2b0e5"} Oct 31 12:56:53.788590 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:53.788553 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:54.087931 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:54.087890 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc"] Oct 31 12:56:54.309118 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:54.293693 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:54.316012 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:54.309917 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" Oct 31 12:56:54.339067 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:54.339036 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc"] Oct 31 12:56:54.363018 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:54.359928 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 12:56:54.363018 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:54.360003 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" podUID="1d9c49d7-4979-4d5c-b94e-d477215dd938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 12:56:54.363018 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:54.360540 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-107955\"/\"default-dockercfg-gxcbm\"" Oct 31 12:56:54.363018 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:54.360769 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-107955\"/\"distinct-2-w0-107955-kserve-self-signed-certs\"" Oct 31 12:56:54.363018 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:54.361423 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-107955\"/\"kube-root-ca.crt\"" Oct 31 12:56:54.363418 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:54.363395 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-107955\"/\"openshift-service-ca.crt\"" Oct 31 12:56:54.419007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:54.393807 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da83bd57-52af-48d3-bd76-42e36c18112f-dshm\") pod \"distinct-2-w0-107955-kserve-64487cdd8f-2xrhc\" (UID: \"da83bd57-52af-48d3-bd76-42e36c18112f\") " pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" Oct 31 12:56:54.419737 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:54.419702 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da83bd57-52af-48d3-bd76-42e36c18112f-home\") pod \"distinct-2-w0-107955-kserve-64487cdd8f-2xrhc\" (UID: \"da83bd57-52af-48d3-bd76-42e36c18112f\") " pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" Oct 31 12:56:54.430254 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:54.430216 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da83bd57-52af-48d3-bd76-42e36c18112f-tmp-dir\") pod \"distinct-2-w0-107955-kserve-64487cdd8f-2xrhc\" (UID: \"da83bd57-52af-48d3-bd76-42e36c18112f\") " pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" Oct 31 12:56:54.430573 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:54.430550 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da83bd57-52af-48d3-bd76-42e36c18112f-tls-certs\") pod \"distinct-2-w0-107955-kserve-64487cdd8f-2xrhc\" (UID: \"da83bd57-52af-48d3-bd76-42e36c18112f\") " pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" Oct 31 12:56:54.449282 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:54.449249 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhhb7\" (UniqueName: \"kubernetes.io/projected/da83bd57-52af-48d3-bd76-42e36c18112f-kube-api-access-lhhb7\") pod \"distinct-2-w0-107955-kserve-64487cdd8f-2xrhc\" (UID: \"da83bd57-52af-48d3-bd76-42e36c18112f\") " pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" Oct 31 12:56:54.449630 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:54.449606 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da83bd57-52af-48d3-bd76-42e36c18112f-model-cache\") pod \"distinct-2-w0-107955-kserve-64487cdd8f-2xrhc\" (UID: \"da83bd57-52af-48d3-bd76-42e36c18112f\") " pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" Oct 31 12:56:54.630482 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:54.630402 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da83bd57-52af-48d3-bd76-42e36c18112f-dshm\") pod \"distinct-2-w0-107955-kserve-64487cdd8f-2xrhc\" (UID: \"da83bd57-52af-48d3-bd76-42e36c18112f\") " pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" Oct 31 12:56:54.671373 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:54.665238 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da83bd57-52af-48d3-bd76-42e36c18112f-dshm\") pod \"distinct-2-w0-107955-kserve-64487cdd8f-2xrhc\" (UID: \"da83bd57-52af-48d3-bd76-42e36c18112f\") " pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" Oct 31 12:56:54.671373 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:54.665385 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da83bd57-52af-48d3-bd76-42e36c18112f-home\") pod \"distinct-2-w0-107955-kserve-64487cdd8f-2xrhc\" (UID: \"da83bd57-52af-48d3-bd76-42e36c18112f\") " pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" Oct 31 12:56:54.671373 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:54.665420 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da83bd57-52af-48d3-bd76-42e36c18112f-tmp-dir\") pod \"distinct-2-w0-107955-kserve-64487cdd8f-2xrhc\" (UID: \"da83bd57-52af-48d3-bd76-42e36c18112f\") " pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" Oct 31 12:56:54.671373 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:54.665466 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da83bd57-52af-48d3-bd76-42e36c18112f-tls-certs\") pod \"distinct-2-w0-107955-kserve-64487cdd8f-2xrhc\" (UID: \"da83bd57-52af-48d3-bd76-42e36c18112f\") " pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" Oct 31 12:56:54.671373 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:54.665512 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhhb7\" (UniqueName: \"kubernetes.io/projected/da83bd57-52af-48d3-bd76-42e36c18112f-kube-api-access-lhhb7\") pod \"distinct-2-w0-107955-kserve-64487cdd8f-2xrhc\" (UID: \"da83bd57-52af-48d3-bd76-42e36c18112f\") " pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" Oct 31 12:56:54.671373 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:54.665646 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da83bd57-52af-48d3-bd76-42e36c18112f-model-cache\") pod \"distinct-2-w0-107955-kserve-64487cdd8f-2xrhc\" (UID: \"da83bd57-52af-48d3-bd76-42e36c18112f\") " pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" Oct 31 12:56:54.671373 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:54.666170 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da83bd57-52af-48d3-bd76-42e36c18112f-model-cache\") pod \"distinct-2-w0-107955-kserve-64487cdd8f-2xrhc\" (UID: \"da83bd57-52af-48d3-bd76-42e36c18112f\") " pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" Oct 31 12:56:54.671373 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:54.666469 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da83bd57-52af-48d3-bd76-42e36c18112f-home\") pod \"distinct-2-w0-107955-kserve-64487cdd8f-2xrhc\" (UID: \"da83bd57-52af-48d3-bd76-42e36c18112f\") " pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" Oct 31 12:56:54.671373 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:54.666699 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da83bd57-52af-48d3-bd76-42e36c18112f-tmp-dir\") pod \"distinct-2-w0-107955-kserve-64487cdd8f-2xrhc\" (UID: \"da83bd57-52af-48d3-bd76-42e36c18112f\") " pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" Oct 31 12:56:54.721817 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:54.721780 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhhb7\" (UniqueName: \"kubernetes.io/projected/da83bd57-52af-48d3-bd76-42e36c18112f-kube-api-access-lhhb7\") pod \"distinct-2-w0-107955-kserve-64487cdd8f-2xrhc\" (UID: \"da83bd57-52af-48d3-bd76-42e36c18112f\") " pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" Oct 31 12:56:54.795267 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:54.795233 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da83bd57-52af-48d3-bd76-42e36c18112f-tls-certs\") pod \"distinct-2-w0-107955-kserve-64487cdd8f-2xrhc\" (UID: \"da83bd57-52af-48d3-bd76-42e36c18112f\") " pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" Oct 31 12:56:54.903010 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:54.902904 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:55.075538 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:55.075507 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" Oct 31 12:56:55.398117 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:55.398078 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 12:56:55.398617 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:55.398143 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" podUID="1d9c49d7-4979-4d5c-b94e-d477215dd938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 12:56:55.755543 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:55.755412 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:55.804511 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:55.804484 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc"] Oct 31 12:56:55.807571 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:56:55.807542 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda83bd57_52af_48d3_bd76_42e36c18112f.slice/crio-f93e3c5751f498c1b8f8db8243e0018ac5bb2154d043448ca3d965ccd5e6f69d WatchSource:0}: Error finding container f93e3c5751f498c1b8f8db8243e0018ac5bb2154d043448ca3d965ccd5e6f69d: Status 404 returned error can't find the container with id f93e3c5751f498c1b8f8db8243e0018ac5bb2154d043448ca3d965ccd5e6f69d Oct 31 12:56:55.809156 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:55.809118 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" event={"ID":"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216","Type":"ContainerStarted","Data":"ec182338bc3277355f454c782942169d58e8ab71578976ef8999c1c47e490a03"} Oct 31 12:56:55.833403 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:55.833358 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" podStartSLOduration=5.833344469 podStartE2EDuration="5.833344469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:56:55.831937356 +0000 UTC m=+1462.315060257" watchObservedRunningTime="2026-10-31 12:56:55.833344469 +0000 UTC m=+1462.316467430" Oct 31 12:56:56.309379 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:56.309346 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 12:56:56.309534 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:56.309399 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" podUID="1d9c49d7-4979-4d5c-b94e-d477215dd938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 12:56:56.576546 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:56.576445 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:56:56.582463 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:56.582435 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:56:56.670395 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:56.670355 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:56.817492 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:56.817448 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" event={"ID":"da83bd57-52af-48d3-bd76-42e36c18112f","Type":"ContainerStarted","Data":"3d1752ddc92abbbb6cf16987d1888bc6c054f2064bfa4cafc8bca8492b1c2f50"} Oct 31 12:56:56.817492 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:56.817495 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" event={"ID":"da83bd57-52af-48d3-bd76-42e36c18112f","Type":"ContainerStarted","Data":"f93e3c5751f498c1b8f8db8243e0018ac5bb2154d043448ca3d965ccd5e6f69d"} Oct 31 12:56:56.817746 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:56.817711 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:56:56.818765 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:56.818743 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:56:56.853677 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:56.841816 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" podStartSLOduration=3.841799174 podStartE2EDuration="3.841799174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:56:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:56:56.840543355 +0000 UTC m=+1463.323666257" watchObservedRunningTime="2026-10-31 12:56:56.841799174 +0000 UTC m=+1463.324922074" Oct 31 12:56:57.082153 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:57.082115 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" Oct 31 12:56:57.309004 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:57.308955 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 12:56:57.309176 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:57.309028 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" podUID="1d9c49d7-4979-4d5c-b94e-d477215dd938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 12:56:57.566045 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:57.565939 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:58.223819 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:58.223780 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:58.309254 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:58.309210 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 12:56:58.309397 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:58.309317 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" podUID="1d9c49d7-4979-4d5c-b94e-d477215dd938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 12:56:58.432598 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:58.432562 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:58.666749 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:58.666663 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:59.381827 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:59.381789 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-6x7qm"] Oct 31 12:56:59.386217 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:59.386195 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-6x7qm" Oct 31 12:56:59.391747 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:59.391720 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c4167730\"" Oct 31 12:56:59.402623 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:59.402599 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-6x7qm"] Oct 31 12:56:59.413526 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:59.413493 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-2wthh"] Oct 31 12:56:59.418520 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:59.418493 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-2wthh" Oct 31 12:56:59.428703 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:59.428678 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c4167730-dockercfg-8vpx9\"" Oct 31 12:56:59.428874 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:59.428845 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-2wthh"] Oct 31 12:56:59.431150 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:59.431128 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:59.448315 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:59.448281 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d8bdee0-d436-425d-8b4d-53337f773d27-plugins-config-volume\") pod \"payload-processing-e2e-mt-c4167730-758cb7b74f-6x7qm\" (UID: \"2d8bdee0-d436-425d-8b4d-53337f773d27\") " pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-6x7qm" Oct 31 12:56:59.448439 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:59.448396 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z594\" (UniqueName: \"kubernetes.io/projected/158edcbd-2810-4cf7-a2e5-2a881ae0568e-kube-api-access-7z594\") pod \"payload-pre-processing-e2e-mt-c4167730-6cb6676695-2wthh\" (UID: \"158edcbd-2810-4cf7-a2e5-2a881ae0568e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-2wthh" Oct 31 12:56:59.448534 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:59.448512 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/158edcbd-2810-4cf7-a2e5-2a881ae0568e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c4167730-6cb6676695-2wthh\" (UID: \"158edcbd-2810-4cf7-a2e5-2a881ae0568e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-2wthh" Oct 31 12:56:59.448685 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:59.448666 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jmw9\" (UniqueName: \"kubernetes.io/projected/2d8bdee0-d436-425d-8b4d-53337f773d27-kube-api-access-6jmw9\") pod \"payload-processing-e2e-mt-c4167730-758cb7b74f-6x7qm\" (UID: \"2d8bdee0-d436-425d-8b4d-53337f773d27\") " pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-6x7qm" Oct 31 12:56:59.558160 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:59.553947 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jmw9\" (UniqueName: \"kubernetes.io/projected/2d8bdee0-d436-425d-8b4d-53337f773d27-kube-api-access-6jmw9\") pod \"payload-processing-e2e-mt-c4167730-758cb7b74f-6x7qm\" (UID: \"2d8bdee0-d436-425d-8b4d-53337f773d27\") " pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-6x7qm" Oct 31 12:56:59.558160 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:59.554132 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d8bdee0-d436-425d-8b4d-53337f773d27-plugins-config-volume\") pod \"payload-processing-e2e-mt-c4167730-758cb7b74f-6x7qm\" (UID: \"2d8bdee0-d436-425d-8b4d-53337f773d27\") " pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-6x7qm" Oct 31 12:56:59.558160 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:59.554214 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z594\" (UniqueName: \"kubernetes.io/projected/158edcbd-2810-4cf7-a2e5-2a881ae0568e-kube-api-access-7z594\") pod \"payload-pre-processing-e2e-mt-c4167730-6cb6676695-2wthh\" (UID: \"158edcbd-2810-4cf7-a2e5-2a881ae0568e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-2wthh" Oct 31 12:56:59.558160 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:59.554318 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/158edcbd-2810-4cf7-a2e5-2a881ae0568e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c4167730-6cb6676695-2wthh\" (UID: \"158edcbd-2810-4cf7-a2e5-2a881ae0568e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-2wthh" Oct 31 12:56:59.558160 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:59.555370 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/158edcbd-2810-4cf7-a2e5-2a881ae0568e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c4167730-6cb6676695-2wthh\" (UID: \"158edcbd-2810-4cf7-a2e5-2a881ae0568e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-2wthh" Oct 31 12:56:59.558160 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:59.556503 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d8bdee0-d436-425d-8b4d-53337f773d27-plugins-config-volume\") pod \"payload-processing-e2e-mt-c4167730-758cb7b74f-6x7qm\" (UID: \"2d8bdee0-d436-425d-8b4d-53337f773d27\") " pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-6x7qm" Oct 31 12:56:59.586388 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:59.586350 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jmw9\" (UniqueName: \"kubernetes.io/projected/2d8bdee0-d436-425d-8b4d-53337f773d27-kube-api-access-6jmw9\") pod \"payload-processing-e2e-mt-c4167730-758cb7b74f-6x7qm\" (UID: \"2d8bdee0-d436-425d-8b4d-53337f773d27\") " pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-6x7qm" Oct 31 12:56:59.597072 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:59.597034 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z594\" (UniqueName: \"kubernetes.io/projected/158edcbd-2810-4cf7-a2e5-2a881ae0568e-kube-api-access-7z594\") pod \"payload-pre-processing-e2e-mt-c4167730-6cb6676695-2wthh\" (UID: \"158edcbd-2810-4cf7-a2e5-2a881ae0568e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-2wthh" Oct 31 12:56:59.634596 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:59.634509 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:56:59.706274 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:59.706236 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-6x7qm" Oct 31 12:56:59.745128 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:56:59.745084 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-2wthh" Oct 31 12:57:00.021205 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:00.021177 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-6x7qm"] Oct 31 12:57:00.021741 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:57:00.021709 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d8bdee0_d436_425d_8b4d_53337f773d27.slice/crio-9ee23c6936abb1573b5b26a4ddd7bab6d3d922fe8079d6fd07b2f787bf4d791c WatchSource:0}: Error finding container 9ee23c6936abb1573b5b26a4ddd7bab6d3d922fe8079d6fd07b2f787bf4d791c: Status 404 returned error can't find the container with id 9ee23c6936abb1573b5b26a4ddd7bab6d3d922fe8079d6fd07b2f787bf4d791c Oct 31 12:57:00.082949 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:57:00.082912 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod158edcbd_2810_4cf7_a2e5_2a881ae0568e.slice/crio-1b767103236c8ed683d9da8f4048cd03a3f94e865b4b42b73a383143c0dbaff9 WatchSource:0}: Error finding container 1b767103236c8ed683d9da8f4048cd03a3f94e865b4b42b73a383143c0dbaff9: Status 404 returned error can't find the container with id 1b767103236c8ed683d9da8f4048cd03a3f94e865b4b42b73a383143c0dbaff9 Oct 31 12:57:00.084805 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:00.084783 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-2wthh"] Oct 31 12:57:00.309910 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:00.309826 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:57:00.309910 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:00.309881 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" podUID="1d9c49d7-4979-4d5c-b94e-d477215dd938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:00.454030 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:00.453339 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:00.599354 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:00.599278 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" Oct 31 12:57:00.599625 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:00.599600 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" Oct 31 12:57:00.601277 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:00.601252 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" Oct 31 12:57:00.863428 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:00.863345 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-2wthh" event={"ID":"158edcbd-2810-4cf7-a2e5-2a881ae0568e","Type":"ContainerStarted","Data":"2f18b66c3e153ad6cb29b8d224c86d5afc857aba8d9c665054491381f590fe0e"} Oct 31 12:57:00.863428 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:00.863408 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-2wthh" event={"ID":"158edcbd-2810-4cf7-a2e5-2a881ae0568e","Type":"ContainerStarted","Data":"1b767103236c8ed683d9da8f4048cd03a3f94e865b4b42b73a383143c0dbaff9"} Oct 31 12:57:00.863625 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:00.863444 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-2wthh" Oct 31 12:57:00.865058 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:00.865031 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-6x7qm" event={"ID":"2d8bdee0-d436-425d-8b4d-53337f773d27","Type":"ContainerStarted","Data":"64be69590e1308cc2f9e44fa61e7bc61364928f40cbd743d63e2d18d74ce8195"} Oct 31 12:57:00.865183 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:00.865074 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-6x7qm" event={"ID":"2d8bdee0-d436-425d-8b4d-53337f773d27","Type":"ContainerStarted","Data":"9ee23c6936abb1573b5b26a4ddd7bab6d3d922fe8079d6fd07b2f787bf4d791c"} Oct 31 12:57:00.865183 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:00.865124 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-6x7qm" Oct 31 12:57:00.890779 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:00.890726 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-2wthh" podStartSLOduration=1.890709046 podStartE2EDuration="1.890709046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:57:00.884682951 +0000 UTC m=+1467.367805852" watchObservedRunningTime="2026-10-31 12:57:00.890709046 +0000 UTC m=+1467.373831949" Oct 31 12:57:00.910007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:00.908799 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-6x7qm" podStartSLOduration=1.9087830559999999 podStartE2EDuration="1.908783056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:57:00.904738533 +0000 UTC m=+1467.387861433" watchObservedRunningTime="2026-10-31 12:57:00.908783056 +0000 UTC m=+1467.391905958" Oct 31 12:57:01.432484 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:01.432447 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:02.862764 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:02.862718 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:04.320008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:04.318075 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn"] Oct 31 12:57:04.369232 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:04.324042 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn" Oct 31 12:57:04.369232 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:04.342499 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c4167730-serving-cert\"" Oct 31 12:57:04.851117 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:04.847554 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75be7457-b259-4516-8c32-cceef659c36b-maas-api-tls\") pod \"maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn\" (UID: \"75be7457-b259-4516-8c32-cceef659c36b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn" Oct 31 12:57:04.852102 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:04.852070 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75be7457-b259-4516-8c32-cceef659c36b-openshift-service-ca\") pod \"maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn\" (UID: \"75be7457-b259-4516-8c32-cceef659c36b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn" Oct 31 12:57:04.852487 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:04.852463 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75be7457-b259-4516-8c32-cceef659c36b-metrics-certs\") pod \"maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn\" (UID: \"75be7457-b259-4516-8c32-cceef659c36b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn" Oct 31 12:57:04.853397 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:04.852916 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljgrv\" (UniqueName: \"kubernetes.io/projected/75be7457-b259-4516-8c32-cceef659c36b-kube-api-access-ljgrv\") pod \"maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn\" (UID: \"75be7457-b259-4516-8c32-cceef659c36b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn" Oct 31 12:57:04.872604 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:04.872572 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn"] Oct 31 12:57:04.878545 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:04.878508 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:04.961489 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:04.961443 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljgrv\" (UniqueName: \"kubernetes.io/projected/75be7457-b259-4516-8c32-cceef659c36b-kube-api-access-ljgrv\") pod \"maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn\" (UID: \"75be7457-b259-4516-8c32-cceef659c36b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn" Oct 31 12:57:04.961689 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:04.961604 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75be7457-b259-4516-8c32-cceef659c36b-maas-api-tls\") pod \"maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn\" (UID: \"75be7457-b259-4516-8c32-cceef659c36b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn" Oct 31 12:57:05.021005 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:05.008834 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljgrv\" (UniqueName: \"kubernetes.io/projected/75be7457-b259-4516-8c32-cceef659c36b-kube-api-access-ljgrv\") pod \"maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn\" (UID: \"75be7457-b259-4516-8c32-cceef659c36b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn" Oct 31 12:57:05.033293 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:05.028448 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75be7457-b259-4516-8c32-cceef659c36b-openshift-service-ca\") pod \"maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn\" (UID: \"75be7457-b259-4516-8c32-cceef659c36b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn" Oct 31 12:57:05.033293 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:05.029072 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75be7457-b259-4516-8c32-cceef659c36b-maas-api-tls\") pod \"maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn\" (UID: \"75be7457-b259-4516-8c32-cceef659c36b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn" Oct 31 12:57:05.059107 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:05.059072 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75be7457-b259-4516-8c32-cceef659c36b-openshift-service-ca\") pod \"maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn\" (UID: \"75be7457-b259-4516-8c32-cceef659c36b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn" Oct 31 12:57:05.059447 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:05.059412 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75be7457-b259-4516-8c32-cceef659c36b-metrics-certs\") pod \"maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn\" (UID: \"75be7457-b259-4516-8c32-cceef659c36b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn" Oct 31 12:57:05.077520 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:05.077491 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" Oct 31 12:57:05.096016 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:05.083943 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75be7457-b259-4516-8c32-cceef659c36b-metrics-certs\") pod \"maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn\" (UID: \"75be7457-b259-4516-8c32-cceef659c36b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn" Oct 31 12:57:05.105449 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:05.105056 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" Oct 31 12:57:05.150099 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:05.150053 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:05.220225 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:05.220190 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" Oct 31 12:57:05.362367 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:05.362282 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn" Oct 31 12:57:05.495885 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:05.488506 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:05.645775 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:05.645751 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn"] Oct 31 12:57:05.648274 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:57:05.648247 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75be7457_b259_4516_8c32_cceef659c36b.slice/crio-390a99e4d008aede758672c9878fa18fb5653afac09820a74af07dc2da949365 WatchSource:0}: Error finding container 390a99e4d008aede758672c9878fa18fb5653afac09820a74af07dc2da949365: Status 404 returned error can't find the container with id 390a99e4d008aede758672c9878fa18fb5653afac09820a74af07dc2da949365 Oct 31 12:57:05.649653 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:05.649637 6841 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 12:57:06.216249 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:06.216204 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn" event={"ID":"75be7457-b259-4516-8c32-cceef659c36b","Type":"ContainerStarted","Data":"390a99e4d008aede758672c9878fa18fb5653afac09820a74af07dc2da949365"} Oct 31 12:57:06.343793 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:06.343757 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:06.556008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:06.555316 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:07.227156 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:07.227115 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn" event={"ID":"75be7457-b259-4516-8c32-cceef659c36b","Type":"ContainerStarted","Data":"3455d8c04f6946cf52c69ff8c874874b24859ff6d8f37e494a60ad944f5f0eba"} Oct 31 12:57:07.227334 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:07.227178 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn" Oct 31 12:57:07.257006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:07.256141 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn" podStartSLOduration=2.717475538 podStartE2EDuration="3.256118606s" totalImagesPullingTime="538.643068ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:57:07.25506187 +0000 UTC m=+1473.738184776" watchObservedRunningTime="2026-10-31 12:57:07.256118606 +0000 UTC m=+1473.739241511" Oct 31 12:57:07.309065 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:07.302486 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:08.025287 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:08.025249 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:08.725868 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:08.725825 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:09.428753 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:09.428716 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:10.188765 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:10.188705 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:10.933484 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:10.929821 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:11.147399 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.147364 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:11.225600 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.225569 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f"] Oct 31 12:57:11.231737 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.231713 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:57:11.231907 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.231831 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f"] Oct 31 12:57:11.237565 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.237539 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-702419-openshift-default-dockercfg-dm2dq\"" Oct 31 12:57:11.280307 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.280270 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-istiod-ca-cert\") pod \"e2e-api-a-702419-openshift-default-7b8c755667-cpq6f\" (UID: \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\") " pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:57:11.280460 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.280344 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-istio-token\") pod \"e2e-api-a-702419-openshift-default-7b8c755667-cpq6f\" (UID: \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\") " pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:57:11.280460 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.280404 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-credential-socket\") pod \"e2e-api-a-702419-openshift-default-7b8c755667-cpq6f\" (UID: \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\") " pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:57:11.280460 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.280425 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-workload-certs\") pod \"e2e-api-a-702419-openshift-default-7b8c755667-cpq6f\" (UID: \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\") " pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:57:11.280568 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.280494 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-istio-podinfo\") pod \"e2e-api-a-702419-openshift-default-7b8c755667-cpq6f\" (UID: \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\") " pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:57:11.280603 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.280567 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-workload-socket\") pod \"e2e-api-a-702419-openshift-default-7b8c755667-cpq6f\" (UID: \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\") " pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:57:11.280603 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.280590 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-istio-envoy\") pod \"e2e-api-a-702419-openshift-default-7b8c755667-cpq6f\" (UID: \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\") " pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:57:11.280670 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.280652 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-istio-data\") pod \"e2e-api-a-702419-openshift-default-7b8c755667-cpq6f\" (UID: \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\") " pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:57:11.280701 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.280678 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pw45\" (UniqueName: \"kubernetes.io/projected/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-kube-api-access-5pw45\") pod \"e2e-api-a-702419-openshift-default-7b8c755667-cpq6f\" (UID: \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\") " pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:57:11.381072 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.381033 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:11.382252 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.382221 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-istio-token\") pod \"e2e-api-a-702419-openshift-default-7b8c755667-cpq6f\" (UID: \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\") " pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:57:11.382480 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.382449 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-credential-socket\") pod \"e2e-api-a-702419-openshift-default-7b8c755667-cpq6f\" (UID: \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\") " pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:57:11.382614 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.382595 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-workload-certs\") pod \"e2e-api-a-702419-openshift-default-7b8c755667-cpq6f\" (UID: \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\") " pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:57:11.382778 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.382760 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-istio-podinfo\") pod \"e2e-api-a-702419-openshift-default-7b8c755667-cpq6f\" (UID: \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\") " pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:57:11.382965 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.382945 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-workload-socket\") pod \"e2e-api-a-702419-openshift-default-7b8c755667-cpq6f\" (UID: \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\") " pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:57:11.392199 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.392165 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-istio-envoy\") pod \"e2e-api-a-702419-openshift-default-7b8c755667-cpq6f\" (UID: \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\") " pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:57:11.392393 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.392374 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-istio-data\") pod \"e2e-api-a-702419-openshift-default-7b8c755667-cpq6f\" (UID: \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\") " pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:57:11.392525 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.392509 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pw45\" (UniqueName: \"kubernetes.io/projected/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-kube-api-access-5pw45\") pod \"e2e-api-a-702419-openshift-default-7b8c755667-cpq6f\" (UID: \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\") " pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:57:11.392834 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.392817 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-istiod-ca-cert\") pod \"e2e-api-a-702419-openshift-default-7b8c755667-cpq6f\" (UID: \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\") " pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:57:11.428007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.393757 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-istiod-ca-cert\") pod \"e2e-api-a-702419-openshift-default-7b8c755667-cpq6f\" (UID: \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\") " pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:57:11.428267 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.398575 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-istio-podinfo\") pod \"e2e-api-a-702419-openshift-default-7b8c755667-cpq6f\" (UID: \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\") " pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:57:11.428409 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.392096 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-credential-socket\") pod \"e2e-api-a-702419-openshift-default-7b8c755667-cpq6f\" (UID: \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\") " pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:57:11.428501 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.391460 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-workload-socket\") pod \"e2e-api-a-702419-openshift-default-7b8c755667-cpq6f\" (UID: \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\") " pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:57:11.428588 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.392327 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-workload-certs\") pod \"e2e-api-a-702419-openshift-default-7b8c755667-cpq6f\" (UID: \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\") " pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:57:11.428665 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.400833 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-istio-data\") pod \"e2e-api-a-702419-openshift-default-7b8c755667-cpq6f\" (UID: \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\") " pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:57:11.437965 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.437887 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-istio-envoy\") pod \"e2e-api-a-702419-openshift-default-7b8c755667-cpq6f\" (UID: \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\") " pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:57:11.460006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.457880 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-istio-token\") pod \"e2e-api-a-702419-openshift-default-7b8c755667-cpq6f\" (UID: \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\") " pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:57:11.460006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.458546 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pw45\" (UniqueName: \"kubernetes.io/projected/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-kube-api-access-5pw45\") pod \"e2e-api-a-702419-openshift-default-7b8c755667-cpq6f\" (UID: \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\") " pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:57:11.549438 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.549355 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:57:11.866254 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.866224 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f"] Oct 31 12:57:11.867091 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:57:11.867064 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f69dc4_ad91_4cdd_a98f_c3e27183a88f.slice/crio-3286ee0b87c6698394df37a8522c357fc0e11cf123f9ad47e3cc25ddf31ea3ef WatchSource:0}: Error finding container 3286ee0b87c6698394df37a8522c357fc0e11cf123f9ad47e3cc25ddf31ea3ef: Status 404 returned error can't find the container with id 3286ee0b87c6698394df37a8522c357fc0e11cf123f9ad47e3cc25ddf31ea3ef Oct 31 12:57:11.869604 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.869574 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:57:11.869696 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.869639 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:57:11.869696 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.869671 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:57:11.897443 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.897420 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:11.929458 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.929432 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-6x7qm" Oct 31 12:57:11.931053 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:11.931032 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-2wthh" Oct 31 12:57:12.146096 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:12.144961 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:12.282271 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:12.282232 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" event={"ID":"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f","Type":"ContainerStarted","Data":"dc9d677ee8e7ea26192864987426be9d2678152d572be3bf1348616c7f6c01cb"} Oct 31 12:57:12.282419 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:12.282293 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" event={"ID":"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f","Type":"ContainerStarted","Data":"3286ee0b87c6698394df37a8522c357fc0e11cf123f9ad47e3cc25ddf31ea3ef"} Oct 31 12:57:12.307280 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:12.307221 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" podStartSLOduration=1.307203522 podStartE2EDuration="1.307203522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:57:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:57:12.304713768 +0000 UTC m=+1478.787836681" watchObservedRunningTime="2026-10-31 12:57:12.307203522 +0000 UTC m=+1478.790326425" Oct 31 12:57:12.550086 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:12.550051 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:57:12.555572 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:12.555542 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:57:12.908859 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:12.908766 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:13.258749 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:13.258722 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn" Oct 31 12:57:13.289220 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:13.289183 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:57:13.290562 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:13.290535 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:57:13.685481 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:13.682997 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:14.536736 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:14.536695 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:14.742387 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:14.742343 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:14.880197 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:14.880116 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw"] Oct 31 12:57:14.884342 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:14.884317 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" Oct 31 12:57:14.899743 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:14.899701 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw"] Oct 31 12:57:14.901395 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:14.901372 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/322f29ac-1b41-4e39-b296-bdedc1fd2690-istio-data\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw\" (UID: \"322f29ac-1b41-4e39-b296-bdedc1fd2690\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" Oct 31 12:57:14.901518 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:14.901427 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/322f29ac-1b41-4e39-b296-bdedc1fd2690-istio-token\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw\" (UID: \"322f29ac-1b41-4e39-b296-bdedc1fd2690\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" Oct 31 12:57:14.901518 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:14.901492 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/322f29ac-1b41-4e39-b296-bdedc1fd2690-istiod-ca-cert\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw\" (UID: \"322f29ac-1b41-4e39-b296-bdedc1fd2690\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" Oct 31 12:57:14.901639 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:14.901560 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v78p9\" (UniqueName: \"kubernetes.io/projected/322f29ac-1b41-4e39-b296-bdedc1fd2690-kube-api-access-v78p9\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw\" (UID: \"322f29ac-1b41-4e39-b296-bdedc1fd2690\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" Oct 31 12:57:14.901702 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:14.901681 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/322f29ac-1b41-4e39-b296-bdedc1fd2690-istio-envoy\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw\" (UID: \"322f29ac-1b41-4e39-b296-bdedc1fd2690\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" Oct 31 12:57:14.901759 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:14.901724 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/322f29ac-1b41-4e39-b296-bdedc1fd2690-workload-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw\" (UID: \"322f29ac-1b41-4e39-b296-bdedc1fd2690\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" Oct 31 12:57:14.901813 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:14.901780 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/322f29ac-1b41-4e39-b296-bdedc1fd2690-workload-certs\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw\" (UID: \"322f29ac-1b41-4e39-b296-bdedc1fd2690\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" Oct 31 12:57:14.901871 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:14.901842 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/322f29ac-1b41-4e39-b296-bdedc1fd2690-credential-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw\" (UID: \"322f29ac-1b41-4e39-b296-bdedc1fd2690\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" Oct 31 12:57:14.902013 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:14.901974 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/322f29ac-1b41-4e39-b296-bdedc1fd2690-istio-podinfo\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw\" (UID: \"322f29ac-1b41-4e39-b296-bdedc1fd2690\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" Oct 31 12:57:15.003475 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:15.003439 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/322f29ac-1b41-4e39-b296-bdedc1fd2690-istio-podinfo\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw\" (UID: \"322f29ac-1b41-4e39-b296-bdedc1fd2690\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" Oct 31 12:57:15.003652 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:15.003525 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/322f29ac-1b41-4e39-b296-bdedc1fd2690-istio-data\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw\" (UID: \"322f29ac-1b41-4e39-b296-bdedc1fd2690\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" Oct 31 12:57:15.003652 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:15.003583 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/322f29ac-1b41-4e39-b296-bdedc1fd2690-istio-token\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw\" (UID: \"322f29ac-1b41-4e39-b296-bdedc1fd2690\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" Oct 31 12:57:15.003652 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:15.003609 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/322f29ac-1b41-4e39-b296-bdedc1fd2690-istiod-ca-cert\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw\" (UID: \"322f29ac-1b41-4e39-b296-bdedc1fd2690\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" Oct 31 12:57:15.003767 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:15.003689 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v78p9\" (UniqueName: \"kubernetes.io/projected/322f29ac-1b41-4e39-b296-bdedc1fd2690-kube-api-access-v78p9\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw\" (UID: \"322f29ac-1b41-4e39-b296-bdedc1fd2690\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" Oct 31 12:57:15.003767 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:15.003733 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/322f29ac-1b41-4e39-b296-bdedc1fd2690-istio-envoy\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw\" (UID: \"322f29ac-1b41-4e39-b296-bdedc1fd2690\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" Oct 31 12:57:15.003861 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:15.003761 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/322f29ac-1b41-4e39-b296-bdedc1fd2690-workload-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw\" (UID: \"322f29ac-1b41-4e39-b296-bdedc1fd2690\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" Oct 31 12:57:15.003861 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:15.003818 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/322f29ac-1b41-4e39-b296-bdedc1fd2690-workload-certs\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw\" (UID: \"322f29ac-1b41-4e39-b296-bdedc1fd2690\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" Oct 31 12:57:15.003960 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:15.003881 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/322f29ac-1b41-4e39-b296-bdedc1fd2690-credential-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw\" (UID: \"322f29ac-1b41-4e39-b296-bdedc1fd2690\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" Oct 31 12:57:15.004076 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:15.004059 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/322f29ac-1b41-4e39-b296-bdedc1fd2690-istio-data\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw\" (UID: \"322f29ac-1b41-4e39-b296-bdedc1fd2690\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" Oct 31 12:57:15.004284 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:15.004253 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/322f29ac-1b41-4e39-b296-bdedc1fd2690-workload-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw\" (UID: \"322f29ac-1b41-4e39-b296-bdedc1fd2690\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" Oct 31 12:57:15.004492 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:15.004474 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/322f29ac-1b41-4e39-b296-bdedc1fd2690-credential-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw\" (UID: \"322f29ac-1b41-4e39-b296-bdedc1fd2690\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" Oct 31 12:57:15.004574 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:15.004472 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/322f29ac-1b41-4e39-b296-bdedc1fd2690-workload-certs\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw\" (UID: \"322f29ac-1b41-4e39-b296-bdedc1fd2690\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" Oct 31 12:57:15.004574 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:15.004529 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/322f29ac-1b41-4e39-b296-bdedc1fd2690-istiod-ca-cert\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw\" (UID: \"322f29ac-1b41-4e39-b296-bdedc1fd2690\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" Oct 31 12:57:15.006206 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:15.006189 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/322f29ac-1b41-4e39-b296-bdedc1fd2690-istio-podinfo\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw\" (UID: \"322f29ac-1b41-4e39-b296-bdedc1fd2690\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" Oct 31 12:57:15.006267 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:15.006204 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/322f29ac-1b41-4e39-b296-bdedc1fd2690-istio-envoy\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw\" (UID: \"322f29ac-1b41-4e39-b296-bdedc1fd2690\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" Oct 31 12:57:15.016139 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:15.016115 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/322f29ac-1b41-4e39-b296-bdedc1fd2690-istio-token\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw\" (UID: \"322f29ac-1b41-4e39-b296-bdedc1fd2690\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" Oct 31 12:57:15.016446 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:15.016421 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v78p9\" (UniqueName: \"kubernetes.io/projected/322f29ac-1b41-4e39-b296-bdedc1fd2690-kube-api-access-v78p9\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw\" (UID: \"322f29ac-1b41-4e39-b296-bdedc1fd2690\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" Oct 31 12:57:15.200858 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:15.200825 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" Oct 31 12:57:15.417843 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:15.417737 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:15.509675 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:15.509598 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw"] Oct 31 12:57:15.513021 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:57:15.512945 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322f29ac_1b41_4e39_b296_bdedc1fd2690.slice/crio-1c0770280ee40c90ba484d67982010540aa213ca4f2e3bbcae9ac5e427d6aba0 WatchSource:0}: Error finding container 1c0770280ee40c90ba484d67982010540aa213ca4f2e3bbcae9ac5e427d6aba0: Status 404 returned error can't find the container with id 1c0770280ee40c90ba484d67982010540aa213ca4f2e3bbcae9ac5e427d6aba0 Oct 31 12:57:15.515663 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:15.515627 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:57:15.515749 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:15.515706 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:57:15.515797 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:15.515750 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:57:15.914911 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:15.914780 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:16.123541 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:16.123473 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:16.315544 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:16.315503 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" event={"ID":"322f29ac-1b41-4e39-b296-bdedc1fd2690","Type":"ContainerStarted","Data":"80bdaa0e243ce1c5db3afa80c9db27bcef9909a61b8bfcccf98eca45cb0b0af6"} Oct 31 12:57:16.315544 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:16.315560 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" event={"ID":"322f29ac-1b41-4e39-b296-bdedc1fd2690","Type":"ContainerStarted","Data":"1c0770280ee40c90ba484d67982010540aa213ca4f2e3bbcae9ac5e427d6aba0"} Oct 31 12:57:16.353030 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:16.351238 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" podStartSLOduration=2.351217634 podStartE2EDuration="2.351217634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:57:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:57:16.339768398 +0000 UTC m=+1482.822891302" watchObservedRunningTime="2026-10-31 12:57:16.351217634 +0000 UTC m=+1482.834340537" Oct 31 12:57:16.475118 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:16.475001 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:17.207004 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:17.204161 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" Oct 31 12:57:17.207004 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:17.204292 6841 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 12:57:17.207004 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:17.204342 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" podUID="322f29ac-1b41-4e39-b296-bdedc1fd2690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 12:57:17.237684 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:17.237639 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:17.891083 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:17.891039 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:18.102453 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:18.102415 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:18.203476 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:18.203436 6841 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 12:57:18.203759 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:18.203726 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" podUID="322f29ac-1b41-4e39-b296-bdedc1fd2690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 12:57:18.307010 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:18.306956 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:19.026751 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:19.026703 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:19.202038 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:19.201972 6841 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 12:57:19.202216 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:19.202083 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" podUID="322f29ac-1b41-4e39-b296-bdedc1fd2690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 12:57:19.273375 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:19.273339 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:19.504913 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:19.504877 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:20.034962 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:57:20.034932 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:57:20.202142 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:20.202109 6841 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 12:57:20.202300 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:20.202177 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" podUID="322f29ac-1b41-4e39-b296-bdedc1fd2690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 12:57:20.319072 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:20.318436 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:20.575381 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:20.574968 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:21.201313 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:21.201273 6841 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 12:57:21.201484 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:21.201330 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" podUID="322f29ac-1b41-4e39-b296-bdedc1fd2690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 12:57:21.404275 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:21.404237 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:21.711097 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:21.711061 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:22.201659 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:22.201625 6841 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 12:57:22.201859 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:22.201689 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" podUID="322f29ac-1b41-4e39-b296-bdedc1fd2690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 12:57:22.440410 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:22.440366 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:23.424719 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:23.424680 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:24.214103 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:24.214066 6841 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:57:24.214372 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:24.214348 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" podUID="322f29ac-1b41-4e39-b296-bdedc1fd2690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:25.268707 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:25.268667 6841 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:57:25.269406 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:25.269369 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" podUID="322f29ac-1b41-4e39-b296-bdedc1fd2690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:25.549070 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:25.542641 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:26.328348 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:26.328296 6841 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:57:26.328929 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:26.328889 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" podUID="322f29ac-1b41-4e39-b296-bdedc1fd2690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:26.626261 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:26.626162 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" Oct 31 12:57:26.626618 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:26.626590 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" Oct 31 12:57:26.627349 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:26.627328 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" Oct 31 12:57:26.742064 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:26.741607 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:27.513438 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:27.513403 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:28.413243 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:28.413188 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:29.197038 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:29.196655 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:30.118363 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:30.118324 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:30.848220 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:30.848174 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:31.052041 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:31.052004 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:31.605710 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:31.605676 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:32.407022 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:32.406875 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:33.207003 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:33.203032 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:34.025010 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:34.020398 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:34.538307 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:34.538261 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z"] Oct 31 12:57:34.544042 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:34.544017 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" Oct 31 12:57:34.561592 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:34.561565 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z"] Oct 31 12:57:34.620858 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:34.620819 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21566728-c2cd-42a6-9ffe-7aaf14243ee8-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z\" (UID: \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" Oct 31 12:57:34.621072 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:34.620868 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t696z\" (UniqueName: \"kubernetes.io/projected/21566728-c2cd-42a6-9ffe-7aaf14243ee8-kube-api-access-t696z\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z\" (UID: \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" Oct 31 12:57:34.621072 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:34.620946 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21566728-c2cd-42a6-9ffe-7aaf14243ee8-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z\" (UID: \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" Oct 31 12:57:34.621072 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:34.621044 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21566728-c2cd-42a6-9ffe-7aaf14243ee8-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z\" (UID: \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" Oct 31 12:57:34.621206 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:34.621126 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21566728-c2cd-42a6-9ffe-7aaf14243ee8-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z\" (UID: \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" Oct 31 12:57:34.621206 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:34.621192 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21566728-c2cd-42a6-9ffe-7aaf14243ee8-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z\" (UID: \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" Oct 31 12:57:34.621319 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:34.621300 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21566728-c2cd-42a6-9ffe-7aaf14243ee8-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z\" (UID: \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" Oct 31 12:57:34.621377 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:34.621341 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21566728-c2cd-42a6-9ffe-7aaf14243ee8-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z\" (UID: \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" Oct 31 12:57:34.621462 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:34.621449 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21566728-c2cd-42a6-9ffe-7aaf14243ee8-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z\" (UID: \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" Oct 31 12:57:34.723167 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:34.723126 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21566728-c2cd-42a6-9ffe-7aaf14243ee8-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z\" (UID: \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" Oct 31 12:57:34.723349 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:34.723180 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t696z\" (UniqueName: \"kubernetes.io/projected/21566728-c2cd-42a6-9ffe-7aaf14243ee8-kube-api-access-t696z\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z\" (UID: \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" Oct 31 12:57:34.723349 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:34.723214 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21566728-c2cd-42a6-9ffe-7aaf14243ee8-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z\" (UID: \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" Oct 31 12:57:34.723349 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:34.723245 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21566728-c2cd-42a6-9ffe-7aaf14243ee8-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z\" (UID: \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" Oct 31 12:57:34.723349 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:34.723278 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21566728-c2cd-42a6-9ffe-7aaf14243ee8-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z\" (UID: \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" Oct 31 12:57:34.723995 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:34.723355 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21566728-c2cd-42a6-9ffe-7aaf14243ee8-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z\" (UID: \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" Oct 31 12:57:34.723995 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:34.723425 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21566728-c2cd-42a6-9ffe-7aaf14243ee8-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z\" (UID: \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" Oct 31 12:57:34.723995 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:34.723455 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21566728-c2cd-42a6-9ffe-7aaf14243ee8-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z\" (UID: \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" Oct 31 12:57:34.723995 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:34.723523 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21566728-c2cd-42a6-9ffe-7aaf14243ee8-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z\" (UID: \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" Oct 31 12:57:34.723995 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:34.723532 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21566728-c2cd-42a6-9ffe-7aaf14243ee8-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z\" (UID: \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" Oct 31 12:57:34.723995 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:34.723856 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21566728-c2cd-42a6-9ffe-7aaf14243ee8-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z\" (UID: \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" Oct 31 12:57:34.723995 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:34.723914 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21566728-c2cd-42a6-9ffe-7aaf14243ee8-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z\" (UID: \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" Oct 31 12:57:34.723995 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:34.723963 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21566728-c2cd-42a6-9ffe-7aaf14243ee8-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z\" (UID: \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" Oct 31 12:57:34.724386 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:34.724367 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21566728-c2cd-42a6-9ffe-7aaf14243ee8-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z\" (UID: \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" Oct 31 12:57:34.725814 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:34.725784 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21566728-c2cd-42a6-9ffe-7aaf14243ee8-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z\" (UID: \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" Oct 31 12:57:34.726035 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:34.725977 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21566728-c2cd-42a6-9ffe-7aaf14243ee8-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z\" (UID: \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" Oct 31 12:57:34.735085 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:34.735060 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21566728-c2cd-42a6-9ffe-7aaf14243ee8-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z\" (UID: \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" Oct 31 12:57:34.735330 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:34.735312 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t696z\" (UniqueName: \"kubernetes.io/projected/21566728-c2cd-42a6-9ffe-7aaf14243ee8-kube-api-access-t696z\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z\" (UID: \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" Oct 31 12:57:34.802654 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:34.802566 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:34.858042 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:34.858002 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" Oct 31 12:57:35.001772 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:35.001745 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:35.236927 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:35.236894 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z"] Oct 31 12:57:35.238151 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:57:35.238127 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21566728_c2cd_42a6_9ffe_7aaf14243ee8.slice/crio-d2b462983ce8adc9d9f19448a1308963d0f672996f0123f828afc42b041d859d WatchSource:0}: Error finding container d2b462983ce8adc9d9f19448a1308963d0f672996f0123f828afc42b041d859d: Status 404 returned error can't find the container with id d2b462983ce8adc9d9f19448a1308963d0f672996f0123f828afc42b041d859d Oct 31 12:57:35.240639 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:35.240608 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:57:35.240710 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:35.240674 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:57:35.240710 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:35.240702 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:57:35.443384 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:35.443348 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:35.914105 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:35.914073 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:36.070034 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:36.067298 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" event={"ID":"21566728-c2cd-42a6-9ffe-7aaf14243ee8","Type":"ContainerStarted","Data":"dc8c120a3032b01ecd3fcb1833a1ce143eeed58653cd4ff5050176a9f091c59c"} Oct 31 12:57:36.070034 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:36.067361 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" event={"ID":"21566728-c2cd-42a6-9ffe-7aaf14243ee8","Type":"ContainerStarted","Data":"d2b462983ce8adc9d9f19448a1308963d0f672996f0123f828afc42b041d859d"} Oct 31 12:57:36.157014 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:36.151954 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:36.366937 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:36.366895 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:36.785027 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:36.784975 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:36.862401 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:36.862370 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" Oct 31 12:57:36.862666 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:36.862443 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 12:57:36.862828 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:36.862803 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" podUID="21566728-c2cd-42a6-9ffe-7aaf14243ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 12:57:37.459865 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:37.459831 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:37.680284 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:37.680244 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:37.860063 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:37.859413 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 12:57:37.860063 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:37.859477 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" podUID="21566728-c2cd-42a6-9ffe-7aaf14243ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 12:57:38.452624 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:38.452586 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:38.866496 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:38.866406 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 12:57:38.866496 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:38.866478 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" podUID="21566728-c2cd-42a6-9ffe-7aaf14243ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 12:57:39.202824 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:39.202720 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:39.436720 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:39.436684 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:39.636655 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:39.636627 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:39.861424 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:39.861382 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 12:57:39.861699 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:39.861665 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" podUID="21566728-c2cd-42a6-9ffe-7aaf14243ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 12:57:40.213021 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:40.212917 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:40.572064 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:40.571854 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" podStartSLOduration=6.571837789 podStartE2EDuration="6.571837789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:57:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:57:36.160343083 +0000 UTC m=+1502.643465984" watchObservedRunningTime="2026-10-31 12:57:40.571837789 +0000 UTC m=+1507.054960690" Oct 31 12:57:40.572888 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:40.572862 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-73503b-598c798485-gpg6n"] Oct 31 12:57:40.578681 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:40.578656 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-73503b-598c798485-gpg6n"] Oct 31 12:57:40.578822 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:40.578750 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-73503b-598c798485-gpg6n" Oct 31 12:57:40.592120 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:40.592096 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-73503b\"" Oct 31 12:57:40.617045 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:40.617013 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:40.718314 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:40.718279 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xvfv\" (UniqueName: \"kubernetes.io/projected/2444a3b7-2746-42e5-821e-b0d6bcb8ee50-kube-api-access-6xvfv\") pod \"payload-processing-e2e-api-b-73503b-598c798485-gpg6n\" (UID: \"2444a3b7-2746-42e5-821e-b0d6bcb8ee50\") " pod="openshift-ingress/payload-processing-e2e-api-b-73503b-598c798485-gpg6n" Oct 31 12:57:40.725246 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:40.725209 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2444a3b7-2746-42e5-821e-b0d6bcb8ee50-plugins-config-volume\") pod \"payload-processing-e2e-api-b-73503b-598c798485-gpg6n\" (UID: \"2444a3b7-2746-42e5-821e-b0d6bcb8ee50\") " pod="openshift-ingress/payload-processing-e2e-api-b-73503b-598c798485-gpg6n" Oct 31 12:57:40.838909 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:40.838820 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:40.843019 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:40.842954 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xvfv\" (UniqueName: \"kubernetes.io/projected/2444a3b7-2746-42e5-821e-b0d6bcb8ee50-kube-api-access-6xvfv\") pod \"payload-processing-e2e-api-b-73503b-598c798485-gpg6n\" (UID: \"2444a3b7-2746-42e5-821e-b0d6bcb8ee50\") " pod="openshift-ingress/payload-processing-e2e-api-b-73503b-598c798485-gpg6n" Oct 31 12:57:40.844395 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:40.843651 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2444a3b7-2746-42e5-821e-b0d6bcb8ee50-plugins-config-volume\") pod \"payload-processing-e2e-api-b-73503b-598c798485-gpg6n\" (UID: \"2444a3b7-2746-42e5-821e-b0d6bcb8ee50\") " pod="openshift-ingress/payload-processing-e2e-api-b-73503b-598c798485-gpg6n" Oct 31 12:57:40.849003 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:40.845383 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2444a3b7-2746-42e5-821e-b0d6bcb8ee50-plugins-config-volume\") pod \"payload-processing-e2e-api-b-73503b-598c798485-gpg6n\" (UID: \"2444a3b7-2746-42e5-821e-b0d6bcb8ee50\") " pod="openshift-ingress/payload-processing-e2e-api-b-73503b-598c798485-gpg6n" Oct 31 12:57:40.858407 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:40.858377 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 12:57:40.858637 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:40.858611 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" podUID="21566728-c2cd-42a6-9ffe-7aaf14243ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 12:57:40.895000 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:40.894947 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xvfv\" (UniqueName: \"kubernetes.io/projected/2444a3b7-2746-42e5-821e-b0d6bcb8ee50-kube-api-access-6xvfv\") pod \"payload-processing-e2e-api-b-73503b-598c798485-gpg6n\" (UID: \"2444a3b7-2746-42e5-821e-b0d6bcb8ee50\") " pod="openshift-ingress/payload-processing-e2e-api-b-73503b-598c798485-gpg6n" Oct 31 12:57:40.902410 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:40.902376 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-73503b-598c798485-gpg6n" Oct 31 12:57:41.312069 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:41.311549 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-73503b-598c798485-gpg6n"] Oct 31 12:57:41.312069 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:41.311904 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:41.315936 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:57:41.315909 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2444a3b7_2746_42e5_821e_b0d6bcb8ee50.slice/crio-1035a7ff292792e6cb87dbe39e6d690448fa0a38c9742a53c88c165d784b63cf WatchSource:0}: Error finding container 1035a7ff292792e6cb87dbe39e6d690448fa0a38c9742a53c88c165d784b63cf: Status 404 returned error can't find the container with id 1035a7ff292792e6cb87dbe39e6d690448fa0a38c9742a53c88c165d784b63cf Oct 31 12:57:41.858949 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:41.858908 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 12:57:41.859162 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:41.858974 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" podUID="21566728-c2cd-42a6-9ffe-7aaf14243ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 12:57:42.127841 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:42.127754 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-73503b-598c798485-gpg6n" event={"ID":"2444a3b7-2746-42e5-821e-b0d6bcb8ee50","Type":"ContainerStarted","Data":"1569e87931e8c1064fc0ba9e80e967e28d91bd3c66601c7ebffc419b358d1279"} Oct 31 12:57:42.128759 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:42.128737 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-73503b-598c798485-gpg6n" Oct 31 12:57:42.129850 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:42.129827 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-73503b-598c798485-gpg6n" event={"ID":"2444a3b7-2746-42e5-821e-b0d6bcb8ee50","Type":"ContainerStarted","Data":"1035a7ff292792e6cb87dbe39e6d690448fa0a38c9742a53c88c165d784b63cf"} Oct 31 12:57:42.130419 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:42.130395 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:42.908069 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:42.908020 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:43.005751 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:43.005721 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" Oct 31 12:57:43.006220 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:43.006195 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" Oct 31 12:57:43.007090 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:43.007060 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" Oct 31 12:57:43.033346 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:43.033294 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-73503b-598c798485-gpg6n" podStartSLOduration=3.033279783 podStartE2EDuration="3.033279783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:57:42.15696051 +0000 UTC m=+1508.640083390" watchObservedRunningTime="2026-10-31 12:57:43.033279783 +0000 UTC m=+1509.516402686" Oct 31 12:57:43.634887 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:43.634760 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:45.578301 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:45.578265 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:47.725201 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:47.722014 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:48.009121 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:48.008115 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s"] Oct 31 12:57:48.110006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:48.089262 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s"] Oct 31 12:57:48.110006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:48.090277 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s" Oct 31 12:57:48.212041 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:48.160761 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-73503b-serving-cert\"" Oct 31 12:57:48.246052 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:48.240612 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fe694ed-e7f7-4a69-8c7a-f16d70f49211-metrics-certs\") pod \"maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s\" (UID: \"4fe694ed-e7f7-4a69-8c7a-f16d70f49211\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s" Oct 31 12:57:48.246052 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:48.240689 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fe694ed-e7f7-4a69-8c7a-f16d70f49211-maas-api-tls\") pod \"maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s\" (UID: \"4fe694ed-e7f7-4a69-8c7a-f16d70f49211\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s" Oct 31 12:57:48.246052 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:48.240736 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxdjj\" (UniqueName: \"kubernetes.io/projected/4fe694ed-e7f7-4a69-8c7a-f16d70f49211-kube-api-access-mxdjj\") pod \"maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s\" (UID: \"4fe694ed-e7f7-4a69-8c7a-f16d70f49211\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s" Oct 31 12:57:48.246052 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:48.240872 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe694ed-e7f7-4a69-8c7a-f16d70f49211-openshift-service-ca\") pod \"maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s\" (UID: \"4fe694ed-e7f7-4a69-8c7a-f16d70f49211\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s" Oct 31 12:57:48.386056 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:48.354881 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fe694ed-e7f7-4a69-8c7a-f16d70f49211-metrics-certs\") pod \"maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s\" (UID: \"4fe694ed-e7f7-4a69-8c7a-f16d70f49211\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s" Oct 31 12:57:48.386056 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:48.354963 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fe694ed-e7f7-4a69-8c7a-f16d70f49211-maas-api-tls\") pod \"maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s\" (UID: \"4fe694ed-e7f7-4a69-8c7a-f16d70f49211\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s" Oct 31 12:57:48.386056 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:48.355018 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxdjj\" (UniqueName: \"kubernetes.io/projected/4fe694ed-e7f7-4a69-8c7a-f16d70f49211-kube-api-access-mxdjj\") pod \"maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s\" (UID: \"4fe694ed-e7f7-4a69-8c7a-f16d70f49211\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s" Oct 31 12:57:48.386056 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:48.355156 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe694ed-e7f7-4a69-8c7a-f16d70f49211-openshift-service-ca\") pod \"maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s\" (UID: \"4fe694ed-e7f7-4a69-8c7a-f16d70f49211\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s" Oct 31 12:57:48.386056 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:48.356225 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe694ed-e7f7-4a69-8c7a-f16d70f49211-openshift-service-ca\") pod \"maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s\" (UID: \"4fe694ed-e7f7-4a69-8c7a-f16d70f49211\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s" Oct 31 12:57:48.416314 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:48.416278 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:48.599155 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:48.585163 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fe694ed-e7f7-4a69-8c7a-f16d70f49211-maas-api-tls\") pod \"maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s\" (UID: \"4fe694ed-e7f7-4a69-8c7a-f16d70f49211\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s" Oct 31 12:57:48.876046 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:48.870785 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxdjj\" (UniqueName: \"kubernetes.io/projected/4fe694ed-e7f7-4a69-8c7a-f16d70f49211-kube-api-access-mxdjj\") pod \"maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s\" (UID: \"4fe694ed-e7f7-4a69-8c7a-f16d70f49211\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s" Oct 31 12:57:48.893020 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:48.890148 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fe694ed-e7f7-4a69-8c7a-f16d70f49211-metrics-certs\") pod \"maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s\" (UID: \"4fe694ed-e7f7-4a69-8c7a-f16d70f49211\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s" Oct 31 12:57:49.061903 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:49.060666 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s" Oct 31 12:57:49.682264 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:49.673786 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:50.170010 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:50.117534 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s"] Oct 31 12:57:50.645186 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:50.644912 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:50.832008 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:57:50.830647 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:57:51.021233 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:51.019371 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s" event={"ID":"4fe694ed-e7f7-4a69-8c7a-f16d70f49211","Type":"ContainerStarted","Data":"44a83bf06124c3539169124bdabd1323d9b2e42f3ee1ba00367a891fa7a0bba9"} Oct 31 12:57:51.503003 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:51.500289 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:52.076584 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:52.076541 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s" event={"ID":"4fe694ed-e7f7-4a69-8c7a-f16d70f49211","Type":"ContainerStarted","Data":"8dae0e23192509123ba037dd2c143803835b7c6429be3dabf364f262e3003509"} Oct 31 12:57:52.078299 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:52.078275 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s" Oct 31 12:57:52.459679 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:52.459638 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:53.620583 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:53.620420 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:54.184073 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:54.183154 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-73503b-598c798485-gpg6n" Oct 31 12:57:54.206450 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:54.206400 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s" podStartSLOduration=6.326367879 podStartE2EDuration="7.206385662s" totalImagesPullingTime="880.017783ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:57:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:57:52.121661225 +0000 UTC m=+1518.604784127" watchObservedRunningTime="2026-10-31 12:57:54.206385662 +0000 UTC m=+1520.689508619" Oct 31 12:57:54.810176 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:54.810137 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:55.499313 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:55.499218 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:55.797391 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:55.797194 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:56.026620 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:56.026590 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:56.879570 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:56.879489 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:57.715777 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:57.714540 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:57.862180 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:57.862125 6841 patch_prober.go:29] interesting pod/e2e-mt-c4167730-openshift-default-578df84889-42vvd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:57:57.862385 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:57.862191 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" podUID="6b1dbe25-ef6f-43bb-a04a-6e497ecbf216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:58.442929 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:58.442710 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:59.219162 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:59.216606 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:57:59.219162 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:59.217828 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s" Oct 31 12:57:59.266685 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:59.266638 6841 generic.go:356] "Generic (PLEG): container finished" podID="6b1dbe25-ef6f-43bb-a04a-6e497ecbf216" containerID="ec182338bc3277355f454c782942169d58e8ab71578976ef8999c1c47e490a03" exitCode=0 Oct 31 12:57:59.266863 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:59.266723 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" event={"ID":"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216","Type":"ContainerDied","Data":"ec182338bc3277355f454c782942169d58e8ab71578976ef8999c1c47e490a03"} Oct 31 12:57:59.267282 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:59.267263 6841 scope.go:122] "RemoveContainer" containerID="ec182338bc3277355f454c782942169d58e8ab71578976ef8999c1c47e490a03" Oct 31 12:57:59.269021 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:59.268955 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:57:59.269212 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:59.269183 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:57:59.269290 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:59.269241 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:57:59.330488 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:59.330142 6841 patch_prober.go:29] interesting pod/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:57:59.330488 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:57:59.330200 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" podUID="d4f69dc4-ad91-4cdd-a98f-c3e27183a88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:58:00.083011 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:00.082475 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:00.276455 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:00.276419 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" event={"ID":"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216","Type":"ContainerStarted","Data":"a1e3bd6f94001f1595270cece6f834825385fb8995cfa9260fde377b24d14a3e"} Oct 31 12:58:00.278228 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:00.278201 6841 generic.go:356] "Generic (PLEG): container finished" podID="d4f69dc4-ad91-4cdd-a98f-c3e27183a88f" containerID="dc9d677ee8e7ea26192864987426be9d2678152d572be3bf1348616c7f6c01cb" exitCode=0 Oct 31 12:58:00.278360 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:00.278269 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" event={"ID":"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f","Type":"ContainerDied","Data":"dc9d677ee8e7ea26192864987426be9d2678152d572be3bf1348616c7f6c01cb"} Oct 31 12:58:00.278643 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:00.278630 6841 scope.go:122] "RemoveContainer" containerID="dc9d677ee8e7ea26192864987426be9d2678152d572be3bf1348616c7f6c01cb" Oct 31 12:58:00.280188 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:00.280162 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:58:00.280256 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:00.280232 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:58:00.280300 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:00.280282 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:58:00.576719 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:00.576655 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:58:00.576918 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:00.576793 6841 patch_prober.go:29] interesting pod/e2e-mt-c4167730-openshift-default-578df84889-42vvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 31 12:58:00.576918 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:00.576872 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" podUID="6b1dbe25-ef6f-43bb-a04a-6e497ecbf216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 31 12:58:00.844007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:00.835562 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:01.299830 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:01.299793 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" event={"ID":"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f","Type":"ContainerStarted","Data":"f4aa3d942d5a8aa117a1a8bda5297903dd08fade22254fd8a8933f1ad34edbd1"} Oct 31 12:58:01.558712 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:01.558625 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:58:01.559054 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:01.559031 6841 patch_prober.go:29] interesting pod/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 12:58:01.559226 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:01.559200 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" podUID="d4f69dc4-ad91-4cdd-a98f-c3e27183a88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 12:58:01.586024 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:01.585972 6841 patch_prober.go:29] interesting pod/e2e-mt-c4167730-openshift-default-578df84889-42vvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 31 12:58:01.586335 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:01.586290 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" podUID="6b1dbe25-ef6f-43bb-a04a-6e497ecbf216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 31 12:58:01.728419 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:01.728382 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:02.475682 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:02.475637 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:02.557212 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:02.557176 6841 patch_prober.go:29] interesting pod/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 12:58:02.557493 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:02.557464 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" podUID="d4f69dc4-ad91-4cdd-a98f-c3e27183a88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 12:58:02.576621 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:02.576585 6841 patch_prober.go:29] interesting pod/e2e-mt-c4167730-openshift-default-578df84889-42vvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 31 12:58:02.576898 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:02.576867 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" podUID="6b1dbe25-ef6f-43bb-a04a-6e497ecbf216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 31 12:58:03.227011 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:03.217889 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:03.567069 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:03.550403 6841 patch_prober.go:29] interesting pod/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 12:58:03.567069 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:03.550464 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" podUID="d4f69dc4-ad91-4cdd-a98f-c3e27183a88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 12:58:03.579017 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:03.576349 6841 patch_prober.go:29] interesting pod/e2e-mt-c4167730-openshift-default-578df84889-42vvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 31 12:58:03.579017 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:03.576409 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" podUID="6b1dbe25-ef6f-43bb-a04a-6e497ecbf216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 31 12:58:03.933760 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:03.933611 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:04.578018 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:04.564790 6841 patch_prober.go:29] interesting pod/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 12:58:04.578018 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:04.564853 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" podUID="d4f69dc4-ad91-4cdd-a98f-c3e27183a88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 12:58:04.580636 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:04.580601 6841 patch_prober.go:29] interesting pod/e2e-mt-c4167730-openshift-default-578df84889-42vvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 31 12:58:04.580895 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:04.580867 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" podUID="6b1dbe25-ef6f-43bb-a04a-6e497ecbf216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 31 12:58:04.624019 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:04.622306 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:05.376544 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:05.376505 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:05.554006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:05.553496 6841 patch_prober.go:29] interesting pod/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 12:58:05.554006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:05.553575 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" podUID="d4f69dc4-ad91-4cdd-a98f-c3e27183a88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 12:58:05.577461 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:05.577425 6841 patch_prober.go:29] interesting pod/e2e-mt-c4167730-openshift-default-578df84889-42vvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 31 12:58:05.577776 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:05.577731 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" podUID="6b1dbe25-ef6f-43bb-a04a-6e497ecbf216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 31 12:58:06.209960 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:06.209923 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:06.561471 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:06.561378 6841 patch_prober.go:29] interesting pod/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 12:58:06.561796 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:06.561763 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" podUID="d4f69dc4-ad91-4cdd-a98f-c3e27183a88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 12:58:06.580394 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:06.580366 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:58:06.580643 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:06.580621 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:58:06.581450 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:06.581431 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 12:58:06.731935 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:06.731898 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:07.002352 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:07.002304 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:07.221789 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:07.221756 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:07.554223 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:07.554193 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:58:07.554586 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:07.554564 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:58:07.555118 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:07.555099 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:58:08.162965 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:08.162932 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:08.914264 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:08.914217 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:09.618168 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:09.618127 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:09.816623 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:09.816585 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:10.330468 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:10.330427 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:11.289009 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:11.281759 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:12.027515 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:12.027465 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:12.407290 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:12.407189 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:13.163780 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:13.163723 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:14.043648 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:14.043610 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:15.028694 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:15.028654 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:15.820321 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:15.818614 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:16.751386 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:16.751352 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:17.608327 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:17.608288 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:18.332041 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:18.331999 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:18.579018 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:18.572598 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:19.506128 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:19.506095 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:19.593182 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:19.593145 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h"] Oct 31 12:58:19.599239 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:19.599213 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" Oct 31 12:58:19.629819 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:19.629791 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h"] Oct 31 12:58:19.675908 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:19.675876 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e3d6506-3350-4dd0-bdf1-efb951353ba5-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h\" (UID: \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" Oct 31 12:58:19.676123 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:19.675917 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e3d6506-3350-4dd0-bdf1-efb951353ba5-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h\" (UID: \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" Oct 31 12:58:19.676123 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:19.675938 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3d6506-3350-4dd0-bdf1-efb951353ba5-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h\" (UID: \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" Oct 31 12:58:19.676123 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:19.676004 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e3d6506-3350-4dd0-bdf1-efb951353ba5-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h\" (UID: \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" Oct 31 12:58:19.676280 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:19.676216 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3d6506-3350-4dd0-bdf1-efb951353ba5-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h\" (UID: \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" Oct 31 12:58:19.676325 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:19.676307 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9kcl\" (UniqueName: \"kubernetes.io/projected/2e3d6506-3350-4dd0-bdf1-efb951353ba5-kube-api-access-q9kcl\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h\" (UID: \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" Oct 31 12:58:19.676465 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:19.676447 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e3d6506-3350-4dd0-bdf1-efb951353ba5-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h\" (UID: \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" Oct 31 12:58:19.676525 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:19.676505 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e3d6506-3350-4dd0-bdf1-efb951353ba5-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h\" (UID: \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" Oct 31 12:58:19.676672 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:19.676652 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e3d6506-3350-4dd0-bdf1-efb951353ba5-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h\" (UID: \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" Oct 31 12:58:19.778597 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:19.778515 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e3d6506-3350-4dd0-bdf1-efb951353ba5-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h\" (UID: \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" Oct 31 12:58:19.778767 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:19.778604 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e3d6506-3350-4dd0-bdf1-efb951353ba5-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h\" (UID: \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" Oct 31 12:58:19.778767 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:19.778632 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e3d6506-3350-4dd0-bdf1-efb951353ba5-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h\" (UID: \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" Oct 31 12:58:19.778767 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:19.778674 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3d6506-3350-4dd0-bdf1-efb951353ba5-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h\" (UID: \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" Oct 31 12:58:19.778767 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:19.778742 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e3d6506-3350-4dd0-bdf1-efb951353ba5-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h\" (UID: \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" Oct 31 12:58:19.778973 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:19.778877 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3d6506-3350-4dd0-bdf1-efb951353ba5-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h\" (UID: \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" Oct 31 12:58:19.778973 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:19.778952 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9kcl\" (UniqueName: \"kubernetes.io/projected/2e3d6506-3350-4dd0-bdf1-efb951353ba5-kube-api-access-q9kcl\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h\" (UID: \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" Oct 31 12:58:19.779735 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:19.779093 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e3d6506-3350-4dd0-bdf1-efb951353ba5-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h\" (UID: \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" Oct 31 12:58:19.779735 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:19.779151 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e3d6506-3350-4dd0-bdf1-efb951353ba5-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h\" (UID: \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" Oct 31 12:58:19.779735 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:19.779220 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3d6506-3350-4dd0-bdf1-efb951353ba5-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h\" (UID: \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" Oct 31 12:58:19.779735 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:19.779438 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e3d6506-3350-4dd0-bdf1-efb951353ba5-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h\" (UID: \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" Oct 31 12:58:19.779735 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:19.779506 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e3d6506-3350-4dd0-bdf1-efb951353ba5-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h\" (UID: \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" Oct 31 12:58:19.779735 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:19.779586 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3d6506-3350-4dd0-bdf1-efb951353ba5-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h\" (UID: \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" Oct 31 12:58:19.779735 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:19.779650 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e3d6506-3350-4dd0-bdf1-efb951353ba5-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h\" (UID: \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" Oct 31 12:58:19.781439 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:19.781409 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e3d6506-3350-4dd0-bdf1-efb951353ba5-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h\" (UID: \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" Oct 31 12:58:19.781578 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:19.781462 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e3d6506-3350-4dd0-bdf1-efb951353ba5-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h\" (UID: \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" Oct 31 12:58:19.793585 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:19.793557 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9kcl\" (UniqueName: \"kubernetes.io/projected/2e3d6506-3350-4dd0-bdf1-efb951353ba5-kube-api-access-q9kcl\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h\" (UID: \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" Oct 31 12:58:19.794136 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:19.794113 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e3d6506-3350-4dd0-bdf1-efb951353ba5-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h\" (UID: \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" Oct 31 12:58:19.913557 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:19.913513 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" Oct 31 12:58:20.041957 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:58:20.041878 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:58:20.222380 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:58:20.222345 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e3d6506_3350_4dd0_bdf1_efb951353ba5.slice/crio-8cc8fdb7d1216d879075fcde4fe03b078a267fab744b2ca5717a3eb60d903a08 WatchSource:0}: Error finding container 8cc8fdb7d1216d879075fcde4fe03b078a267fab744b2ca5717a3eb60d903a08: Status 404 returned error can't find the container with id 8cc8fdb7d1216d879075fcde4fe03b078a267fab744b2ca5717a3eb60d903a08 Oct 31 12:58:20.222513 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:20.222394 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h"] Oct 31 12:58:20.225195 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:20.225163 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:58:20.225302 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:20.225228 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:58:20.225302 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:20.225260 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:58:20.466380 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:20.466140 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:20.517121 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:20.517085 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" event={"ID":"2e3d6506-3350-4dd0-bdf1-efb951353ba5","Type":"ContainerStarted","Data":"96d3e66485c0b36a610ac6bcdba9a79e8d7b6fc9fb5b0d16d6a1ae50290dc257"} Oct 31 12:58:20.517525 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:20.517142 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" event={"ID":"2e3d6506-3350-4dd0-bdf1-efb951353ba5","Type":"ContainerStarted","Data":"8cc8fdb7d1216d879075fcde4fe03b078a267fab744b2ca5717a3eb60d903a08"} Oct 31 12:58:20.543677 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:20.543630 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" podStartSLOduration=1.543616282 podStartE2EDuration="1.543616282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:58:20.54143453 +0000 UTC m=+1547.024557453" watchObservedRunningTime="2026-10-31 12:58:20.543616282 +0000 UTC m=+1547.026739185" Oct 31 12:58:20.737829 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:20.737777 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:20.914063 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:20.914021 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" Oct 31 12:58:21.350007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:21.342116 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:21.916114 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:21.916071 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 12:58:21.916509 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:21.916125 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" podUID="2e3d6506-3350-4dd0-bdf1-efb951353ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 12:58:22.180061 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:22.179834 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:22.914419 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:22.914385 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 12:58:22.914571 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:22.914441 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" podUID="2e3d6506-3350-4dd0-bdf1-efb951353ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 12:58:23.138314 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:23.138276 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:23.914472 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:23.914439 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 12:58:23.914633 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:23.914490 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" podUID="2e3d6506-3350-4dd0-bdf1-efb951353ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 12:58:24.065636 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:24.065599 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:24.915016 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:24.914568 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 12:58:24.915016 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:24.914639 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" podUID="2e3d6506-3350-4dd0-bdf1-efb951353ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 12:58:24.979282 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:24.977640 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:25.675176 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:25.675140 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:25.917025 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:25.914133 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 12:58:25.917025 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:25.914198 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" podUID="2e3d6506-3350-4dd0-bdf1-efb951353ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 12:58:26.220004 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:26.215577 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:26.914866 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:26.914371 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 12:58:26.914866 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:26.914451 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" podUID="2e3d6506-3350-4dd0-bdf1-efb951353ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 12:58:26.943350 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:26.940419 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:27.609453 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:27.603368 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:27.906973 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:27.906880 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:28.136208 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:28.136145 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:28.318003 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:28.309292 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" Oct 31 12:58:28.341056 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:28.341019 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" Oct 31 12:58:28.355874 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:28.355783 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" Oct 31 12:58:28.361151 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:28.360902 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:28.531518 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:28.531480 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f"] Oct 31 12:58:28.531718 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:28.531696 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" podUID="d4f69dc4-ad91-4cdd-a98f-c3e27183a88f" containerName="istio-proxy" containerID="cri-o://f4aa3d942d5a8aa117a1a8bda5297903dd08fade22254fd8a8933f1ad34edbd1" gracePeriod=30 Oct 31 12:58:28.704000 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:28.697636 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:29.247009 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:29.241161 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:29.453379 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:29.453341 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:30.142635 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:30.142597 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:30.695794 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:30.695758 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:31.003620 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:31.003534 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:31.267700 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:31.267609 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:31.470443 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:31.470408 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:31.916457 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:31.916371 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:32.622642 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:32.622607 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:33.338159 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:33.338058 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:33.745172 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:33.745139 6841 generic.go:356] "Generic (PLEG): container finished" podID="d4f69dc4-ad91-4cdd-a98f-c3e27183a88f" containerID="f4aa3d942d5a8aa117a1a8bda5297903dd08fade22254fd8a8933f1ad34edbd1" exitCode=0 Oct 31 12:58:33.745385 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:33.745366 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" event={"ID":"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f","Type":"ContainerDied","Data":"f4aa3d942d5a8aa117a1a8bda5297903dd08fade22254fd8a8933f1ad34edbd1"} Oct 31 12:58:33.745467 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:33.745433 6841 scope.go:122] "RemoveContainer" containerID="dc9d677ee8e7ea26192864987426be9d2678152d572be3bf1348616c7f6c01cb" Oct 31 12:58:33.819941 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:33.819917 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:58:34.001497 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:34.001460 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-istio-podinfo\") pod \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\" (UID: \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\") " Oct 31 12:58:34.001686 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:34.001524 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-workload-socket\") pod \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\" (UID: \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\") " Oct 31 12:58:34.001686 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:34.001545 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-istiod-ca-cert\") pod \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\" (UID: \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\") " Oct 31 12:58:34.001686 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:34.001593 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-istio-data\") pod \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\" (UID: \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\") " Oct 31 12:58:34.001686 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:34.001626 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-istio-token\" (UniqueName: \"kubernetes.io/projected/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-istio-token\") pod \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\" (UID: \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\") " Oct 31 12:58:34.001917 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:34.001719 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-workload-certs\") pod \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\" (UID: \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\") " Oct 31 12:58:34.001917 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:34.001807 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-istio-envoy\") pod \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\" (UID: \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\") " Oct 31 12:58:34.002076 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:34.001915 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-workload-socket" pod "d4f69dc4-ad91-4cdd-a98f-c3e27183a88f" (UID: "d4f69dc4-ad91-4cdd-a98f-c3e27183a88f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:58:34.002076 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:34.002057 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-istio-data" pod "d4f69dc4-ad91-4cdd-a98f-c3e27183a88f" (UID: "d4f69dc4-ad91-4cdd-a98f-c3e27183a88f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:58:34.002182 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:34.002075 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-workload-certs" pod "d4f69dc4-ad91-4cdd-a98f-c3e27183a88f" (UID: "d4f69dc4-ad91-4cdd-a98f-c3e27183a88f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:58:34.002182 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:34.002112 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-kube-api-access-5pw45\" (UniqueName: \"kubernetes.io/projected/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-kube-api-access-5pw45\") pod \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\" (UID: \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\") " Oct 31 12:58:34.002182 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:34.002134 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-istiod-ca-cert" pod "d4f69dc4-ad91-4cdd-a98f-c3e27183a88f" (UID: "d4f69dc4-ad91-4cdd-a98f-c3e27183a88f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:58:34.002336 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:34.002188 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-credential-socket\") pod \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\" (UID: \"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f\") " Oct 31 12:58:34.002494 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:34.002470 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-credential-socket" pod "d4f69dc4-ad91-4cdd-a98f-c3e27183a88f" (UID: "d4f69dc4-ad91-4cdd-a98f-c3e27183a88f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:58:34.003031 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:34.002975 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:34.003175 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:34.003035 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:34.003175 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:34.003051 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:34.003175 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:34.003066 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:34.003175 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:34.003080 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:34.003861 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:34.003835 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-istio-podinfo" pod "d4f69dc4-ad91-4cdd-a98f-c3e27183a88f" (UID: "d4f69dc4-ad91-4cdd-a98f-c3e27183a88f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:58:34.004106 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:34.004082 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-istio-token" pod "d4f69dc4-ad91-4cdd-a98f-c3e27183a88f" (UID: "d4f69dc4-ad91-4cdd-a98f-c3e27183a88f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:58:34.004199 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:34.004116 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-istio-envoy" pod "d4f69dc4-ad91-4cdd-a98f-c3e27183a88f" (UID: "d4f69dc4-ad91-4cdd-a98f-c3e27183a88f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:58:34.004199 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:34.004144 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-kube-api-access-5pw45" pod "d4f69dc4-ad91-4cdd-a98f-c3e27183a88f" (UID: "d4f69dc4-ad91-4cdd-a98f-c3e27183a88f"). InnerVolumeSpecName "kube-api-access-5pw45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:58:34.104728 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:34.104701 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:34.104728 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:34.104727 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:34.104728 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:34.104736 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:34.104963 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:34.104745 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pw45\" (UniqueName: \"kubernetes.io/projected/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f-kube-api-access-5pw45\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:34.272445 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:34.272352 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:34.753785 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:34.753758 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" Oct 31 12:58:34.753785 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:34.753775 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f" event={"ID":"d4f69dc4-ad91-4cdd-a98f-c3e27183a88f","Type":"ContainerDied","Data":"3286ee0b87c6698394df37a8522c357fc0e11cf123f9ad47e3cc25ddf31ea3ef"} Oct 31 12:58:34.754324 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:34.753839 6841 scope.go:122] "RemoveContainer" containerID="f4aa3d942d5a8aa117a1a8bda5297903dd08fade22254fd8a8933f1ad34edbd1" Oct 31 12:58:34.797250 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:34.797217 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f"] Oct 31 12:58:34.800457 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:34.800430 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-702419-openshift-default-7b8c755667-cpq6f"] Oct 31 12:58:34.862741 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:34.862689 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:35.160249 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:35.160159 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:35.388346 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:35.388310 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:35.643543 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:35.643501 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4f69dc4-ad91-4cdd-a98f-c3e27183a88f" path="/var/lib/kubelet/pods/d4f69dc4-ad91-4cdd-a98f-c3e27183a88f/volumes" Oct 31 12:58:35.690807 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:35.690762 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:36.042169 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:36.041355 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:36.385211 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:36.385123 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:36.613796 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:36.613764 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-15c8ff-6d795ffc67-vlfh6"] Oct 31 12:58:36.614465 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:36.614448 6841 cpu_manager.go:414] "containerMap: removing container" podUID="d4f69dc4-ad91-4cdd-a98f-c3e27183a88f" containerName="istio-proxy" Oct 31 12:58:36.614519 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:36.614466 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4f69dc4-ad91-4cdd-a98f-c3e27183a88f" containerName="istio-proxy" Oct 31 12:58:36.614519 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:36.614488 6841 cpu_manager.go:414] "containerMap: removing container" podUID="d4f69dc4-ad91-4cdd-a98f-c3e27183a88f" containerName="istio-proxy" Oct 31 12:58:36.614519 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:36.614493 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4f69dc4-ad91-4cdd-a98f-c3e27183a88f" containerName="istio-proxy" Oct 31 12:58:36.614619 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:36.614607 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4f69dc4-ad91-4cdd-a98f-c3e27183a88f" containerName="istio-proxy" Oct 31 12:58:36.614651 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:36.614619 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4f69dc4-ad91-4cdd-a98f-c3e27183a88f" containerName="istio-proxy" Oct 31 12:58:36.619707 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:36.619692 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-15c8ff-6d795ffc67-vlfh6" Oct 31 12:58:36.632367 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:36.632342 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-15c8ff\"" Oct 31 12:58:36.637710 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:36.635209 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-15c8ff-dockercfg-rgkjk\"" Oct 31 12:58:36.637710 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:36.637183 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-15c8ff-6d795ffc67-vlfh6"] Oct 31 12:58:36.653718 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:36.653689 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-15c8ff-7579cfb7c7-nnxnw"] Oct 31 12:58:36.658307 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:36.658283 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-15c8ff-7579cfb7c7-nnxnw" Oct 31 12:58:36.672581 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:36.672552 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:36.676336 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:36.676279 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-15c8ff-7579cfb7c7-nnxnw"] Oct 31 12:58:36.682830 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:36.682802 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a16b3c1-9c3f-4019-83c6-3ed3d085bc14-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-15c8ff-6d795ffc67-vlfh6\" (UID: \"5a16b3c1-9c3f-4019-83c6-3ed3d085bc14\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-15c8ff-6d795ffc67-vlfh6" Oct 31 12:58:36.682956 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:36.682873 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f62tl\" (UniqueName: \"kubernetes.io/projected/5a16b3c1-9c3f-4019-83c6-3ed3d085bc14-kube-api-access-f62tl\") pod \"payload-pre-processing-e2e-api-a-15c8ff-6d795ffc67-vlfh6\" (UID: \"5a16b3c1-9c3f-4019-83c6-3ed3d085bc14\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-15c8ff-6d795ffc67-vlfh6" Oct 31 12:58:36.851003 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:36.847074 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv6vp\" (UniqueName: \"kubernetes.io/projected/8efcc86a-0f1a-4e90-8763-fb436dfa7bfb-kube-api-access-gv6vp\") pod \"payload-processing-e2e-api-a-15c8ff-7579cfb7c7-nnxnw\" (UID: \"8efcc86a-0f1a-4e90-8763-fb436dfa7bfb\") " pod="openshift-ingress/payload-processing-e2e-api-a-15c8ff-7579cfb7c7-nnxnw" Oct 31 12:58:36.851003 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:36.847134 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a16b3c1-9c3f-4019-83c6-3ed3d085bc14-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-15c8ff-6d795ffc67-vlfh6\" (UID: \"5a16b3c1-9c3f-4019-83c6-3ed3d085bc14\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-15c8ff-6d795ffc67-vlfh6" Oct 31 12:58:36.851003 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:36.847184 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f62tl\" (UniqueName: \"kubernetes.io/projected/5a16b3c1-9c3f-4019-83c6-3ed3d085bc14-kube-api-access-f62tl\") pod \"payload-pre-processing-e2e-api-a-15c8ff-6d795ffc67-vlfh6\" (UID: \"5a16b3c1-9c3f-4019-83c6-3ed3d085bc14\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-15c8ff-6d795ffc67-vlfh6" Oct 31 12:58:36.851003 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:36.847215 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8efcc86a-0f1a-4e90-8763-fb436dfa7bfb-plugins-config-volume\") pod \"payload-processing-e2e-api-a-15c8ff-7579cfb7c7-nnxnw\" (UID: \"8efcc86a-0f1a-4e90-8763-fb436dfa7bfb\") " pod="openshift-ingress/payload-processing-e2e-api-a-15c8ff-7579cfb7c7-nnxnw" Oct 31 12:58:36.877007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:36.876232 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a16b3c1-9c3f-4019-83c6-3ed3d085bc14-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-15c8ff-6d795ffc67-vlfh6\" (UID: \"5a16b3c1-9c3f-4019-83c6-3ed3d085bc14\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-15c8ff-6d795ffc67-vlfh6" Oct 31 12:58:36.925059 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:36.922727 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:36.952142 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:36.936468 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-73503b-598c798485-gpg6n"] Oct 31 12:58:36.952142 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:36.952090 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv6vp\" (UniqueName: \"kubernetes.io/projected/8efcc86a-0f1a-4e90-8763-fb436dfa7bfb-kube-api-access-gv6vp\") pod \"payload-processing-e2e-api-a-15c8ff-7579cfb7c7-nnxnw\" (UID: \"8efcc86a-0f1a-4e90-8763-fb436dfa7bfb\") " pod="openshift-ingress/payload-processing-e2e-api-a-15c8ff-7579cfb7c7-nnxnw" Oct 31 12:58:36.952425 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:36.952164 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8efcc86a-0f1a-4e90-8763-fb436dfa7bfb-plugins-config-volume\") pod \"payload-processing-e2e-api-a-15c8ff-7579cfb7c7-nnxnw\" (UID: \"8efcc86a-0f1a-4e90-8763-fb436dfa7bfb\") " pod="openshift-ingress/payload-processing-e2e-api-a-15c8ff-7579cfb7c7-nnxnw" Oct 31 12:58:36.963391 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:36.953277 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8efcc86a-0f1a-4e90-8763-fb436dfa7bfb-plugins-config-volume\") pod \"payload-processing-e2e-api-a-15c8ff-7579cfb7c7-nnxnw\" (UID: \"8efcc86a-0f1a-4e90-8763-fb436dfa7bfb\") " pod="openshift-ingress/payload-processing-e2e-api-a-15c8ff-7579cfb7c7-nnxnw" Oct 31 12:58:37.035197 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:37.031719 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv6vp\" (UniqueName: \"kubernetes.io/projected/8efcc86a-0f1a-4e90-8763-fb436dfa7bfb-kube-api-access-gv6vp\") pod \"payload-processing-e2e-api-a-15c8ff-7579cfb7c7-nnxnw\" (UID: \"8efcc86a-0f1a-4e90-8763-fb436dfa7bfb\") " pod="openshift-ingress/payload-processing-e2e-api-a-15c8ff-7579cfb7c7-nnxnw" Oct 31 12:58:37.043120 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:37.043080 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f62tl\" (UniqueName: \"kubernetes.io/projected/5a16b3c1-9c3f-4019-83c6-3ed3d085bc14-kube-api-access-f62tl\") pod \"payload-pre-processing-e2e-api-a-15c8ff-6d795ffc67-vlfh6\" (UID: \"5a16b3c1-9c3f-4019-83c6-3ed3d085bc14\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-15c8ff-6d795ffc67-vlfh6" Oct 31 12:58:37.157004 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:37.151058 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:37.241010 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:37.234790 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-15c8ff-6d795ffc67-vlfh6" Oct 31 12:58:37.274247 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:37.274212 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-15c8ff-7579cfb7c7-nnxnw" Oct 31 12:58:37.386218 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:37.385944 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:37.585138 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:37.585053 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:37.624511 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:58:37.624480 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8efcc86a_0f1a_4e90_8763_fb436dfa7bfb.slice/crio-f51159d3998cbf40c10cc61111af6b65e02b369439380d521804b328d7763f00 WatchSource:0}: Error finding container f51159d3998cbf40c10cc61111af6b65e02b369439380d521804b328d7763f00: Status 404 returned error can't find the container with id f51159d3998cbf40c10cc61111af6b65e02b369439380d521804b328d7763f00 Oct 31 12:58:37.625736 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:58:37.625712 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a16b3c1_9c3f_4019_83c6_3ed3d085bc14.slice/crio-36fb544ebd913a3c095ac45ec329d0ab4169ecbf9ccf19291bb5e9dc04711297 WatchSource:0}: Error finding container 36fb544ebd913a3c095ac45ec329d0ab4169ecbf9ccf19291bb5e9dc04711297: Status 404 returned error can't find the container with id 36fb544ebd913a3c095ac45ec329d0ab4169ecbf9ccf19291bb5e9dc04711297 Oct 31 12:58:37.625974 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:37.625952 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-15c8ff-6d795ffc67-vlfh6"] Oct 31 12:58:37.630360 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:37.630328 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-15c8ff-7579cfb7c7-nnxnw"] Oct 31 12:58:37.895567 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:37.895459 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-15c8ff-7579cfb7c7-nnxnw" event={"ID":"8efcc86a-0f1a-4e90-8763-fb436dfa7bfb","Type":"ContainerStarted","Data":"5a2fc796f276ec2cb96331ef5dfa0f1952a1e6eef6e9d403493dc61b582f2c1f"} Oct 31 12:58:37.895567 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:37.895527 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-15c8ff-7579cfb7c7-nnxnw" Oct 31 12:58:37.895567 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:37.895551 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-15c8ff-7579cfb7c7-nnxnw" event={"ID":"8efcc86a-0f1a-4e90-8763-fb436dfa7bfb","Type":"ContainerStarted","Data":"f51159d3998cbf40c10cc61111af6b65e02b369439380d521804b328d7763f00"} Oct 31 12:58:37.897170 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:37.897146 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-15c8ff-6d795ffc67-vlfh6" event={"ID":"5a16b3c1-9c3f-4019-83c6-3ed3d085bc14","Type":"ContainerStarted","Data":"f67d4e2f724a0ea6f370c9cec883f7cb5acf20882d4ab7b36775600ed7dcfb13"} Oct 31 12:58:37.897301 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:37.897188 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-15c8ff-6d795ffc67-vlfh6" event={"ID":"5a16b3c1-9c3f-4019-83c6-3ed3d085bc14","Type":"ContainerStarted","Data":"36fb544ebd913a3c095ac45ec329d0ab4169ecbf9ccf19291bb5e9dc04711297"} Oct 31 12:58:37.897301 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:37.897214 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-15c8ff-6d795ffc67-vlfh6" Oct 31 12:58:37.916536 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:37.916492 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-15c8ff-7579cfb7c7-nnxnw" podStartSLOduration=1.916479051 podStartE2EDuration="1.916479051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:58:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:58:37.914058022 +0000 UTC m=+1564.397180938" watchObservedRunningTime="2026-10-31 12:58:37.916479051 +0000 UTC m=+1564.399601951" Oct 31 12:58:37.938756 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:37.938698 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-15c8ff-6d795ffc67-vlfh6" podStartSLOduration=1.938684226 podStartE2EDuration="1.938684226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:58:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:58:37.938492733 +0000 UTC m=+1564.421615635" watchObservedRunningTime="2026-10-31 12:58:37.938684226 +0000 UTC m=+1564.421807126" Oct 31 12:58:38.532903 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:38.532862 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-6x7qm"] Oct 31 12:58:38.578411 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:38.578379 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-2wthh"] Oct 31 12:58:38.628687 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:38.628618 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:38.858685 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:38.858599 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:39.732064 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:39.728598 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:40.793546 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:40.793517 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:41.231057 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:41.229330 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:41.862708 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:41.862674 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn"] Oct 31 12:58:41.867456 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:41.867424 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn" Oct 31 12:58:41.876181 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:41.876132 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-15c8ff-serving-cert\"" Oct 31 12:58:41.881939 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:41.881707 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn"] Oct 31 12:58:41.937007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:41.936957 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-73503b-598c798485-gpg6n" podUID="2444a3b7-2746-42e5-821e-b0d6bcb8ee50" containerName="payload-processing" containerID="cri-o://1569e87931e8c1064fc0ba9e80e967e28d91bd3c66601c7ebffc419b358d1279" gracePeriod=25 Oct 31 12:58:41.955673 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:41.955645 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwntb\" (UniqueName: \"kubernetes.io/projected/c14527eb-216b-4d44-a5a0-c9de5de775b1-kube-api-access-zwntb\") pod \"maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn\" (UID: \"c14527eb-216b-4d44-a5a0-c9de5de775b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn" Oct 31 12:58:41.955831 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:41.955813 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c14527eb-216b-4d44-a5a0-c9de5de775b1-metrics-certs\") pod \"maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn\" (UID: \"c14527eb-216b-4d44-a5a0-c9de5de775b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn" Oct 31 12:58:41.955871 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:41.955857 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c14527eb-216b-4d44-a5a0-c9de5de775b1-openshift-service-ca\") pod \"maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn\" (UID: \"c14527eb-216b-4d44-a5a0-c9de5de775b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn" Oct 31 12:58:41.955972 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:41.955957 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c14527eb-216b-4d44-a5a0-c9de5de775b1-maas-api-tls\") pod \"maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn\" (UID: \"c14527eb-216b-4d44-a5a0-c9de5de775b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn" Oct 31 12:58:42.011090 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:42.011049 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:42.057662 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:42.057637 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c14527eb-216b-4d44-a5a0-c9de5de775b1-metrics-certs\") pod \"maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn\" (UID: \"c14527eb-216b-4d44-a5a0-c9de5de775b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn" Oct 31 12:58:42.057760 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:42.057676 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c14527eb-216b-4d44-a5a0-c9de5de775b1-openshift-service-ca\") pod \"maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn\" (UID: \"c14527eb-216b-4d44-a5a0-c9de5de775b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn" Oct 31 12:58:42.057910 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:42.057882 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c14527eb-216b-4d44-a5a0-c9de5de775b1-maas-api-tls\") pod \"maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn\" (UID: \"c14527eb-216b-4d44-a5a0-c9de5de775b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn" Oct 31 12:58:42.058186 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:42.058166 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwntb\" (UniqueName: \"kubernetes.io/projected/c14527eb-216b-4d44-a5a0-c9de5de775b1-kube-api-access-zwntb\") pod \"maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn\" (UID: \"c14527eb-216b-4d44-a5a0-c9de5de775b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn" Oct 31 12:58:42.058427 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:42.058405 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c14527eb-216b-4d44-a5a0-c9de5de775b1-openshift-service-ca\") pod \"maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn\" (UID: \"c14527eb-216b-4d44-a5a0-c9de5de775b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn" Oct 31 12:58:42.060056 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:42.060026 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c14527eb-216b-4d44-a5a0-c9de5de775b1-metrics-certs\") pod \"maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn\" (UID: \"c14527eb-216b-4d44-a5a0-c9de5de775b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn" Oct 31 12:58:42.060145 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:42.060107 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c14527eb-216b-4d44-a5a0-c9de5de775b1-maas-api-tls\") pod \"maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn\" (UID: \"c14527eb-216b-4d44-a5a0-c9de5de775b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn" Oct 31 12:58:42.120313 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:42.112145 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwntb\" (UniqueName: \"kubernetes.io/projected/c14527eb-216b-4d44-a5a0-c9de5de775b1-kube-api-access-zwntb\") pod \"maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn\" (UID: \"c14527eb-216b-4d44-a5a0-c9de5de775b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn" Oct 31 12:58:42.191955 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:42.191926 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn" Oct 31 12:58:42.262723 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:42.262586 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:42.295661 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:42.295635 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-73503b-598c798485-gpg6n" Oct 31 12:58:42.377472 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:42.377393 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2444a3b7-2746-42e5-821e-b0d6bcb8ee50-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2444a3b7-2746-42e5-821e-b0d6bcb8ee50-plugins-config-volume\") pod \"2444a3b7-2746-42e5-821e-b0d6bcb8ee50\" (UID: \"2444a3b7-2746-42e5-821e-b0d6bcb8ee50\") " Oct 31 12:58:42.377617 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:42.377590 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2444a3b7-2746-42e5-821e-b0d6bcb8ee50-kube-api-access-6xvfv\" (UniqueName: \"kubernetes.io/projected/2444a3b7-2746-42e5-821e-b0d6bcb8ee50-kube-api-access-6xvfv\") pod \"2444a3b7-2746-42e5-821e-b0d6bcb8ee50\" (UID: \"2444a3b7-2746-42e5-821e-b0d6bcb8ee50\") " Oct 31 12:58:42.377795 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:42.377768 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2444a3b7-2746-42e5-821e-b0d6bcb8ee50-plugins-config-volume" pod "2444a3b7-2746-42e5-821e-b0d6bcb8ee50" (UID: "2444a3b7-2746-42e5-821e-b0d6bcb8ee50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:58:42.378195 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:42.378170 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2444a3b7-2746-42e5-821e-b0d6bcb8ee50-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:42.379728 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:42.379699 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2444a3b7-2746-42e5-821e-b0d6bcb8ee50-kube-api-access-6xvfv" pod "2444a3b7-2746-42e5-821e-b0d6bcb8ee50" (UID: "2444a3b7-2746-42e5-821e-b0d6bcb8ee50"). InnerVolumeSpecName "kube-api-access-6xvfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:58:42.506131 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:42.501222 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xvfv\" (UniqueName: \"kubernetes.io/projected/2444a3b7-2746-42e5-821e-b0d6bcb8ee50-kube-api-access-6xvfv\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:42.535676 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:42.535526 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn"] Oct 31 12:58:42.941125 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:42.941085 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:43.006290 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:43.006251 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn" event={"ID":"c14527eb-216b-4d44-a5a0-c9de5de775b1","Type":"ContainerStarted","Data":"22a008ed990c7d75b9e338a48cd893800cc23702d09466b7546e355273064285"} Oct 31 12:58:43.007843 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:43.007810 6841 generic.go:356] "Generic (PLEG): container finished" podID="2444a3b7-2746-42e5-821e-b0d6bcb8ee50" containerID="1569e87931e8c1064fc0ba9e80e967e28d91bd3c66601c7ebffc419b358d1279" exitCode=0 Oct 31 12:58:43.008011 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:43.007876 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-73503b-598c798485-gpg6n" Oct 31 12:58:43.008011 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:43.007873 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-73503b-598c798485-gpg6n" event={"ID":"2444a3b7-2746-42e5-821e-b0d6bcb8ee50","Type":"ContainerDied","Data":"1569e87931e8c1064fc0ba9e80e967e28d91bd3c66601c7ebffc419b358d1279"} Oct 31 12:58:43.008144 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:43.008016 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-73503b-598c798485-gpg6n" event={"ID":"2444a3b7-2746-42e5-821e-b0d6bcb8ee50","Type":"ContainerDied","Data":"1035a7ff292792e6cb87dbe39e6d690448fa0a38c9742a53c88c165d784b63cf"} Oct 31 12:58:43.008144 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:43.008051 6841 scope.go:122] "RemoveContainer" containerID="1569e87931e8c1064fc0ba9e80e967e28d91bd3c66601c7ebffc419b358d1279" Oct 31 12:58:43.018594 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:43.018577 6841 scope.go:122] "RemoveContainer" containerID="1569e87931e8c1064fc0ba9e80e967e28d91bd3c66601c7ebffc419b358d1279" Oct 31 12:58:43.018868 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:58:43.018850 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1569e87931e8c1064fc0ba9e80e967e28d91bd3c66601c7ebffc419b358d1279\": container with ID starting with 1569e87931e8c1064fc0ba9e80e967e28d91bd3c66601c7ebffc419b358d1279 not found: ID does not exist" containerID="1569e87931e8c1064fc0ba9e80e967e28d91bd3c66601c7ebffc419b358d1279" Oct 31 12:58:43.018908 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:43.018876 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1569e87931e8c1064fc0ba9e80e967e28d91bd3c66601c7ebffc419b358d1279"} err="failed to get container status \"1569e87931e8c1064fc0ba9e80e967e28d91bd3c66601c7ebffc419b358d1279\": rpc error: code = NotFound desc = could not find container \"1569e87931e8c1064fc0ba9e80e967e28d91bd3c66601c7ebffc419b358d1279\": container with ID starting with 1569e87931e8c1064fc0ba9e80e967e28d91bd3c66601c7ebffc419b358d1279 not found: ID does not exist" Oct 31 12:58:43.033877 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:43.033851 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-73503b-598c798485-gpg6n"] Oct 31 12:58:43.038909 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:43.038886 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-73503b-598c798485-gpg6n"] Oct 31 12:58:43.373699 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:43.373656 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s"] Oct 31 12:58:43.374200 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:43.373905 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s" podUID="4fe694ed-e7f7-4a69-8c7a-f16d70f49211" containerName="maas-api" containerID="cri-o://8dae0e23192509123ba037dd2c143803835b7c6429be3dabf364f262e3003509" gracePeriod=30 Oct 31 12:58:43.530785 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:43.528198 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:43.533420 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:43.533382 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-6x7qm" podUID="2d8bdee0-d436-425d-8b4d-53337f773d27" containerName="payload-processing" containerID="cri-o://64be69590e1308cc2f9e44fa61e7bc61364928f40cbd743d63e2d18d74ce8195" gracePeriod=25 Oct 31 12:58:43.608029 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:43.598097 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-2wthh" podUID="158edcbd-2810-4cf7-a2e5-2a881ae0568e" containerName="payload-pre-processing" containerID="cri-o://2f18b66c3e153ad6cb29b8d224c86d5afc857aba8d9c665054491381f590fe0e" gracePeriod=25 Oct 31 12:58:43.810470 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:43.751675 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:43.824012 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:43.823496 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2444a3b7-2746-42e5-821e-b0d6bcb8ee50" path="/var/lib/kubelet/pods/2444a3b7-2746-42e5-821e-b0d6bcb8ee50/volumes" Oct 31 12:58:43.826216 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:43.825890 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s" Oct 31 12:58:43.948048 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:43.946648 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fe694ed-e7f7-4a69-8c7a-f16d70f49211-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fe694ed-e7f7-4a69-8c7a-f16d70f49211-metrics-certs\") pod \"4fe694ed-e7f7-4a69-8c7a-f16d70f49211\" (UID: \"4fe694ed-e7f7-4a69-8c7a-f16d70f49211\") " Oct 31 12:58:43.955092 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:43.953684 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fe694ed-e7f7-4a69-8c7a-f16d70f49211-kube-api-access-mxdjj\" (UniqueName: \"kubernetes.io/projected/4fe694ed-e7f7-4a69-8c7a-f16d70f49211-kube-api-access-mxdjj\") pod \"4fe694ed-e7f7-4a69-8c7a-f16d70f49211\" (UID: \"4fe694ed-e7f7-4a69-8c7a-f16d70f49211\") " Oct 31 12:58:43.955092 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:43.953806 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fe694ed-e7f7-4a69-8c7a-f16d70f49211-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe694ed-e7f7-4a69-8c7a-f16d70f49211-openshift-service-ca\") pod \"4fe694ed-e7f7-4a69-8c7a-f16d70f49211\" (UID: \"4fe694ed-e7f7-4a69-8c7a-f16d70f49211\") " Oct 31 12:58:43.955092 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:43.953858 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fe694ed-e7f7-4a69-8c7a-f16d70f49211-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fe694ed-e7f7-4a69-8c7a-f16d70f49211-maas-api-tls\") pod \"4fe694ed-e7f7-4a69-8c7a-f16d70f49211\" (UID: \"4fe694ed-e7f7-4a69-8c7a-f16d70f49211\") " Oct 31 12:58:43.956792 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:43.956749 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fe694ed-e7f7-4a69-8c7a-f16d70f49211-openshift-service-ca" pod "4fe694ed-e7f7-4a69-8c7a-f16d70f49211" (UID: "4fe694ed-e7f7-4a69-8c7a-f16d70f49211"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:58:43.960208 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:43.959803 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fe694ed-e7f7-4a69-8c7a-f16d70f49211-maas-api-tls" pod "4fe694ed-e7f7-4a69-8c7a-f16d70f49211" (UID: "4fe694ed-e7f7-4a69-8c7a-f16d70f49211"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:58:43.960208 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:43.959843 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe694ed-e7f7-4a69-8c7a-f16d70f49211-kube-api-access-mxdjj" pod "4fe694ed-e7f7-4a69-8c7a-f16d70f49211" (UID: "4fe694ed-e7f7-4a69-8c7a-f16d70f49211"). InnerVolumeSpecName "kube-api-access-mxdjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:58:43.960389 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:43.960307 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fe694ed-e7f7-4a69-8c7a-f16d70f49211-metrics-certs" pod "4fe694ed-e7f7-4a69-8c7a-f16d70f49211" (UID: "4fe694ed-e7f7-4a69-8c7a-f16d70f49211"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:58:43.986493 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:43.986280 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:44.071704 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:44.071667 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe694ed-e7f7-4a69-8c7a-f16d70f49211-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:44.071931 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:44.071915 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fe694ed-e7f7-4a69-8c7a-f16d70f49211-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:44.072049 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:44.072035 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fe694ed-e7f7-4a69-8c7a-f16d70f49211-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:44.072134 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:44.072121 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxdjj\" (UniqueName: \"kubernetes.io/projected/4fe694ed-e7f7-4a69-8c7a-f16d70f49211-kube-api-access-mxdjj\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:44.075707 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:44.075302 6841 generic.go:356] "Generic (PLEG): container finished" podID="4fe694ed-e7f7-4a69-8c7a-f16d70f49211" containerID="8dae0e23192509123ba037dd2c143803835b7c6429be3dabf364f262e3003509" exitCode=0 Oct 31 12:58:44.075707 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:44.075436 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s" event={"ID":"4fe694ed-e7f7-4a69-8c7a-f16d70f49211","Type":"ContainerDied","Data":"8dae0e23192509123ba037dd2c143803835b7c6429be3dabf364f262e3003509"} Oct 31 12:58:44.075707 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:44.075495 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s" event={"ID":"4fe694ed-e7f7-4a69-8c7a-f16d70f49211","Type":"ContainerDied","Data":"44a83bf06124c3539169124bdabd1323d9b2e42f3ee1ba00367a891fa7a0bba9"} Oct 31 12:58:44.075707 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:44.075531 6841 scope.go:122] "RemoveContainer" containerID="8dae0e23192509123ba037dd2c143803835b7c6429be3dabf364f262e3003509" Oct 31 12:58:44.080016 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:44.076159 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s" Oct 31 12:58:44.091396 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:44.091369 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn" event={"ID":"c14527eb-216b-4d44-a5a0-c9de5de775b1","Type":"ContainerStarted","Data":"9fbf6b126621a52ea5ac1b944cc40e391e9202ea9d31bedcd4e87b7dd334e9d6"} Oct 31 12:58:44.091817 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:44.091710 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn" Oct 31 12:58:44.097809 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:44.097787 6841 scope.go:122] "RemoveContainer" containerID="8dae0e23192509123ba037dd2c143803835b7c6429be3dabf364f262e3003509" Oct 31 12:58:44.098177 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:58:44.098155 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dae0e23192509123ba037dd2c143803835b7c6429be3dabf364f262e3003509\": container with ID starting with 8dae0e23192509123ba037dd2c143803835b7c6429be3dabf364f262e3003509 not found: ID does not exist" containerID="8dae0e23192509123ba037dd2c143803835b7c6429be3dabf364f262e3003509" Oct 31 12:58:44.098281 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:44.098189 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dae0e23192509123ba037dd2c143803835b7c6429be3dabf364f262e3003509"} err="failed to get container status \"8dae0e23192509123ba037dd2c143803835b7c6429be3dabf364f262e3003509\": rpc error: code = NotFound desc = could not find container \"8dae0e23192509123ba037dd2c143803835b7c6429be3dabf364f262e3003509\": container with ID starting with 8dae0e23192509123ba037dd2c143803835b7c6429be3dabf364f262e3003509 not found: ID does not exist" Oct 31 12:58:44.171873 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:44.171842 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s"] Oct 31 12:58:44.180658 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:44.180633 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-73503b-59f9d475dd-mqp4s"] Oct 31 12:58:44.200574 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:44.200520 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn" podStartSLOduration=2.709001416 podStartE2EDuration="3.200507173s" totalImagesPullingTime="491.505757ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:58:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:58:44.198757645 +0000 UTC m=+1570.681880549" watchObservedRunningTime="2026-10-31 12:58:44.200507173 +0000 UTC m=+1570.683630075" Oct 31 12:58:44.294274 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:44.294193 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:44.547515 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:44.547429 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:44.807065 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:44.806634 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:45.329040 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:45.329003 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:45.647849 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:45.647772 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fe694ed-e7f7-4a69-8c7a-f16d70f49211" path="/var/lib/kubelet/pods/4fe694ed-e7f7-4a69-8c7a-f16d70f49211/volumes" Oct 31 12:58:46.107527 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:46.107481 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:46.804009 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:46.803628 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:47.615006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:47.614007 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:48.305412 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:48.305363 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:48.999009 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:48.995856 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-15c8ff-6d795ffc67-vlfh6" Oct 31 12:58:49.000275 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:48.999666 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-15c8ff-7579cfb7c7-nnxnw" Oct 31 12:58:49.007034 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:49.004961 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:49.702739 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:49.702570 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:50.036081 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:58:50.035969 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:58:50.106806 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:50.106780 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn" Oct 31 12:58:50.537286 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:50.537242 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:51.244604 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:51.244539 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:51.917783 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:51.917744 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:51.929795 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:51.929474 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-2wthh" podUID="158edcbd-2810-4cf7-a2e5-2a881ae0568e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 12:58:51.931536 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:51.931502 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-6x7qm" podUID="2d8bdee0-d436-425d-8b4d-53337f773d27" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 12:58:52.620428 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:52.620386 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:53.413803 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:53.413751 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:54.239123 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:54.232240 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:54.939493 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:54.939456 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:55.238844 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:55.238808 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:55.584344 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:55.584266 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn"] Oct 31 12:58:55.584912 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:55.584519 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn" podUID="75be7457-b259-4516-8c32-cceef659c36b" containerName="maas-api" containerID="cri-o://3455d8c04f6946cf52c69ff8c874874b24859ff6d8f37e494a60ad944f5f0eba" gracePeriod=30 Oct 31 12:58:55.633346 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:55.633232 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:55.896406 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:55.896323 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:56.155053 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.150264 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf"] Oct 31 12:58:56.155053 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.151447 6841 cpu_manager.go:414] "containerMap: removing container" podUID="4fe694ed-e7f7-4a69-8c7a-f16d70f49211" containerName="maas-api" Oct 31 12:58:56.155053 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.152083 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fe694ed-e7f7-4a69-8c7a-f16d70f49211" containerName="maas-api" Oct 31 12:58:56.155053 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.152159 6841 cpu_manager.go:414] "containerMap: removing container" podUID="2444a3b7-2746-42e5-821e-b0d6bcb8ee50" containerName="payload-processing" Oct 31 12:58:56.155053 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.152167 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2444a3b7-2746-42e5-821e-b0d6bcb8ee50" containerName="payload-processing" Oct 31 12:58:56.155053 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.152386 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fe694ed-e7f7-4a69-8c7a-f16d70f49211" containerName="maas-api" Oct 31 12:58:56.155053 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.152418 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="2444a3b7-2746-42e5-821e-b0d6bcb8ee50" containerName="payload-processing" Oct 31 12:58:56.164283 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.157746 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:58:56.164283 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.158544 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:56.164283 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.162057 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn" Oct 31 12:58:56.169650 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.169326 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-21c556-openshift-default-dockercfg-xpcwb\"" Oct 31 12:58:56.174018 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.173653 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf"] Oct 31 12:58:56.177139 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.177114 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75be7457-b259-4516-8c32-cceef659c36b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/75be7457-b259-4516-8c32-cceef659c36b-metrics-certs\") pod \"75be7457-b259-4516-8c32-cceef659c36b\" (UID: \"75be7457-b259-4516-8c32-cceef659c36b\") " Oct 31 12:58:56.177281 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.177163 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75be7457-b259-4516-8c32-cceef659c36b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75be7457-b259-4516-8c32-cceef659c36b-maas-api-tls\") pod \"75be7457-b259-4516-8c32-cceef659c36b\" (UID: \"75be7457-b259-4516-8c32-cceef659c36b\") " Oct 31 12:58:56.177471 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.177447 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75be7457-b259-4516-8c32-cceef659c36b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75be7457-b259-4516-8c32-cceef659c36b-openshift-service-ca\") pod \"75be7457-b259-4516-8c32-cceef659c36b\" (UID: \"75be7457-b259-4516-8c32-cceef659c36b\") " Oct 31 12:58:56.177562 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.177531 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75be7457-b259-4516-8c32-cceef659c36b-kube-api-access-ljgrv\" (UniqueName: \"kubernetes.io/projected/75be7457-b259-4516-8c32-cceef659c36b-kube-api-access-ljgrv\") pod \"75be7457-b259-4516-8c32-cceef659c36b\" (UID: \"75be7457-b259-4516-8c32-cceef659c36b\") " Oct 31 12:58:56.178093 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.177898 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad431a60-f82d-4b0b-928d-dec7ff77f067-credential-socket\") pod \"e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf\" (UID: \"ad431a60-f82d-4b0b-928d-dec7ff77f067\") " pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:58:56.178093 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.177945 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad431a60-f82d-4b0b-928d-dec7ff77f067-istio-data\") pod \"e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf\" (UID: \"ad431a60-f82d-4b0b-928d-dec7ff77f067\") " pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:58:56.178093 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.178048 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkmck\" (UniqueName: \"kubernetes.io/projected/ad431a60-f82d-4b0b-928d-dec7ff77f067-kube-api-access-pkmck\") pod \"e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf\" (UID: \"ad431a60-f82d-4b0b-928d-dec7ff77f067\") " pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:58:56.178324 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.178184 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75be7457-b259-4516-8c32-cceef659c36b-openshift-service-ca" pod "75be7457-b259-4516-8c32-cceef659c36b" (UID: "75be7457-b259-4516-8c32-cceef659c36b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:58:56.178324 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.178232 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad431a60-f82d-4b0b-928d-dec7ff77f067-workload-socket\") pod \"e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf\" (UID: \"ad431a60-f82d-4b0b-928d-dec7ff77f067\") " pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:58:56.178437 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.178339 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad431a60-f82d-4b0b-928d-dec7ff77f067-istio-token\") pod \"e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf\" (UID: \"ad431a60-f82d-4b0b-928d-dec7ff77f067\") " pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:58:56.178437 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.178421 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad431a60-f82d-4b0b-928d-dec7ff77f067-istio-envoy\") pod \"e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf\" (UID: \"ad431a60-f82d-4b0b-928d-dec7ff77f067\") " pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:58:56.178553 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.178503 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad431a60-f82d-4b0b-928d-dec7ff77f067-istiod-ca-cert\") pod \"e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf\" (UID: \"ad431a60-f82d-4b0b-928d-dec7ff77f067\") " pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:58:56.178553 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.178535 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad431a60-f82d-4b0b-928d-dec7ff77f067-workload-certs\") pod \"e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf\" (UID: \"ad431a60-f82d-4b0b-928d-dec7ff77f067\") " pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:58:56.179061 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.178674 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad431a60-f82d-4b0b-928d-dec7ff77f067-istio-podinfo\") pod \"e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf\" (UID: \"ad431a60-f82d-4b0b-928d-dec7ff77f067\") " pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:58:56.179061 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.178835 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75be7457-b259-4516-8c32-cceef659c36b-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:56.181550 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.181507 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75be7457-b259-4516-8c32-cceef659c36b-maas-api-tls" pod "75be7457-b259-4516-8c32-cceef659c36b" (UID: "75be7457-b259-4516-8c32-cceef659c36b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:58:56.183557 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.183535 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75be7457-b259-4516-8c32-cceef659c36b-metrics-certs" pod "75be7457-b259-4516-8c32-cceef659c36b" (UID: "75be7457-b259-4516-8c32-cceef659c36b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:58:56.189087 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.188504 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75be7457-b259-4516-8c32-cceef659c36b-kube-api-access-ljgrv" pod "75be7457-b259-4516-8c32-cceef659c36b" (UID: "75be7457-b259-4516-8c32-cceef659c36b"). InnerVolumeSpecName "kube-api-access-ljgrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:58:56.279872 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.279837 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad431a60-f82d-4b0b-928d-dec7ff77f067-istio-podinfo\") pod \"e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf\" (UID: \"ad431a60-f82d-4b0b-928d-dec7ff77f067\") " pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:58:56.280091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.279906 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad431a60-f82d-4b0b-928d-dec7ff77f067-credential-socket\") pod \"e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf\" (UID: \"ad431a60-f82d-4b0b-928d-dec7ff77f067\") " pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:58:56.280091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.279932 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad431a60-f82d-4b0b-928d-dec7ff77f067-istio-data\") pod \"e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf\" (UID: \"ad431a60-f82d-4b0b-928d-dec7ff77f067\") " pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:58:56.280091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.280003 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkmck\" (UniqueName: \"kubernetes.io/projected/ad431a60-f82d-4b0b-928d-dec7ff77f067-kube-api-access-pkmck\") pod \"e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf\" (UID: \"ad431a60-f82d-4b0b-928d-dec7ff77f067\") " pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:58:56.280091 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.280075 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad431a60-f82d-4b0b-928d-dec7ff77f067-workload-socket\") pod \"e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf\" (UID: \"ad431a60-f82d-4b0b-928d-dec7ff77f067\") " pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:58:56.280329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.280138 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad431a60-f82d-4b0b-928d-dec7ff77f067-istio-token\") pod \"e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf\" (UID: \"ad431a60-f82d-4b0b-928d-dec7ff77f067\") " pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:58:56.280329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.280184 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad431a60-f82d-4b0b-928d-dec7ff77f067-istio-envoy\") pod \"e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf\" (UID: \"ad431a60-f82d-4b0b-928d-dec7ff77f067\") " pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:58:56.280329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.280248 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad431a60-f82d-4b0b-928d-dec7ff77f067-istiod-ca-cert\") pod \"e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf\" (UID: \"ad431a60-f82d-4b0b-928d-dec7ff77f067\") " pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:58:56.280329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.280288 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad431a60-f82d-4b0b-928d-dec7ff77f067-workload-certs\") pod \"e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf\" (UID: \"ad431a60-f82d-4b0b-928d-dec7ff77f067\") " pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:58:56.280549 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.280415 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad431a60-f82d-4b0b-928d-dec7ff77f067-credential-socket\") pod \"e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf\" (UID: \"ad431a60-f82d-4b0b-928d-dec7ff77f067\") " pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:58:56.280549 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.280424 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad431a60-f82d-4b0b-928d-dec7ff77f067-istio-data\") pod \"e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf\" (UID: \"ad431a60-f82d-4b0b-928d-dec7ff77f067\") " pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:58:56.280549 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.280434 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljgrv\" (UniqueName: \"kubernetes.io/projected/75be7457-b259-4516-8c32-cceef659c36b-kube-api-access-ljgrv\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:56.280549 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.280468 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad431a60-f82d-4b0b-928d-dec7ff77f067-workload-socket\") pod \"e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf\" (UID: \"ad431a60-f82d-4b0b-928d-dec7ff77f067\") " pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:58:56.280549 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.280486 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75be7457-b259-4516-8c32-cceef659c36b-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:56.280549 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.280515 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75be7457-b259-4516-8c32-cceef659c36b-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:56.280858 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.280719 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad431a60-f82d-4b0b-928d-dec7ff77f067-workload-certs\") pod \"e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf\" (UID: \"ad431a60-f82d-4b0b-928d-dec7ff77f067\") " pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:58:56.281164 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.281142 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad431a60-f82d-4b0b-928d-dec7ff77f067-istiod-ca-cert\") pod \"e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf\" (UID: \"ad431a60-f82d-4b0b-928d-dec7ff77f067\") " pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:58:56.282561 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.282537 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad431a60-f82d-4b0b-928d-dec7ff77f067-istio-envoy\") pod \"e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf\" (UID: \"ad431a60-f82d-4b0b-928d-dec7ff77f067\") " pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:58:56.283092 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.283068 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad431a60-f82d-4b0b-928d-dec7ff77f067-istio-podinfo\") pod \"e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf\" (UID: \"ad431a60-f82d-4b0b-928d-dec7ff77f067\") " pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:58:56.290309 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.290286 6841 generic.go:356] "Generic (PLEG): container finished" podID="75be7457-b259-4516-8c32-cceef659c36b" containerID="3455d8c04f6946cf52c69ff8c874874b24859ff6d8f37e494a60ad944f5f0eba" exitCode=0 Oct 31 12:58:56.290399 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.290327 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn" Oct 31 12:58:56.290399 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.290365 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn" event={"ID":"75be7457-b259-4516-8c32-cceef659c36b","Type":"ContainerDied","Data":"3455d8c04f6946cf52c69ff8c874874b24859ff6d8f37e494a60ad944f5f0eba"} Oct 31 12:58:56.290488 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.290412 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn" event={"ID":"75be7457-b259-4516-8c32-cceef659c36b","Type":"ContainerDied","Data":"390a99e4d008aede758672c9878fa18fb5653afac09820a74af07dc2da949365"} Oct 31 12:58:56.290488 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.290434 6841 scope.go:122] "RemoveContainer" containerID="3455d8c04f6946cf52c69ff8c874874b24859ff6d8f37e494a60ad944f5f0eba" Oct 31 12:58:56.296717 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.296696 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad431a60-f82d-4b0b-928d-dec7ff77f067-istio-token\") pod \"e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf\" (UID: \"ad431a60-f82d-4b0b-928d-dec7ff77f067\") " pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:58:56.297519 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.297494 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkmck\" (UniqueName: \"kubernetes.io/projected/ad431a60-f82d-4b0b-928d-dec7ff77f067-kube-api-access-pkmck\") pod \"e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf\" (UID: \"ad431a60-f82d-4b0b-928d-dec7ff77f067\") " pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:58:56.302726 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.302076 6841 scope.go:122] "RemoveContainer" containerID="3455d8c04f6946cf52c69ff8c874874b24859ff6d8f37e494a60ad944f5f0eba" Oct 31 12:58:56.302726 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:58:56.302374 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3455d8c04f6946cf52c69ff8c874874b24859ff6d8f37e494a60ad944f5f0eba\": container with ID starting with 3455d8c04f6946cf52c69ff8c874874b24859ff6d8f37e494a60ad944f5f0eba not found: ID does not exist" containerID="3455d8c04f6946cf52c69ff8c874874b24859ff6d8f37e494a60ad944f5f0eba" Oct 31 12:58:56.302726 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.302406 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3455d8c04f6946cf52c69ff8c874874b24859ff6d8f37e494a60ad944f5f0eba"} err="failed to get container status \"3455d8c04f6946cf52c69ff8c874874b24859ff6d8f37e494a60ad944f5f0eba\": rpc error: code = NotFound desc = could not find container \"3455d8c04f6946cf52c69ff8c874874b24859ff6d8f37e494a60ad944f5f0eba\": container with ID starting with 3455d8c04f6946cf52c69ff8c874874b24859ff6d8f37e494a60ad944f5f0eba not found: ID does not exist" Oct 31 12:58:56.315851 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.315817 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn"] Oct 31 12:58:56.320761 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.320739 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-mh9wn"] Oct 31 12:58:56.371842 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.371814 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:56.483270 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.483234 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:58:56.593004 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.587173 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:56.903027 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.902975 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf"] Oct 31 12:58:56.903308 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:58:56.903290 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad431a60_f82d_4b0b_928d_dec7ff77f067.slice/crio-acddfcafba90047f1d7531c69526547217c6b69b62e6d0aef2cc8b9934a6753b WatchSource:0}: Error finding container acddfcafba90047f1d7531c69526547217c6b69b62e6d0aef2cc8b9934a6753b: Status 404 returned error can't find the container with id acddfcafba90047f1d7531c69526547217c6b69b62e6d0aef2cc8b9934a6753b Oct 31 12:58:56.905867 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.905835 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:58:56.905963 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.905900 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:58:56.905963 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:56.905930 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:58:57.129005 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:57.123492 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:57.300505 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:57.300470 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" event={"ID":"ad431a60-f82d-4b0b-928d-dec7ff77f067","Type":"ContainerStarted","Data":"b1ee81c03b1d8adac98f61d130c9d997e94dc5767a755c0992b3c980210bb0e0"} Oct 31 12:58:57.300693 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:57.300532 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" event={"ID":"ad431a60-f82d-4b0b-928d-dec7ff77f067","Type":"ContainerStarted","Data":"acddfcafba90047f1d7531c69526547217c6b69b62e6d0aef2cc8b9934a6753b"} Oct 31 12:58:57.330703 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:57.330655 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" podStartSLOduration=1.330640321 podStartE2EDuration="1.330640321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:58:57.325562358 +0000 UTC m=+1583.808685467" watchObservedRunningTime="2026-10-31 12:58:57.330640321 +0000 UTC m=+1583.813763226" Oct 31 12:58:57.494004 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:57.490130 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:58:57.527010 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:57.522264 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:58:57.643186 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:57.643110 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75be7457-b259-4516-8c32-cceef659c36b" path="/var/lib/kubelet/pods/75be7457-b259-4516-8c32-cceef659c36b/volumes" Oct 31 12:58:57.981283 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:57.981248 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:58.317491 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:58.317411 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:58:58.318126 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:58.318097 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:58:58.817003 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:58.814307 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:58:59.822923 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:58:59.822890 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:00.127534 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:00.127445 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:00.465016 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:00.461671 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:01.189787 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:01.189753 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:01.255115 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:01.255081 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-21c556-7f5b69bb87-xd67g"] Oct 31 12:59:01.255812 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:01.255797 6841 cpu_manager.go:414] "containerMap: removing container" podUID="75be7457-b259-4516-8c32-cceef659c36b" containerName="maas-api" Oct 31 12:59:01.255885 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:01.255813 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75be7457-b259-4516-8c32-cceef659c36b" containerName="maas-api" Oct 31 12:59:01.255919 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:01.255899 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="75be7457-b259-4516-8c32-cceef659c36b" containerName="maas-api" Oct 31 12:59:01.261065 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:01.261039 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-21c556-7f5b69bb87-xd67g" Oct 31 12:59:01.276267 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:01.276238 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-21c556-7f5b69bb87-xd67g"] Oct 31 12:59:01.403766 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:01.403162 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-21c556\"" Oct 31 12:59:01.403766 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:01.403332 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-21c556-dockercfg-nvrnl\"" Oct 31 12:59:01.438268 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:01.438222 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/312077be-9bec-47bf-9479-d8710e4740c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-21c556-7f5b69bb87-xd67g\" (UID: \"312077be-9bec-47bf-9479-d8710e4740c2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-21c556-7f5b69bb87-xd67g" Oct 31 12:59:01.471003 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:01.463384 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:01.471003 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:01.464309 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk5zn\" (UniqueName: \"kubernetes.io/projected/312077be-9bec-47bf-9479-d8710e4740c2-kube-api-access-sk5zn\") pod \"payload-pre-processing-e2e-api-b-21c556-7f5b69bb87-xd67g\" (UID: \"312077be-9bec-47bf-9479-d8710e4740c2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-21c556-7f5b69bb87-xd67g" Oct 31 12:59:01.566680 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:01.566636 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk5zn\" (UniqueName: \"kubernetes.io/projected/312077be-9bec-47bf-9479-d8710e4740c2-kube-api-access-sk5zn\") pod \"payload-pre-processing-e2e-api-b-21c556-7f5b69bb87-xd67g\" (UID: \"312077be-9bec-47bf-9479-d8710e4740c2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-21c556-7f5b69bb87-xd67g" Oct 31 12:59:01.566837 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:01.566717 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/312077be-9bec-47bf-9479-d8710e4740c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-21c556-7f5b69bb87-xd67g\" (UID: \"312077be-9bec-47bf-9479-d8710e4740c2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-21c556-7f5b69bb87-xd67g" Oct 31 12:59:01.567890 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:01.567870 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/312077be-9bec-47bf-9479-d8710e4740c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-21c556-7f5b69bb87-xd67g\" (UID: \"312077be-9bec-47bf-9479-d8710e4740c2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-21c556-7f5b69bb87-xd67g" Oct 31 12:59:01.587057 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:01.587026 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk5zn\" (UniqueName: \"kubernetes.io/projected/312077be-9bec-47bf-9479-d8710e4740c2-kube-api-access-sk5zn\") pod \"payload-pre-processing-e2e-api-b-21c556-7f5b69bb87-xd67g\" (UID: \"312077be-9bec-47bf-9479-d8710e4740c2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-21c556-7f5b69bb87-xd67g" Oct 31 12:59:01.878739 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:01.878664 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-21c556-7f5b69bb87-xd67g" Oct 31 12:59:01.928908 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:01.928871 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-2wthh" podUID="158edcbd-2810-4cf7-a2e5-2a881ae0568e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 12:59:01.929110 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:01.928891 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-6x7qm" podUID="2d8bdee0-d436-425d-8b4d-53337f773d27" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 12:59:02.166203 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:02.166175 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-21c556-7f5b69bb87-xd67g"] Oct 31 12:59:02.166769 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:59:02.166740 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod312077be_9bec_47bf_9479_d8710e4740c2.slice/crio-c46d2d46aac85de46786ec77f81c40588c59f2266c7c5e078195f60118e283ef WatchSource:0}: Error finding container c46d2d46aac85de46786ec77f81c40588c59f2266c7c5e078195f60118e283ef: Status 404 returned error can't find the container with id c46d2d46aac85de46786ec77f81c40588c59f2266c7c5e078195f60118e283ef Oct 31 12:59:02.561004 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:02.554421 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:02.586918 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:02.586884 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-21c556-7f5b69bb87-xd67g" event={"ID":"312077be-9bec-47bf-9479-d8710e4740c2","Type":"ContainerStarted","Data":"09dff4e745d2cd7deb94c84c5b819541f8c3b71c6df22bfa45c7ef8275f1debf"} Oct 31 12:59:02.587105 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:02.586941 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-21c556-7f5b69bb87-xd67g" event={"ID":"312077be-9bec-47bf-9479-d8710e4740c2","Type":"ContainerStarted","Data":"c46d2d46aac85de46786ec77f81c40588c59f2266c7c5e078195f60118e283ef"} Oct 31 12:59:02.587105 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:02.586964 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-21c556-7f5b69bb87-xd67g" Oct 31 12:59:02.609690 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:02.609637 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-21c556-7f5b69bb87-xd67g" podStartSLOduration=1.6096251289999999 podStartE2EDuration="1.609625129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:02.605724822 +0000 UTC m=+1589.088847724" watchObservedRunningTime="2026-10-31 12:59:02.609625129 +0000 UTC m=+1589.092748030" Oct 31 12:59:03.815298 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:03.815265 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-6x7qm" Oct 31 12:59:03.936256 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:03.935693 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d8bdee0-d436-425d-8b4d-53337f773d27-kube-api-access-6jmw9\" (UniqueName: \"kubernetes.io/projected/2d8bdee0-d436-425d-8b4d-53337f773d27-kube-api-access-6jmw9\") pod \"2d8bdee0-d436-425d-8b4d-53337f773d27\" (UID: \"2d8bdee0-d436-425d-8b4d-53337f773d27\") " Oct 31 12:59:03.936256 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:03.935838 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d8bdee0-d436-425d-8b4d-53337f773d27-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d8bdee0-d436-425d-8b4d-53337f773d27-plugins-config-volume\") pod \"2d8bdee0-d436-425d-8b4d-53337f773d27\" (UID: \"2d8bdee0-d436-425d-8b4d-53337f773d27\") " Oct 31 12:59:03.937944 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:03.937608 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d8bdee0-d436-425d-8b4d-53337f773d27-plugins-config-volume" pod "2d8bdee0-d436-425d-8b4d-53337f773d27" (UID: "2d8bdee0-d436-425d-8b4d-53337f773d27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:59:03.940495 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:03.940470 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d8bdee0-d436-425d-8b4d-53337f773d27-kube-api-access-6jmw9" pod "2d8bdee0-d436-425d-8b4d-53337f773d27" (UID: "2d8bdee0-d436-425d-8b4d-53337f773d27"). InnerVolumeSpecName "kube-api-access-6jmw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:03.960555 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:03.960349 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:03.989679 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:03.989652 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-2wthh" Oct 31 12:59:04.038688 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.038644 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jmw9\" (UniqueName: \"kubernetes.io/projected/2d8bdee0-d436-425d-8b4d-53337f773d27-kube-api-access-6jmw9\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:04.038688 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.038680 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d8bdee0-d436-425d-8b4d-53337f773d27-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:04.139708 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.139632 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/158edcbd-2810-4cf7-a2e5-2a881ae0568e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/158edcbd-2810-4cf7-a2e5-2a881ae0568e-plugins-config-volume\") pod \"158edcbd-2810-4cf7-a2e5-2a881ae0568e\" (UID: \"158edcbd-2810-4cf7-a2e5-2a881ae0568e\") " Oct 31 12:59:04.139856 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.139768 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/158edcbd-2810-4cf7-a2e5-2a881ae0568e-kube-api-access-7z594\" (UniqueName: \"kubernetes.io/projected/158edcbd-2810-4cf7-a2e5-2a881ae0568e-kube-api-access-7z594\") pod \"158edcbd-2810-4cf7-a2e5-2a881ae0568e\" (UID: \"158edcbd-2810-4cf7-a2e5-2a881ae0568e\") " Oct 31 12:59:04.140025 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.139974 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/158edcbd-2810-4cf7-a2e5-2a881ae0568e-plugins-config-volume" pod "158edcbd-2810-4cf7-a2e5-2a881ae0568e" (UID: "158edcbd-2810-4cf7-a2e5-2a881ae0568e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:59:04.140223 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.140201 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/158edcbd-2810-4cf7-a2e5-2a881ae0568e-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:04.141726 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.141707 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/158edcbd-2810-4cf7-a2e5-2a881ae0568e-kube-api-access-7z594" pod "158edcbd-2810-4cf7-a2e5-2a881ae0568e" (UID: "158edcbd-2810-4cf7-a2e5-2a881ae0568e"). InnerVolumeSpecName "kube-api-access-7z594". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:04.190244 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.190215 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:04.241927 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.241898 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z594\" (UniqueName: \"kubernetes.io/projected/158edcbd-2810-4cf7-a2e5-2a881ae0568e-kube-api-access-7z594\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:04.246921 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.246895 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-l6mb5"] Oct 31 12:59:04.247584 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.247569 6841 cpu_manager.go:414] "containerMap: removing container" podUID="2d8bdee0-d436-425d-8b4d-53337f773d27" containerName="payload-processing" Oct 31 12:59:04.247633 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.247584 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d8bdee0-d436-425d-8b4d-53337f773d27" containerName="payload-processing" Oct 31 12:59:04.247633 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.247611 6841 cpu_manager.go:414] "containerMap: removing container" podUID="158edcbd-2810-4cf7-a2e5-2a881ae0568e" containerName="payload-pre-processing" Oct 31 12:59:04.247633 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.247618 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="158edcbd-2810-4cf7-a2e5-2a881ae0568e" containerName="payload-pre-processing" Oct 31 12:59:04.247740 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.247728 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d8bdee0-d436-425d-8b4d-53337f773d27" containerName="payload-processing" Oct 31 12:59:04.247773 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.247742 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="158edcbd-2810-4cf7-a2e5-2a881ae0568e" containerName="payload-pre-processing" Oct 31 12:59:04.251257 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.251238 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-l6mb5" Oct 31 12:59:04.260017 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.259745 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c4167730-dockercfg-c5ghn\"" Oct 31 12:59:04.261893 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.261868 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-x5snf"] Oct 31 12:59:04.267831 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.267810 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-l6mb5"] Oct 31 12:59:04.267964 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.267915 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-x5snf" Oct 31 12:59:04.277199 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.277174 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-x5snf"] Oct 31 12:59:04.388006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.385110 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0db0ab8b-39fe-485d-9d81-c75676f13374-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c4167730-6cb6676695-l6mb5\" (UID: \"0db0ab8b-39fe-485d-9d81-c75676f13374\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-l6mb5" Oct 31 12:59:04.388006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.385231 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fddad4ae-9da5-4956-a2ae-fa2d0b075f2c-plugins-config-volume\") pod \"payload-processing-e2e-mt-c4167730-758cb7b74f-x5snf\" (UID: \"fddad4ae-9da5-4956-a2ae-fa2d0b075f2c\") " pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-x5snf" Oct 31 12:59:04.388006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.385291 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs8dk\" (UniqueName: \"kubernetes.io/projected/fddad4ae-9da5-4956-a2ae-fa2d0b075f2c-kube-api-access-fs8dk\") pod \"payload-processing-e2e-mt-c4167730-758cb7b74f-x5snf\" (UID: \"fddad4ae-9da5-4956-a2ae-fa2d0b075f2c\") " pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-x5snf" Oct 31 12:59:04.388006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.385316 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvnh4\" (UniqueName: \"kubernetes.io/projected/0db0ab8b-39fe-485d-9d81-c75676f13374-kube-api-access-gvnh4\") pod \"payload-pre-processing-e2e-mt-c4167730-6cb6676695-l6mb5\" (UID: \"0db0ab8b-39fe-485d-9d81-c75676f13374\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-l6mb5" Oct 31 12:59:04.426057 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.425727 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:04.493005 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.492122 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fddad4ae-9da5-4956-a2ae-fa2d0b075f2c-plugins-config-volume\") pod \"payload-processing-e2e-mt-c4167730-758cb7b74f-x5snf\" (UID: \"fddad4ae-9da5-4956-a2ae-fa2d0b075f2c\") " pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-x5snf" Oct 31 12:59:04.493005 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.492212 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs8dk\" (UniqueName: \"kubernetes.io/projected/fddad4ae-9da5-4956-a2ae-fa2d0b075f2c-kube-api-access-fs8dk\") pod \"payload-processing-e2e-mt-c4167730-758cb7b74f-x5snf\" (UID: \"fddad4ae-9da5-4956-a2ae-fa2d0b075f2c\") " pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-x5snf" Oct 31 12:59:04.493005 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.492244 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvnh4\" (UniqueName: \"kubernetes.io/projected/0db0ab8b-39fe-485d-9d81-c75676f13374-kube-api-access-gvnh4\") pod \"payload-pre-processing-e2e-mt-c4167730-6cb6676695-l6mb5\" (UID: \"0db0ab8b-39fe-485d-9d81-c75676f13374\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-l6mb5" Oct 31 12:59:04.493005 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.492442 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0db0ab8b-39fe-485d-9d81-c75676f13374-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c4167730-6cb6676695-l6mb5\" (UID: \"0db0ab8b-39fe-485d-9d81-c75676f13374\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-l6mb5" Oct 31 12:59:04.497096 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.493596 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0db0ab8b-39fe-485d-9d81-c75676f13374-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c4167730-6cb6676695-l6mb5\" (UID: \"0db0ab8b-39fe-485d-9d81-c75676f13374\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-l6mb5" Oct 31 12:59:04.497096 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.494367 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fddad4ae-9da5-4956-a2ae-fa2d0b075f2c-plugins-config-volume\") pod \"payload-processing-e2e-mt-c4167730-758cb7b74f-x5snf\" (UID: \"fddad4ae-9da5-4956-a2ae-fa2d0b075f2c\") " pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-x5snf" Oct 31 12:59:04.537110 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.536179 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvnh4\" (UniqueName: \"kubernetes.io/projected/0db0ab8b-39fe-485d-9d81-c75676f13374-kube-api-access-gvnh4\") pod \"payload-pre-processing-e2e-mt-c4167730-6cb6676695-l6mb5\" (UID: \"0db0ab8b-39fe-485d-9d81-c75676f13374\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-l6mb5" Oct 31 12:59:04.537110 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.536746 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs8dk\" (UniqueName: \"kubernetes.io/projected/fddad4ae-9da5-4956-a2ae-fa2d0b075f2c-kube-api-access-fs8dk\") pod \"payload-processing-e2e-mt-c4167730-758cb7b74f-x5snf\" (UID: \"fddad4ae-9da5-4956-a2ae-fa2d0b075f2c\") " pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-x5snf" Oct 31 12:59:04.587449 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.579764 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-l6mb5" Oct 31 12:59:04.590371 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.589322 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-x5snf" Oct 31 12:59:04.618620 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.617249 6841 generic.go:356] "Generic (PLEG): container finished" podID="2d8bdee0-d436-425d-8b4d-53337f773d27" containerID="64be69590e1308cc2f9e44fa61e7bc61364928f40cbd743d63e2d18d74ce8195" exitCode=0 Oct 31 12:59:04.618620 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.617492 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-6x7qm" event={"ID":"2d8bdee0-d436-425d-8b4d-53337f773d27","Type":"ContainerDied","Data":"64be69590e1308cc2f9e44fa61e7bc61364928f40cbd743d63e2d18d74ce8195"} Oct 31 12:59:04.618620 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.617547 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-6x7qm" event={"ID":"2d8bdee0-d436-425d-8b4d-53337f773d27","Type":"ContainerDied","Data":"9ee23c6936abb1573b5b26a4ddd7bab6d3d922fe8079d6fd07b2f787bf4d791c"} Oct 31 12:59:04.618620 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.617581 6841 scope.go:122] "RemoveContainer" containerID="64be69590e1308cc2f9e44fa61e7bc61364928f40cbd743d63e2d18d74ce8195" Oct 31 12:59:04.618620 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.617752 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-6x7qm" Oct 31 12:59:04.627778 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.626627 6841 generic.go:356] "Generic (PLEG): container finished" podID="158edcbd-2810-4cf7-a2e5-2a881ae0568e" containerID="2f18b66c3e153ad6cb29b8d224c86d5afc857aba8d9c665054491381f590fe0e" exitCode=0 Oct 31 12:59:04.627778 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.626722 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-2wthh" event={"ID":"158edcbd-2810-4cf7-a2e5-2a881ae0568e","Type":"ContainerDied","Data":"2f18b66c3e153ad6cb29b8d224c86d5afc857aba8d9c665054491381f590fe0e"} Oct 31 12:59:04.627778 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.626773 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-2wthh" event={"ID":"158edcbd-2810-4cf7-a2e5-2a881ae0568e","Type":"ContainerDied","Data":"1b767103236c8ed683d9da8f4048cd03a3f94e865b4b42b73a383143c0dbaff9"} Oct 31 12:59:04.627778 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.626854 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-2wthh" Oct 31 12:59:04.675495 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.675337 6841 scope.go:122] "RemoveContainer" containerID="64be69590e1308cc2f9e44fa61e7bc61364928f40cbd743d63e2d18d74ce8195" Oct 31 12:59:04.679873 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:59:04.678713 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64be69590e1308cc2f9e44fa61e7bc61364928f40cbd743d63e2d18d74ce8195\": container with ID starting with 64be69590e1308cc2f9e44fa61e7bc61364928f40cbd743d63e2d18d74ce8195 not found: ID does not exist" containerID="64be69590e1308cc2f9e44fa61e7bc61364928f40cbd743d63e2d18d74ce8195" Oct 31 12:59:04.679873 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.678746 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64be69590e1308cc2f9e44fa61e7bc61364928f40cbd743d63e2d18d74ce8195"} err="failed to get container status \"64be69590e1308cc2f9e44fa61e7bc61364928f40cbd743d63e2d18d74ce8195\": rpc error: code = NotFound desc = could not find container \"64be69590e1308cc2f9e44fa61e7bc61364928f40cbd743d63e2d18d74ce8195\": container with ID starting with 64be69590e1308cc2f9e44fa61e7bc61364928f40cbd743d63e2d18d74ce8195 not found: ID does not exist" Oct 31 12:59:04.679873 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.678766 6841 scope.go:122] "RemoveContainer" containerID="2f18b66c3e153ad6cb29b8d224c86d5afc857aba8d9c665054491381f590fe0e" Oct 31 12:59:04.694993 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.694422 6841 scope.go:122] "RemoveContainer" containerID="2f18b66c3e153ad6cb29b8d224c86d5afc857aba8d9c665054491381f590fe0e" Oct 31 12:59:04.694993 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:59:04.694792 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f18b66c3e153ad6cb29b8d224c86d5afc857aba8d9c665054491381f590fe0e\": container with ID starting with 2f18b66c3e153ad6cb29b8d224c86d5afc857aba8d9c665054491381f590fe0e not found: ID does not exist" containerID="2f18b66c3e153ad6cb29b8d224c86d5afc857aba8d9c665054491381f590fe0e" Oct 31 12:59:04.695125 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.694825 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f18b66c3e153ad6cb29b8d224c86d5afc857aba8d9c665054491381f590fe0e"} err="failed to get container status \"2f18b66c3e153ad6cb29b8d224c86d5afc857aba8d9c665054491381f590fe0e\": rpc error: code = NotFound desc = could not find container \"2f18b66c3e153ad6cb29b8d224c86d5afc857aba8d9c665054491381f590fe0e\": container with ID starting with 2f18b66c3e153ad6cb29b8d224c86d5afc857aba8d9c665054491381f590fe0e not found: ID does not exist" Oct 31 12:59:04.707867 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.707842 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-2wthh"] Oct 31 12:59:04.727773 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.727747 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-2wthh"] Oct 31 12:59:04.745921 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.745890 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-6x7qm"] Oct 31 12:59:04.754534 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.754510 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-6x7qm"] Oct 31 12:59:04.904830 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:59:04.904781 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db0ab8b_39fe_485d_9d81_c75676f13374.slice/crio-05531e978f9e9d2d5e79cb48d8b2aa69f2a67e4185315941a483d50ebf654d0b WatchSource:0}: Error finding container 05531e978f9e9d2d5e79cb48d8b2aa69f2a67e4185315941a483d50ebf654d0b: Status 404 returned error can't find the container with id 05531e978f9e9d2d5e79cb48d8b2aa69f2a67e4185315941a483d50ebf654d0b Oct 31 12:59:04.905212 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.905181 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-l6mb5"] Oct 31 12:59:04.942114 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:04.942091 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-x5snf"] Oct 31 12:59:04.945272 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:59:04.945247 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfddad4ae_9da5_4956_a2ae_fa2d0b075f2c.slice/crio-13263f03d22ad9ac1d7c77719f264f6a91c85700b38a24b83493a407d9c95e52 WatchSource:0}: Error finding container 13263f03d22ad9ac1d7c77719f264f6a91c85700b38a24b83493a407d9c95e52: Status 404 returned error can't find the container with id 13263f03d22ad9ac1d7c77719f264f6a91c85700b38a24b83493a407d9c95e52 Oct 31 12:59:05.112142 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:05.112094 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:05.645094 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:05.645063 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="158edcbd-2810-4cf7-a2e5-2a881ae0568e" path="/var/lib/kubelet/pods/158edcbd-2810-4cf7-a2e5-2a881ae0568e/volumes" Oct 31 12:59:05.645499 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:05.645483 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d8bdee0-d436-425d-8b4d-53337f773d27" path="/var/lib/kubelet/pods/2d8bdee0-d436-425d-8b4d-53337f773d27/volumes" Oct 31 12:59:05.645786 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:05.645774 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-x5snf" Oct 31 12:59:05.645823 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:05.645816 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-l6mb5" Oct 31 12:59:05.645857 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:05.645833 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-x5snf" event={"ID":"fddad4ae-9da5-4956-a2ae-fa2d0b075f2c","Type":"ContainerStarted","Data":"c51d581a359187bb0416ffa2b2f206769b7197fd8bd9559a736d54efb2b0e319"} Oct 31 12:59:05.645857 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:05.645851 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-x5snf" event={"ID":"fddad4ae-9da5-4956-a2ae-fa2d0b075f2c","Type":"ContainerStarted","Data":"13263f03d22ad9ac1d7c77719f264f6a91c85700b38a24b83493a407d9c95e52"} Oct 31 12:59:05.645924 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:05.645867 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-l6mb5" event={"ID":"0db0ab8b-39fe-485d-9d81-c75676f13374","Type":"ContainerStarted","Data":"816e01486b7c1fbd7fbf964ef937c3e2acad92fafa0920a5691a16f4217f5c57"} Oct 31 12:59:05.645924 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:05.645880 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-l6mb5" event={"ID":"0db0ab8b-39fe-485d-9d81-c75676f13374","Type":"ContainerStarted","Data":"05531e978f9e9d2d5e79cb48d8b2aa69f2a67e4185315941a483d50ebf654d0b"} Oct 31 12:59:05.707498 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:05.707443 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-x5snf" podStartSLOduration=1.707429337 podStartE2EDuration="1.707429337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:05.705815801 +0000 UTC m=+1592.188938735" watchObservedRunningTime="2026-10-31 12:59:05.707429337 +0000 UTC m=+1592.190552237" Oct 31 12:59:05.729885 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:05.729823 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-l6mb5" podStartSLOduration=1.7298074140000002 podStartE2EDuration="1.729807414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:05.726540533 +0000 UTC m=+1592.209663435" watchObservedRunningTime="2026-10-31 12:59:05.729807414 +0000 UTC m=+1592.212930319" Oct 31 12:59:05.947568 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:05.947529 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:06.641594 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:06.641557 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp"] Oct 31 12:59:06.648546 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:06.648519 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp" Oct 31 12:59:06.663679 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:06.663638 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp"] Oct 31 12:59:06.665808 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:06.665303 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-21c556-serving-cert\"" Oct 31 12:59:06.745846 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:06.745818 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c25646f3-cd9d-493c-bfef-692bf398d20f-maas-api-tls\") pod \"maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp\" (UID: \"c25646f3-cd9d-493c-bfef-692bf398d20f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp" Oct 31 12:59:06.746038 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:06.745886 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg2hw\" (UniqueName: \"kubernetes.io/projected/c25646f3-cd9d-493c-bfef-692bf398d20f-kube-api-access-gg2hw\") pod \"maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp\" (UID: \"c25646f3-cd9d-493c-bfef-692bf398d20f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp" Oct 31 12:59:06.746102 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:06.746052 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c25646f3-cd9d-493c-bfef-692bf398d20f-openshift-service-ca\") pod \"maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp\" (UID: \"c25646f3-cd9d-493c-bfef-692bf398d20f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp" Oct 31 12:59:06.746261 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:06.746242 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c25646f3-cd9d-493c-bfef-692bf398d20f-metrics-certs\") pod \"maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp\" (UID: \"c25646f3-cd9d-493c-bfef-692bf398d20f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp" Oct 31 12:59:06.806407 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:06.806324 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:06.854667 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:06.854627 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c25646f3-cd9d-493c-bfef-692bf398d20f-maas-api-tls\") pod \"maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp\" (UID: \"c25646f3-cd9d-493c-bfef-692bf398d20f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp" Oct 31 12:59:06.855023 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:06.854723 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg2hw\" (UniqueName: \"kubernetes.io/projected/c25646f3-cd9d-493c-bfef-692bf398d20f-kube-api-access-gg2hw\") pod \"maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp\" (UID: \"c25646f3-cd9d-493c-bfef-692bf398d20f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp" Oct 31 12:59:06.855023 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:06.854784 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c25646f3-cd9d-493c-bfef-692bf398d20f-openshift-service-ca\") pod \"maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp\" (UID: \"c25646f3-cd9d-493c-bfef-692bf398d20f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp" Oct 31 12:59:06.855023 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:06.854916 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c25646f3-cd9d-493c-bfef-692bf398d20f-metrics-certs\") pod \"maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp\" (UID: \"c25646f3-cd9d-493c-bfef-692bf398d20f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp" Oct 31 12:59:06.865032 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:06.864194 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c25646f3-cd9d-493c-bfef-692bf398d20f-openshift-service-ca\") pod \"maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp\" (UID: \"c25646f3-cd9d-493c-bfef-692bf398d20f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp" Oct 31 12:59:06.902054 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:06.894700 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg2hw\" (UniqueName: \"kubernetes.io/projected/c25646f3-cd9d-493c-bfef-692bf398d20f-kube-api-access-gg2hw\") pod \"maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp\" (UID: \"c25646f3-cd9d-493c-bfef-692bf398d20f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp" Oct 31 12:59:06.909269 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:06.902907 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c25646f3-cd9d-493c-bfef-692bf398d20f-maas-api-tls\") pod \"maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp\" (UID: \"c25646f3-cd9d-493c-bfef-692bf398d20f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp" Oct 31 12:59:06.918009 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:06.912249 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c25646f3-cd9d-493c-bfef-692bf398d20f-metrics-certs\") pod \"maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp\" (UID: \"c25646f3-cd9d-493c-bfef-692bf398d20f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp" Oct 31 12:59:06.988001 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:06.984446 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp" Oct 31 12:59:07.035037 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:07.034498 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:07.358055 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:07.348762 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp"] Oct 31 12:59:07.367348 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:59:07.367303 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc25646f3_cd9d_493c_bfef_692bf398d20f.slice/crio-53f71c8a7a59a564998ca9ab6e116b8ce583284b3e5e0e9f226b8d5c9ddf8e39 WatchSource:0}: Error finding container 53f71c8a7a59a564998ca9ab6e116b8ce583284b3e5e0e9f226b8d5c9ddf8e39: Status 404 returned error can't find the container with id 53f71c8a7a59a564998ca9ab6e116b8ce583284b3e5e0e9f226b8d5c9ddf8e39 Oct 31 12:59:07.635976 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:07.633038 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:07.662212 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:07.662177 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp" event={"ID":"c25646f3-cd9d-493c-bfef-692bf398d20f","Type":"ContainerStarted","Data":"53f71c8a7a59a564998ca9ab6e116b8ce583284b3e5e0e9f226b8d5c9ddf8e39"} Oct 31 12:59:08.418113 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:08.417871 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:08.670507 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:08.670413 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp" event={"ID":"c25646f3-cd9d-493c-bfef-692bf398d20f","Type":"ContainerStarted","Data":"bbc75af6dc03cb69db4ada126f12254513fde386bdd1d60640143ac66b8437cb"} Oct 31 12:59:08.670507 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:08.670494 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp" Oct 31 12:59:08.694455 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:08.694413 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp" podStartSLOduration=2.25346825 podStartE2EDuration="2.694400751s" totalImagesPullingTime="440.932501ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:59:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:08.693358431 +0000 UTC m=+1595.176481347" watchObservedRunningTime="2026-10-31 12:59:08.694400751 +0000 UTC m=+1595.177523651" Oct 31 12:59:09.172090 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:09.172053 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:10.005021 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:09.991307 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:10.553056 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:10.553020 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-z959n"] Oct 31 12:59:10.561153 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:10.561123 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-z959n" Oct 31 12:59:10.569793 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:10.569734 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c4167730-serving-cert\"" Oct 31 12:59:10.574289 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:10.574261 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-z959n"] Oct 31 12:59:10.672197 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:10.672167 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrf9v\" (UniqueName: \"kubernetes.io/projected/ceab3693-ea13-4097-a44c-37561dd698d4-kube-api-access-xrf9v\") pod \"maas-api-e2e-mt-c4167730-85cb9464bc-z959n\" (UID: \"ceab3693-ea13-4097-a44c-37561dd698d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-z959n" Oct 31 12:59:10.672385 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:10.672324 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceab3693-ea13-4097-a44c-37561dd698d4-openshift-service-ca\") pod \"maas-api-e2e-mt-c4167730-85cb9464bc-z959n\" (UID: \"ceab3693-ea13-4097-a44c-37561dd698d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-z959n" Oct 31 12:59:10.672385 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:10.672352 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ceab3693-ea13-4097-a44c-37561dd698d4-maas-api-tls\") pod \"maas-api-e2e-mt-c4167730-85cb9464bc-z959n\" (UID: \"ceab3693-ea13-4097-a44c-37561dd698d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-z959n" Oct 31 12:59:10.672586 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:10.672563 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceab3693-ea13-4097-a44c-37561dd698d4-metrics-certs\") pod \"maas-api-e2e-mt-c4167730-85cb9464bc-z959n\" (UID: \"ceab3693-ea13-4097-a44c-37561dd698d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-z959n" Oct 31 12:59:10.761913 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:10.759581 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:10.785429 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:10.780826 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrf9v\" (UniqueName: \"kubernetes.io/projected/ceab3693-ea13-4097-a44c-37561dd698d4-kube-api-access-xrf9v\") pod \"maas-api-e2e-mt-c4167730-85cb9464bc-z959n\" (UID: \"ceab3693-ea13-4097-a44c-37561dd698d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-z959n" Oct 31 12:59:10.785429 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:10.781074 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceab3693-ea13-4097-a44c-37561dd698d4-openshift-service-ca\") pod \"maas-api-e2e-mt-c4167730-85cb9464bc-z959n\" (UID: \"ceab3693-ea13-4097-a44c-37561dd698d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-z959n" Oct 31 12:59:10.785429 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:10.781104 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ceab3693-ea13-4097-a44c-37561dd698d4-maas-api-tls\") pod \"maas-api-e2e-mt-c4167730-85cb9464bc-z959n\" (UID: \"ceab3693-ea13-4097-a44c-37561dd698d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-z959n" Oct 31 12:59:10.785429 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:10.783577 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceab3693-ea13-4097-a44c-37561dd698d4-openshift-service-ca\") pod \"maas-api-e2e-mt-c4167730-85cb9464bc-z959n\" (UID: \"ceab3693-ea13-4097-a44c-37561dd698d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-z959n" Oct 31 12:59:10.785785 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:10.785470 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceab3693-ea13-4097-a44c-37561dd698d4-metrics-certs\") pod \"maas-api-e2e-mt-c4167730-85cb9464bc-z959n\" (UID: \"ceab3693-ea13-4097-a44c-37561dd698d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-z959n" Oct 31 12:59:10.841762 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:10.841435 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ceab3693-ea13-4097-a44c-37561dd698d4-maas-api-tls\") pod \"maas-api-e2e-mt-c4167730-85cb9464bc-z959n\" (UID: \"ceab3693-ea13-4097-a44c-37561dd698d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-z959n" Oct 31 12:59:10.841916 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:10.841882 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceab3693-ea13-4097-a44c-37561dd698d4-metrics-certs\") pod \"maas-api-e2e-mt-c4167730-85cb9464bc-z959n\" (UID: \"ceab3693-ea13-4097-a44c-37561dd698d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-z959n" Oct 31 12:59:10.858072 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:10.854014 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrf9v\" (UniqueName: \"kubernetes.io/projected/ceab3693-ea13-4097-a44c-37561dd698d4-kube-api-access-xrf9v\") pod \"maas-api-e2e-mt-c4167730-85cb9464bc-z959n\" (UID: \"ceab3693-ea13-4097-a44c-37561dd698d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-z959n" Oct 31 12:59:10.883624 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:10.883396 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-z959n" Oct 31 12:59:11.074004 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:11.073559 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:11.180876 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:11.180844 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-z959n"] Oct 31 12:59:11.181083 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:59:11.181055 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceab3693_ea13_4097_a44c_37561dd698d4.slice/crio-c9eefd5db859722b084b035101fba43af31d2b4b672a8d830332a8b5c049dece WatchSource:0}: Error finding container c9eefd5db859722b084b035101fba43af31d2b4b672a8d830332a8b5c049dece: Status 404 returned error can't find the container with id c9eefd5db859722b084b035101fba43af31d2b4b672a8d830332a8b5c049dece Oct 31 12:59:11.726780 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:11.726700 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-z959n" event={"ID":"ceab3693-ea13-4097-a44c-37561dd698d4","Type":"ContainerStarted","Data":"a1fd1bb3bc4cd3d712566013da5c3088749c965f1639de82a98f58a2ea1ea3de"} Oct 31 12:59:11.726780 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:11.726762 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-z959n" event={"ID":"ceab3693-ea13-4097-a44c-37561dd698d4","Type":"ContainerStarted","Data":"c9eefd5db859722b084b035101fba43af31d2b4b672a8d830332a8b5c049dece"} Oct 31 12:59:11.726964 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:11.726789 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-z959n" Oct 31 12:59:11.780852 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:11.774423 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-z959n" podStartSLOduration=1.334593998 podStartE2EDuration="1.774400309s" totalImagesPullingTime="439.806311ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:59:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:11.75869452 +0000 UTC m=+1598.241817424" watchObservedRunningTime="2026-10-31 12:59:11.774400309 +0000 UTC m=+1598.257523213" Oct 31 12:59:11.825620 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:11.820541 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:12.033296 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:12.033258 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:12.777810 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:12.777773 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:13.612195 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:13.608949 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:13.612195 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:13.610156 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-21c556-7f5b69bb87-xd67g" Oct 31 12:59:14.530006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:14.529048 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:14.682928 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:14.682902 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp" Oct 31 12:59:15.607816 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:15.604421 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:16.428014 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:16.427669 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:16.657677 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:16.657650 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-x5snf" Oct 31 12:59:16.658078 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:16.657727 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-l6mb5" Oct 31 12:59:17.361273 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:17.361232 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:17.755431 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:17.755403 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-z959n" Oct 31 12:59:18.126556 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:18.126463 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:18.880100 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:18.880068 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:19.092779 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:19.092745 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:19.882969 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:19.882877 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:20.036108 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:59:20.036076 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:59:20.090633 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:20.090603 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:20.633599 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:20.633384 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:21.435895 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:21.435858 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:21.636675 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:21.636640 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:22.219795 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:22.219760 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:23.145508 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:23.144896 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:24.323004 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:24.320116 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:25.461279 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:25.461243 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:26.492008 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:26.483044 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11391-2b4b25"] Oct 31 12:59:26.530945 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:26.530802 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:26.567689 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:26.567656 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11391-2b4b25"] Oct 31 12:59:26.592048 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:26.591173 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11391-2b4b25" Oct 31 12:59:26.600831 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:26.600809 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mn5fx\"" Oct 31 12:59:26.693679 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:26.693639 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7br7z\" (UniqueName: \"kubernetes.io/projected/38fb7737-2348-460e-a5e6-a4e401c50b36-kube-api-access-7br7z\") pod \"test-curl-11391-2b4b25\" (UID: \"38fb7737-2348-460e-a5e6-a4e401c50b36\") " pod="opendatahub/test-curl-11391-2b4b25" Oct 31 12:59:26.795797 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:26.795713 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7br7z\" (UniqueName: \"kubernetes.io/projected/38fb7737-2348-460e-a5e6-a4e401c50b36-kube-api-access-7br7z\") pod \"test-curl-11391-2b4b25\" (UID: \"38fb7737-2348-460e-a5e6-a4e401c50b36\") " pod="opendatahub/test-curl-11391-2b4b25" Oct 31 12:59:26.808187 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:26.808162 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7br7z\" (UniqueName: \"kubernetes.io/projected/38fb7737-2348-460e-a5e6-a4e401c50b36-kube-api-access-7br7z\") pod \"test-curl-11391-2b4b25\" (UID: \"38fb7737-2348-460e-a5e6-a4e401c50b36\") " pod="opendatahub/test-curl-11391-2b4b25" Oct 31 12:59:26.917866 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:26.917835 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11391-2b4b25" Oct 31 12:59:27.230226 ip-10-0-138-70 kubenswrapper[6841]: W1031 12:59:27.230197 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38fb7737_2348_460e_a5e6_a4e401c50b36.slice/crio-3ad055665fae54e62ba8a01b2caa76fc1f0f9c315c09163a974897b53a3fb998 WatchSource:0}: Error finding container 3ad055665fae54e62ba8a01b2caa76fc1f0f9c315c09163a974897b53a3fb998: Status 404 returned error can't find the container with id 3ad055665fae54e62ba8a01b2caa76fc1f0f9c315c09163a974897b53a3fb998 Oct 31 12:59:27.230827 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:27.230724 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11391-2b4b25"] Oct 31 12:59:27.345006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:27.338865 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:28.007524 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:28.007484 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11391-2b4b25" event={"ID":"38fb7737-2348-460e-a5e6-a4e401c50b36","Type":"ContainerStarted","Data":"3ad055665fae54e62ba8a01b2caa76fc1f0f9c315c09163a974897b53a3fb998"} Oct 31 12:59:28.202711 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:28.202630 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:28.372224 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:28.372186 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-15c8ff-7579cfb7c7-nnxnw"] Oct 31 12:59:28.379010 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:28.378745 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-15c8ff-6d795ffc67-vlfh6"] Oct 31 12:59:28.533887 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:28.533852 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:28.763048 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:28.761679 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:28.965006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:28.964122 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:29.016440 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:29.016365 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11391-2b4b25" event={"ID":"38fb7737-2348-460e-a5e6-a4e401c50b36","Type":"ContainerStarted","Data":"c4ba3630e74121bfd51da3089cc3158e7dd9e5370eeab3fa124557fbf102a712"} Oct 31 12:59:29.032722 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:29.032676 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11391-2b4b25" podStartSLOduration=1.947989494 podStartE2EDuration="3.032664515s" totalImagesPullingTime="1.084675021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:59:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:29.031650575 +0000 UTC m=+1615.514773478" watchObservedRunningTime="2026-10-31 12:59:29.032664515 +0000 UTC m=+1615.515787420" Oct 31 12:59:29.388830 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:29.388746 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:29.869506 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:29.869387 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11391-2b4b25"] Oct 31 12:59:29.893822 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:29.893782 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11391-2b4b25"] Oct 31 12:59:30.034932 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:30.034897 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:30.473080 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:30.473039 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:30.896067 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:30.887933 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:31.048956 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:31.048911 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11391-2b4b25" podUID="38fb7737-2348-460e-a5e6-a4e401c50b36" containerName="test-curl-11391-2b4b25" containerID="cri-o://c4ba3630e74121bfd51da3089cc3158e7dd9e5370eeab3fa124557fbf102a712" gracePeriod=2 Oct 31 12:59:31.052479 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:59:31.052431 6841 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11391-2b4b25\" is forbidden: User \"system:node:ip-10-0-138-70.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-70.us-east-2.compute.internal' and this object" podUID="38fb7737-2348-460e-a5e6-a4e401c50b36" pod="opendatahub/test-curl-11391-2b4b25" Oct 31 12:59:31.633685 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:31.633607 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:32.141927 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:32.141877 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn"] Oct 31 12:59:32.142551 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:32.142147 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn" podUID="c14527eb-216b-4d44-a5a0-c9de5de775b1" containerName="maas-api" containerID="cri-o://9fbf6b126621a52ea5ac1b944cc40e391e9202ea9d31bedcd4e87b7dd334e9d6" gracePeriod=30 Oct 31 12:59:32.301228 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:32.301189 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:32.624533 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:32.622319 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:32.701012 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:32.699224 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn" Oct 31 12:59:32.828272 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:32.828187 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c14527eb-216b-4d44-a5a0-c9de5de775b1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c14527eb-216b-4d44-a5a0-c9de5de775b1-maas-api-tls\") pod \"c14527eb-216b-4d44-a5a0-c9de5de775b1\" (UID: \"c14527eb-216b-4d44-a5a0-c9de5de775b1\") " Oct 31 12:59:32.828447 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:32.828293 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c14527eb-216b-4d44-a5a0-c9de5de775b1-kube-api-access-zwntb\" (UniqueName: \"kubernetes.io/projected/c14527eb-216b-4d44-a5a0-c9de5de775b1-kube-api-access-zwntb\") pod \"c14527eb-216b-4d44-a5a0-c9de5de775b1\" (UID: \"c14527eb-216b-4d44-a5a0-c9de5de775b1\") " Oct 31 12:59:32.828447 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:32.828361 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c14527eb-216b-4d44-a5a0-c9de5de775b1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c14527eb-216b-4d44-a5a0-c9de5de775b1-metrics-certs\") pod \"c14527eb-216b-4d44-a5a0-c9de5de775b1\" (UID: \"c14527eb-216b-4d44-a5a0-c9de5de775b1\") " Oct 31 12:59:32.828447 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:32.828394 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c14527eb-216b-4d44-a5a0-c9de5de775b1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c14527eb-216b-4d44-a5a0-c9de5de775b1-openshift-service-ca\") pod \"c14527eb-216b-4d44-a5a0-c9de5de775b1\" (UID: \"c14527eb-216b-4d44-a5a0-c9de5de775b1\") " Oct 31 12:59:32.828843 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:32.828806 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c14527eb-216b-4d44-a5a0-c9de5de775b1-openshift-service-ca" pod "c14527eb-216b-4d44-a5a0-c9de5de775b1" (UID: "c14527eb-216b-4d44-a5a0-c9de5de775b1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:59:32.829313 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:32.829288 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c14527eb-216b-4d44-a5a0-c9de5de775b1-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:32.830504 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:32.830481 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c14527eb-216b-4d44-a5a0-c9de5de775b1-metrics-certs" pod "c14527eb-216b-4d44-a5a0-c9de5de775b1" (UID: "c14527eb-216b-4d44-a5a0-c9de5de775b1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:59:32.830696 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:32.830675 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c14527eb-216b-4d44-a5a0-c9de5de775b1-maas-api-tls" pod "c14527eb-216b-4d44-a5a0-c9de5de775b1" (UID: "c14527eb-216b-4d44-a5a0-c9de5de775b1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:59:32.831526 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:32.831507 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c14527eb-216b-4d44-a5a0-c9de5de775b1-kube-api-access-zwntb" pod "c14527eb-216b-4d44-a5a0-c9de5de775b1" (UID: "c14527eb-216b-4d44-a5a0-c9de5de775b1"). InnerVolumeSpecName "kube-api-access-zwntb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:32.930853 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:32.930818 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c14527eb-216b-4d44-a5a0-c9de5de775b1-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:32.930853 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:32.930851 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwntb\" (UniqueName: \"kubernetes.io/projected/c14527eb-216b-4d44-a5a0-c9de5de775b1-kube-api-access-zwntb\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:32.931116 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:32.930865 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c14527eb-216b-4d44-a5a0-c9de5de775b1-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:33.065288 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:33.065261 6841 generic.go:356] "Generic (PLEG): container finished" podID="c14527eb-216b-4d44-a5a0-c9de5de775b1" containerID="9fbf6b126621a52ea5ac1b944cc40e391e9202ea9d31bedcd4e87b7dd334e9d6" exitCode=0 Oct 31 12:59:33.065404 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:33.065307 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn" Oct 31 12:59:33.065404 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:33.065345 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn" event={"ID":"c14527eb-216b-4d44-a5a0-c9de5de775b1","Type":"ContainerDied","Data":"9fbf6b126621a52ea5ac1b944cc40e391e9202ea9d31bedcd4e87b7dd334e9d6"} Oct 31 12:59:33.065404 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:33.065394 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn" event={"ID":"c14527eb-216b-4d44-a5a0-c9de5de775b1","Type":"ContainerDied","Data":"22a008ed990c7d75b9e338a48cd893800cc23702d09466b7546e355273064285"} Oct 31 12:59:33.065530 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:33.065414 6841 scope.go:122] "RemoveContainer" containerID="9fbf6b126621a52ea5ac1b944cc40e391e9202ea9d31bedcd4e87b7dd334e9d6" Oct 31 12:59:33.118914 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:33.118887 6841 scope.go:122] "RemoveContainer" containerID="9fbf6b126621a52ea5ac1b944cc40e391e9202ea9d31bedcd4e87b7dd334e9d6" Oct 31 12:59:33.119375 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:59:33.119335 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fbf6b126621a52ea5ac1b944cc40e391e9202ea9d31bedcd4e87b7dd334e9d6\": container with ID starting with 9fbf6b126621a52ea5ac1b944cc40e391e9202ea9d31bedcd4e87b7dd334e9d6 not found: ID does not exist" containerID="9fbf6b126621a52ea5ac1b944cc40e391e9202ea9d31bedcd4e87b7dd334e9d6" Oct 31 12:59:33.119490 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:33.119389 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fbf6b126621a52ea5ac1b944cc40e391e9202ea9d31bedcd4e87b7dd334e9d6"} err="failed to get container status \"9fbf6b126621a52ea5ac1b944cc40e391e9202ea9d31bedcd4e87b7dd334e9d6\": rpc error: code = NotFound desc = could not find container \"9fbf6b126621a52ea5ac1b944cc40e391e9202ea9d31bedcd4e87b7dd334e9d6\": container with ID starting with 9fbf6b126621a52ea5ac1b944cc40e391e9202ea9d31bedcd4e87b7dd334e9d6 not found: ID does not exist" Oct 31 12:59:33.134932 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:33.133600 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn"] Oct 31 12:59:33.138102 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:33.138064 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-15c8ff-7b6bdfc8d-7j5nn"] Oct 31 12:59:33.277529 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:33.277502 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11391-2b4b25" Oct 31 12:59:33.338429 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:33.338394 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38fb7737-2348-460e-a5e6-a4e401c50b36-kube-api-access-7br7z\" (UniqueName: \"kubernetes.io/projected/38fb7737-2348-460e-a5e6-a4e401c50b36-kube-api-access-7br7z\") pod \"38fb7737-2348-460e-a5e6-a4e401c50b36\" (UID: \"38fb7737-2348-460e-a5e6-a4e401c50b36\") " Oct 31 12:59:33.340643 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:33.340576 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38fb7737-2348-460e-a5e6-a4e401c50b36-kube-api-access-7br7z" pod "38fb7737-2348-460e-a5e6-a4e401c50b36" (UID: "38fb7737-2348-460e-a5e6-a4e401c50b36"). InnerVolumeSpecName "kube-api-access-7br7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:33.377244 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:33.376067 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-15c8ff-7579cfb7c7-nnxnw" podUID="8efcc86a-0f1a-4e90-8763-fb436dfa7bfb" containerName="payload-processing" containerID="cri-o://5a2fc796f276ec2cb96331ef5dfa0f1952a1e6eef6e9d403493dc61b582f2c1f" gracePeriod=25 Oct 31 12:59:33.394295 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:33.393074 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-15c8ff-6d795ffc67-vlfh6" podUID="5a16b3c1-9c3f-4019-83c6-3ed3d085bc14" containerName="payload-pre-processing" containerID="cri-o://f67d4e2f724a0ea6f370c9cec883f7cb5acf20882d4ab7b36775600ed7dcfb13" gracePeriod=25 Oct 31 12:59:33.441114 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:33.440229 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7br7z\" (UniqueName: \"kubernetes.io/projected/38fb7737-2348-460e-a5e6-a4e401c50b36-kube-api-access-7br7z\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:33.441114 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:33.440596 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:33.649819 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:33.649782 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38fb7737-2348-460e-a5e6-a4e401c50b36" path="/var/lib/kubelet/pods/38fb7737-2348-460e-a5e6-a4e401c50b36/volumes" Oct 31 12:59:33.650433 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:33.650410 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c14527eb-216b-4d44-a5a0-c9de5de775b1" path="/var/lib/kubelet/pods/c14527eb-216b-4d44-a5a0-c9de5de775b1/volumes" Oct 31 12:59:33.685730 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:33.685692 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-15c8ff-7579cfb7c7-nnxnw" Oct 31 12:59:33.705907 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:33.705881 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-15c8ff-6d795ffc67-vlfh6" Oct 31 12:59:33.745006 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:33.744952 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8efcc86a-0f1a-4e90-8763-fb436dfa7bfb-kube-api-access-gv6vp\" (UniqueName: \"kubernetes.io/projected/8efcc86a-0f1a-4e90-8763-fb436dfa7bfb-kube-api-access-gv6vp\") pod \"8efcc86a-0f1a-4e90-8763-fb436dfa7bfb\" (UID: \"8efcc86a-0f1a-4e90-8763-fb436dfa7bfb\") " Oct 31 12:59:33.745184 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:33.745087 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8efcc86a-0f1a-4e90-8763-fb436dfa7bfb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8efcc86a-0f1a-4e90-8763-fb436dfa7bfb-plugins-config-volume\") pod \"8efcc86a-0f1a-4e90-8763-fb436dfa7bfb\" (UID: \"8efcc86a-0f1a-4e90-8763-fb436dfa7bfb\") " Oct 31 12:59:33.745368 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:33.745346 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a16b3c1-9c3f-4019-83c6-3ed3d085bc14-kube-api-access-f62tl\" (UniqueName: \"kubernetes.io/projected/5a16b3c1-9c3f-4019-83c6-3ed3d085bc14-kube-api-access-f62tl\") pod \"5a16b3c1-9c3f-4019-83c6-3ed3d085bc14\" (UID: \"5a16b3c1-9c3f-4019-83c6-3ed3d085bc14\") " Oct 31 12:59:33.745470 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:33.745445 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8efcc86a-0f1a-4e90-8763-fb436dfa7bfb-plugins-config-volume" pod "8efcc86a-0f1a-4e90-8763-fb436dfa7bfb" (UID: "8efcc86a-0f1a-4e90-8763-fb436dfa7bfb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:59:33.745525 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:33.745511 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a16b3c1-9c3f-4019-83c6-3ed3d085bc14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a16b3c1-9c3f-4019-83c6-3ed3d085bc14-plugins-config-volume\") pod \"5a16b3c1-9c3f-4019-83c6-3ed3d085bc14\" (UID: \"5a16b3c1-9c3f-4019-83c6-3ed3d085bc14\") " Oct 31 12:59:33.746255 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:33.746232 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a16b3c1-9c3f-4019-83c6-3ed3d085bc14-plugins-config-volume" pod "5a16b3c1-9c3f-4019-83c6-3ed3d085bc14" (UID: "5a16b3c1-9c3f-4019-83c6-3ed3d085bc14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:59:33.746401 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:33.746246 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8efcc86a-0f1a-4e90-8763-fb436dfa7bfb-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:33.748199 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:33.748172 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a16b3c1-9c3f-4019-83c6-3ed3d085bc14-kube-api-access-f62tl" pod "5a16b3c1-9c3f-4019-83c6-3ed3d085bc14" (UID: "5a16b3c1-9c3f-4019-83c6-3ed3d085bc14"). InnerVolumeSpecName "kube-api-access-f62tl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:33.748349 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:33.748332 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8efcc86a-0f1a-4e90-8763-fb436dfa7bfb-kube-api-access-gv6vp" pod "8efcc86a-0f1a-4e90-8763-fb436dfa7bfb" (UID: "8efcc86a-0f1a-4e90-8763-fb436dfa7bfb"). InnerVolumeSpecName "kube-api-access-gv6vp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:33.847998 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:33.847909 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f62tl\" (UniqueName: \"kubernetes.io/projected/5a16b3c1-9c3f-4019-83c6-3ed3d085bc14-kube-api-access-f62tl\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:33.847998 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:33.847946 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a16b3c1-9c3f-4019-83c6-3ed3d085bc14-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:33.847998 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:33.847961 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv6vp\" (UniqueName: \"kubernetes.io/projected/8efcc86a-0f1a-4e90-8763-fb436dfa7bfb-kube-api-access-gv6vp\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:34.073731 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:34.073698 6841 generic.go:356] "Generic (PLEG): container finished" podID="8efcc86a-0f1a-4e90-8763-fb436dfa7bfb" containerID="5a2fc796f276ec2cb96331ef5dfa0f1952a1e6eef6e9d403493dc61b582f2c1f" exitCode=0 Oct 31 12:59:34.073903 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:34.073757 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-15c8ff-7579cfb7c7-nnxnw" Oct 31 12:59:34.073903 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:34.073774 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-15c8ff-7579cfb7c7-nnxnw" event={"ID":"8efcc86a-0f1a-4e90-8763-fb436dfa7bfb","Type":"ContainerDied","Data":"5a2fc796f276ec2cb96331ef5dfa0f1952a1e6eef6e9d403493dc61b582f2c1f"} Oct 31 12:59:34.073903 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:34.073847 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-15c8ff-7579cfb7c7-nnxnw" event={"ID":"8efcc86a-0f1a-4e90-8763-fb436dfa7bfb","Type":"ContainerDied","Data":"f51159d3998cbf40c10cc61111af6b65e02b369439380d521804b328d7763f00"} Oct 31 12:59:34.073903 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:34.073878 6841 scope.go:122] "RemoveContainer" containerID="5a2fc796f276ec2cb96331ef5dfa0f1952a1e6eef6e9d403493dc61b582f2c1f" Oct 31 12:59:34.075896 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:34.075873 6841 generic.go:356] "Generic (PLEG): container finished" podID="5a16b3c1-9c3f-4019-83c6-3ed3d085bc14" containerID="f67d4e2f724a0ea6f370c9cec883f7cb5acf20882d4ab7b36775600ed7dcfb13" exitCode=0 Oct 31 12:59:34.076025 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:34.075965 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-15c8ff-6d795ffc67-vlfh6" Oct 31 12:59:34.076025 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:34.075966 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-15c8ff-6d795ffc67-vlfh6" event={"ID":"5a16b3c1-9c3f-4019-83c6-3ed3d085bc14","Type":"ContainerDied","Data":"f67d4e2f724a0ea6f370c9cec883f7cb5acf20882d4ab7b36775600ed7dcfb13"} Oct 31 12:59:34.076163 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:34.076043 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-15c8ff-6d795ffc67-vlfh6" event={"ID":"5a16b3c1-9c3f-4019-83c6-3ed3d085bc14","Type":"ContainerDied","Data":"36fb544ebd913a3c095ac45ec329d0ab4169ecbf9ccf19291bb5e9dc04711297"} Oct 31 12:59:34.077589 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:34.077564 6841 generic.go:356] "Generic (PLEG): container finished" podID="38fb7737-2348-460e-a5e6-a4e401c50b36" containerID="c4ba3630e74121bfd51da3089cc3158e7dd9e5370eeab3fa124557fbf102a712" exitCode=137 Oct 31 12:59:34.077714 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:34.077632 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11391-2b4b25" Oct 31 12:59:34.088120 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:34.088100 6841 scope.go:122] "RemoveContainer" containerID="5a2fc796f276ec2cb96331ef5dfa0f1952a1e6eef6e9d403493dc61b582f2c1f" Oct 31 12:59:34.088406 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:59:34.088387 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a2fc796f276ec2cb96331ef5dfa0f1952a1e6eef6e9d403493dc61b582f2c1f\": container with ID starting with 5a2fc796f276ec2cb96331ef5dfa0f1952a1e6eef6e9d403493dc61b582f2c1f not found: ID does not exist" containerID="5a2fc796f276ec2cb96331ef5dfa0f1952a1e6eef6e9d403493dc61b582f2c1f" Oct 31 12:59:34.088506 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:34.088413 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a2fc796f276ec2cb96331ef5dfa0f1952a1e6eef6e9d403493dc61b582f2c1f"} err="failed to get container status \"5a2fc796f276ec2cb96331ef5dfa0f1952a1e6eef6e9d403493dc61b582f2c1f\": rpc error: code = NotFound desc = could not find container \"5a2fc796f276ec2cb96331ef5dfa0f1952a1e6eef6e9d403493dc61b582f2c1f\": container with ID starting with 5a2fc796f276ec2cb96331ef5dfa0f1952a1e6eef6e9d403493dc61b582f2c1f not found: ID does not exist" Oct 31 12:59:34.088506 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:34.088431 6841 scope.go:122] "RemoveContainer" containerID="f67d4e2f724a0ea6f370c9cec883f7cb5acf20882d4ab7b36775600ed7dcfb13" Oct 31 12:59:34.097252 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:34.097232 6841 scope.go:122] "RemoveContainer" containerID="f67d4e2f724a0ea6f370c9cec883f7cb5acf20882d4ab7b36775600ed7dcfb13" Oct 31 12:59:34.097551 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:59:34.097534 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f67d4e2f724a0ea6f370c9cec883f7cb5acf20882d4ab7b36775600ed7dcfb13\": container with ID starting with f67d4e2f724a0ea6f370c9cec883f7cb5acf20882d4ab7b36775600ed7dcfb13 not found: ID does not exist" containerID="f67d4e2f724a0ea6f370c9cec883f7cb5acf20882d4ab7b36775600ed7dcfb13" Oct 31 12:59:34.097600 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:34.097561 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f67d4e2f724a0ea6f370c9cec883f7cb5acf20882d4ab7b36775600ed7dcfb13"} err="failed to get container status \"f67d4e2f724a0ea6f370c9cec883f7cb5acf20882d4ab7b36775600ed7dcfb13\": rpc error: code = NotFound desc = could not find container \"f67d4e2f724a0ea6f370c9cec883f7cb5acf20882d4ab7b36775600ed7dcfb13\": container with ID starting with f67d4e2f724a0ea6f370c9cec883f7cb5acf20882d4ab7b36775600ed7dcfb13 not found: ID does not exist" Oct 31 12:59:34.097600 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:34.097577 6841 scope.go:122] "RemoveContainer" containerID="c4ba3630e74121bfd51da3089cc3158e7dd9e5370eeab3fa124557fbf102a712" Oct 31 12:59:34.104556 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:34.104526 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-15c8ff-6d795ffc67-vlfh6"] Oct 31 12:59:34.108969 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:34.108944 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-15c8ff-6d795ffc67-vlfh6"] Oct 31 12:59:34.109870 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:34.109854 6841 scope.go:122] "RemoveContainer" containerID="c4ba3630e74121bfd51da3089cc3158e7dd9e5370eeab3fa124557fbf102a712" Oct 31 12:59:34.110175 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:59:34.110156 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4ba3630e74121bfd51da3089cc3158e7dd9e5370eeab3fa124557fbf102a712\": container with ID starting with c4ba3630e74121bfd51da3089cc3158e7dd9e5370eeab3fa124557fbf102a712 not found: ID does not exist" containerID="c4ba3630e74121bfd51da3089cc3158e7dd9e5370eeab3fa124557fbf102a712" Oct 31 12:59:34.110224 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:34.110184 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4ba3630e74121bfd51da3089cc3158e7dd9e5370eeab3fa124557fbf102a712"} err="failed to get container status \"c4ba3630e74121bfd51da3089cc3158e7dd9e5370eeab3fa124557fbf102a712\": rpc error: code = NotFound desc = could not find container \"c4ba3630e74121bfd51da3089cc3158e7dd9e5370eeab3fa124557fbf102a712\": container with ID starting with c4ba3630e74121bfd51da3089cc3158e7dd9e5370eeab3fa124557fbf102a712 not found: ID does not exist" Oct 31 12:59:34.122437 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:34.122416 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-15c8ff-7579cfb7c7-nnxnw"] Oct 31 12:59:34.127881 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:34.127858 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-15c8ff-7579cfb7c7-nnxnw"] Oct 31 12:59:34.224029 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:34.210706 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:34.784498 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:34.784363 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:34.982388 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:34.982353 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:35.291112 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:35.291068 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:35.543683 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:35.543651 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:35.677074 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:35.674610 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a16b3c1-9c3f-4019-83c6-3ed3d085bc14" path="/var/lib/kubelet/pods/5a16b3c1-9c3f-4019-83c6-3ed3d085bc14/volumes" Oct 31 12:59:35.677074 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:35.675566 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8efcc86a-0f1a-4e90-8763-fb436dfa7bfb" path="/var/lib/kubelet/pods/8efcc86a-0f1a-4e90-8763-fb436dfa7bfb/volumes" Oct 31 12:59:35.771050 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:35.767075 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:35.982550 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:35.982514 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:36.098040 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:36.098010 6841 generic.go:356] "Generic (PLEG): container finished" podID="ad431a60-f82d-4b0b-928d-dec7ff77f067" containerID="b1ee81c03b1d8adac98f61d130c9d997e94dc5767a755c0992b3c980210bb0e0" exitCode=0 Oct 31 12:59:36.098040 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:36.098029 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" event={"ID":"ad431a60-f82d-4b0b-928d-dec7ff77f067","Type":"ContainerDied","Data":"b1ee81c03b1d8adac98f61d130c9d997e94dc5767a755c0992b3c980210bb0e0"} Oct 31 12:59:36.098488 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:36.098462 6841 scope.go:122] "RemoveContainer" containerID="b1ee81c03b1d8adac98f61d130c9d997e94dc5767a755c0992b3c980210bb0e0" Oct 31 12:59:36.100129 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:36.100102 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:59:36.100209 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:36.100166 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:59:36.100209 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:36.100198 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:59:36.421183 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:36.421107 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:36.639261 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:36.639216 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:36.864005 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:36.862005 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:37.107012 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:37.106964 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" event={"ID":"ad431a60-f82d-4b0b-928d-dec7ff77f067","Type":"ContainerStarted","Data":"9badce8a2785675a76d09d8fba11270834b28fa9a30d6a238d79f5792455d3bf"} Oct 31 12:59:37.484318 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:37.484283 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:59:37.485653 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:37.485631 6841 patch_prober.go:29] interesting pod/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 12:59:37.485719 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:37.485691 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" podUID="ad431a60-f82d-4b0b-928d-dec7ff77f067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 12:59:37.639003 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:37.633804 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:37.867606 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:37.867489 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:38.067472 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:38.067437 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:38.491019 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:38.489485 6841 patch_prober.go:29] interesting pod/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 12:59:38.491019 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:38.489559 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" podUID="ad431a60-f82d-4b0b-928d-dec7ff77f067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 12:59:38.543004 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:38.537594 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:39.224022 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:39.221095 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:39.484586 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:39.484503 6841 patch_prober.go:29] interesting pod/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 12:59:39.484586 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:39.484560 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" podUID="ad431a60-f82d-4b0b-928d-dec7ff77f067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 12:59:39.908058 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:39.905857 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:40.485078 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:40.485042 6841 patch_prober.go:29] interesting pod/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 12:59:40.485242 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:40.485103 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" podUID="ad431a60-f82d-4b0b-928d-dec7ff77f067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 12:59:40.627005 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:40.613922 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:41.398041 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:41.395198 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:41.485163 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:41.485131 6841 patch_prober.go:29] interesting pod/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 12:59:41.485336 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:41.485216 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" podUID="ad431a60-f82d-4b0b-928d-dec7ff77f067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 12:59:42.022007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:42.020062 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:42.231044 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:42.229143 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-21c556-7f5b69bb87-xd67g"] Oct 31 12:59:42.390197 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:42.390076 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:42.601011 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:42.594950 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:42.866000 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:42.865955 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:59:42.866526 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:42.866501 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:59:42.867349 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:42.867329 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:59:43.018601 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:43.018566 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:43.366183 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:43.366151 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:43.559729 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:43.559692 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp"] Oct 31 12:59:43.560026 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:43.559994 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp" podUID="c25646f3-cd9d-493c-bfef-692bf398d20f" containerName="maas-api" containerID="cri-o://bbc75af6dc03cb69db4ada126f12254513fde386bdd1d60640143ac66b8437cb" gracePeriod=30 Oct 31 12:59:43.720341 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:43.720306 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:44.092640 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:44.092541 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:44.157844 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:44.157783 6841 generic.go:356] "Generic (PLEG): container finished" podID="c25646f3-cd9d-493c-bfef-692bf398d20f" containerID="bbc75af6dc03cb69db4ada126f12254513fde386bdd1d60640143ac66b8437cb" exitCode=0 Oct 31 12:59:44.158017 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:44.157853 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp" event={"ID":"c25646f3-cd9d-493c-bfef-692bf398d20f","Type":"ContainerDied","Data":"bbc75af6dc03cb69db4ada126f12254513fde386bdd1d60640143ac66b8437cb"} Oct 31 12:59:44.158017 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:44.157911 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp" event={"ID":"c25646f3-cd9d-493c-bfef-692bf398d20f","Type":"ContainerDied","Data":"53f71c8a7a59a564998ca9ab6e116b8ce583284b3e5e0e9f226b8d5c9ddf8e39"} Oct 31 12:59:44.158017 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:44.157935 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53f71c8a7a59a564998ca9ab6e116b8ce583284b3e5e0e9f226b8d5c9ddf8e39" Oct 31 12:59:44.159589 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:44.159570 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp" Oct 31 12:59:44.278094 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:44.278043 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c25646f3-cd9d-493c-bfef-692bf398d20f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c25646f3-cd9d-493c-bfef-692bf398d20f-maas-api-tls\") pod \"c25646f3-cd9d-493c-bfef-692bf398d20f\" (UID: \"c25646f3-cd9d-493c-bfef-692bf398d20f\") " Oct 31 12:59:44.278282 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:44.278206 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c25646f3-cd9d-493c-bfef-692bf398d20f-kube-api-access-gg2hw\" (UniqueName: \"kubernetes.io/projected/c25646f3-cd9d-493c-bfef-692bf398d20f-kube-api-access-gg2hw\") pod \"c25646f3-cd9d-493c-bfef-692bf398d20f\" (UID: \"c25646f3-cd9d-493c-bfef-692bf398d20f\") " Oct 31 12:59:44.278282 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:44.278268 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c25646f3-cd9d-493c-bfef-692bf398d20f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c25646f3-cd9d-493c-bfef-692bf398d20f-openshift-service-ca\") pod \"c25646f3-cd9d-493c-bfef-692bf398d20f\" (UID: \"c25646f3-cd9d-493c-bfef-692bf398d20f\") " Oct 31 12:59:44.278439 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:44.278420 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c25646f3-cd9d-493c-bfef-692bf398d20f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c25646f3-cd9d-493c-bfef-692bf398d20f-metrics-certs\") pod \"c25646f3-cd9d-493c-bfef-692bf398d20f\" (UID: \"c25646f3-cd9d-493c-bfef-692bf398d20f\") " Oct 31 12:59:44.278646 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:44.278618 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c25646f3-cd9d-493c-bfef-692bf398d20f-openshift-service-ca" pod "c25646f3-cd9d-493c-bfef-692bf398d20f" (UID: "c25646f3-cd9d-493c-bfef-692bf398d20f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:59:44.279175 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:44.279114 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c25646f3-cd9d-493c-bfef-692bf398d20f-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:44.280247 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:44.280221 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c25646f3-cd9d-493c-bfef-692bf398d20f-maas-api-tls" pod "c25646f3-cd9d-493c-bfef-692bf398d20f" (UID: "c25646f3-cd9d-493c-bfef-692bf398d20f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:59:44.280384 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:44.280365 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c25646f3-cd9d-493c-bfef-692bf398d20f-kube-api-access-gg2hw" pod "c25646f3-cd9d-493c-bfef-692bf398d20f" (UID: "c25646f3-cd9d-493c-bfef-692bf398d20f"). InnerVolumeSpecName "kube-api-access-gg2hw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:44.280433 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:44.280405 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c25646f3-cd9d-493c-bfef-692bf398d20f-metrics-certs" pod "c25646f3-cd9d-493c-bfef-692bf398d20f" (UID: "c25646f3-cd9d-493c-bfef-692bf398d20f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:59:44.380118 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:44.380044 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c25646f3-cd9d-493c-bfef-692bf398d20f-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:44.380118 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:44.380071 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c25646f3-cd9d-493c-bfef-692bf398d20f-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:44.380118 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:44.380081 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg2hw\" (UniqueName: \"kubernetes.io/projected/c25646f3-cd9d-493c-bfef-692bf398d20f-kube-api-access-gg2hw\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:45.153007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:45.152960 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:45.165220 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:45.165195 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp" Oct 31 12:59:45.193625 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:45.193597 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp"] Oct 31 12:59:45.198187 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:45.198163 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-21c556-799d6bb5c6-6bhwp"] Oct 31 12:59:45.643539 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:45.643508 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c25646f3-cd9d-493c-bfef-692bf398d20f" path="/var/lib/kubelet/pods/c25646f3-cd9d-493c-bfef-692bf398d20f/volumes" Oct 31 12:59:46.040499 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:46.040460 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:46.816796 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:46.816756 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:47.229721 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:47.229684 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-21c556-7f5b69bb87-xd67g" podUID="312077be-9bec-47bf-9479-d8710e4740c2" containerName="payload-pre-processing" containerID="cri-o://09dff4e745d2cd7deb94c84c5b819541f8c3b71c6df22bfa45c7ef8275f1debf" gracePeriod=25 Oct 31 12:59:47.512305 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:47.512279 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-21c556-7f5b69bb87-xd67g" Oct 31 12:59:47.605019 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:47.604965 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:47.632488 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:47.632442 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/312077be-9bec-47bf-9479-d8710e4740c2-kube-api-access-sk5zn\" (UniqueName: \"kubernetes.io/projected/312077be-9bec-47bf-9479-d8710e4740c2-kube-api-access-sk5zn\") pod \"312077be-9bec-47bf-9479-d8710e4740c2\" (UID: \"312077be-9bec-47bf-9479-d8710e4740c2\") " Oct 31 12:59:47.632798 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:47.632772 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/312077be-9bec-47bf-9479-d8710e4740c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/312077be-9bec-47bf-9479-d8710e4740c2-plugins-config-volume\") pod \"312077be-9bec-47bf-9479-d8710e4740c2\" (UID: \"312077be-9bec-47bf-9479-d8710e4740c2\") " Oct 31 12:59:47.645906 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:47.642312 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/312077be-9bec-47bf-9479-d8710e4740c2-plugins-config-volume" pod "312077be-9bec-47bf-9479-d8710e4740c2" (UID: "312077be-9bec-47bf-9479-d8710e4740c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:59:47.661535 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:47.661493 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/312077be-9bec-47bf-9479-d8710e4740c2-kube-api-access-sk5zn" pod "312077be-9bec-47bf-9479-d8710e4740c2" (UID: "312077be-9bec-47bf-9479-d8710e4740c2"). InnerVolumeSpecName "kube-api-access-sk5zn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:47.734864 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:47.734811 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk5zn\" (UniqueName: \"kubernetes.io/projected/312077be-9bec-47bf-9479-d8710e4740c2-kube-api-access-sk5zn\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:47.734864 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:47.734850 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/312077be-9bec-47bf-9479-d8710e4740c2-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:48.190958 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:48.190923 6841 generic.go:356] "Generic (PLEG): container finished" podID="312077be-9bec-47bf-9479-d8710e4740c2" containerID="09dff4e745d2cd7deb94c84c5b819541f8c3b71c6df22bfa45c7ef8275f1debf" exitCode=0 Oct 31 12:59:48.191367 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:48.191014 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-21c556-7f5b69bb87-xd67g" Oct 31 12:59:48.191367 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:48.191010 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-21c556-7f5b69bb87-xd67g" event={"ID":"312077be-9bec-47bf-9479-d8710e4740c2","Type":"ContainerDied","Data":"09dff4e745d2cd7deb94c84c5b819541f8c3b71c6df22bfa45c7ef8275f1debf"} Oct 31 12:59:48.191367 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:48.191131 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-21c556-7f5b69bb87-xd67g" event={"ID":"312077be-9bec-47bf-9479-d8710e4740c2","Type":"ContainerDied","Data":"c46d2d46aac85de46786ec77f81c40588c59f2266c7c5e078195f60118e283ef"} Oct 31 12:59:48.191367 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:48.191155 6841 scope.go:122] "RemoveContainer" containerID="09dff4e745d2cd7deb94c84c5b819541f8c3b71c6df22bfa45c7ef8275f1debf" Oct 31 12:59:48.203814 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:48.203797 6841 scope.go:122] "RemoveContainer" containerID="09dff4e745d2cd7deb94c84c5b819541f8c3b71c6df22bfa45c7ef8275f1debf" Oct 31 12:59:48.204117 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:59:48.204095 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09dff4e745d2cd7deb94c84c5b819541f8c3b71c6df22bfa45c7ef8275f1debf\": container with ID starting with 09dff4e745d2cd7deb94c84c5b819541f8c3b71c6df22bfa45c7ef8275f1debf not found: ID does not exist" containerID="09dff4e745d2cd7deb94c84c5b819541f8c3b71c6df22bfa45c7ef8275f1debf" Oct 31 12:59:48.204209 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:48.204124 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09dff4e745d2cd7deb94c84c5b819541f8c3b71c6df22bfa45c7ef8275f1debf"} err="failed to get container status \"09dff4e745d2cd7deb94c84c5b819541f8c3b71c6df22bfa45c7ef8275f1debf\": rpc error: code = NotFound desc = could not find container \"09dff4e745d2cd7deb94c84c5b819541f8c3b71c6df22bfa45c7ef8275f1debf\": container with ID starting with 09dff4e745d2cd7deb94c84c5b819541f8c3b71c6df22bfa45c7ef8275f1debf not found: ID does not exist" Oct 31 12:59:48.234661 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:48.234633 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-21c556-7f5b69bb87-xd67g"] Oct 31 12:59:48.274007 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:48.271052 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-21c556-7f5b69bb87-xd67g"] Oct 31 12:59:48.370194 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:48.370149 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:49.049048 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:49.049015 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:49.642729 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:49.642685 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="312077be-9bec-47bf-9479-d8710e4740c2" path="/var/lib/kubelet/pods/312077be-9bec-47bf-9479-d8710e4740c2/volumes" Oct 31 12:59:49.877004 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:49.873170 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:50.034969 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:59:50.034936 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 12:59:50.344608 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:50.344582 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:50.656064 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:50.655950 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:50.861165 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:50.861131 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf"] Oct 31 12:59:50.861367 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:50.861330 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" podUID="ad431a60-f82d-4b0b-928d-dec7ff77f067" containerName="istio-proxy" containerID="cri-o://9badce8a2785675a76d09d8fba11270834b28fa9a30d6a238d79f5792455d3bf" gracePeriod=30 Oct 31 12:59:51.029250 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:51.029215 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:51.433146 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:51.433047 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:52.230812 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:52.230758 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:53.175504 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:53.175468 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:53.951274 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:53.950922 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:54.769557 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:54.768046 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:55.626482 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:55.626441 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:56.151665 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.151603 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:59:56.224302 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.224268 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad431a60-f82d-4b0b-928d-dec7ff77f067-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad431a60-f82d-4b0b-928d-dec7ff77f067-istio-podinfo\") pod \"ad431a60-f82d-4b0b-928d-dec7ff77f067\" (UID: \"ad431a60-f82d-4b0b-928d-dec7ff77f067\") " Oct 31 12:59:56.224469 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.224331 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad431a60-f82d-4b0b-928d-dec7ff77f067-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad431a60-f82d-4b0b-928d-dec7ff77f067-credential-socket\") pod \"ad431a60-f82d-4b0b-928d-dec7ff77f067\" (UID: \"ad431a60-f82d-4b0b-928d-dec7ff77f067\") " Oct 31 12:59:56.224469 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.224425 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad431a60-f82d-4b0b-928d-dec7ff77f067-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad431a60-f82d-4b0b-928d-dec7ff77f067-istiod-ca-cert\") pod \"ad431a60-f82d-4b0b-928d-dec7ff77f067\" (UID: \"ad431a60-f82d-4b0b-928d-dec7ff77f067\") " Oct 31 12:59:56.224629 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.224496 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad431a60-f82d-4b0b-928d-dec7ff77f067-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad431a60-f82d-4b0b-928d-dec7ff77f067-workload-certs\") pod \"ad431a60-f82d-4b0b-928d-dec7ff77f067\" (UID: \"ad431a60-f82d-4b0b-928d-dec7ff77f067\") " Oct 31 12:59:56.224629 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.224586 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad431a60-f82d-4b0b-928d-dec7ff77f067-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad431a60-f82d-4b0b-928d-dec7ff77f067-istio-envoy\") pod \"ad431a60-f82d-4b0b-928d-dec7ff77f067\" (UID: \"ad431a60-f82d-4b0b-928d-dec7ff77f067\") " Oct 31 12:59:56.224752 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.224735 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad431a60-f82d-4b0b-928d-dec7ff77f067-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad431a60-f82d-4b0b-928d-dec7ff77f067-istio-data\") pod \"ad431a60-f82d-4b0b-928d-dec7ff77f067\" (UID: \"ad431a60-f82d-4b0b-928d-dec7ff77f067\") " Oct 31 12:59:56.224814 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.224796 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad431a60-f82d-4b0b-928d-dec7ff77f067-kube-api-access-pkmck\" (UniqueName: \"kubernetes.io/projected/ad431a60-f82d-4b0b-928d-dec7ff77f067-kube-api-access-pkmck\") pod \"ad431a60-f82d-4b0b-928d-dec7ff77f067\" (UID: \"ad431a60-f82d-4b0b-928d-dec7ff77f067\") " Oct 31 12:59:56.224909 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.224885 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad431a60-f82d-4b0b-928d-dec7ff77f067-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad431a60-f82d-4b0b-928d-dec7ff77f067-workload-socket\") pod \"ad431a60-f82d-4b0b-928d-dec7ff77f067\" (UID: \"ad431a60-f82d-4b0b-928d-dec7ff77f067\") " Oct 31 12:59:56.224971 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.224954 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad431a60-f82d-4b0b-928d-dec7ff77f067-istio-token\" (UniqueName: \"kubernetes.io/projected/ad431a60-f82d-4b0b-928d-dec7ff77f067-istio-token\") pod \"ad431a60-f82d-4b0b-928d-dec7ff77f067\" (UID: \"ad431a60-f82d-4b0b-928d-dec7ff77f067\") " Oct 31 12:59:56.225054 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.225028 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad431a60-f82d-4b0b-928d-dec7ff77f067-workload-certs" pod "ad431a60-f82d-4b0b-928d-dec7ff77f067" (UID: "ad431a60-f82d-4b0b-928d-dec7ff77f067"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:56.225355 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.225199 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad431a60-f82d-4b0b-928d-dec7ff77f067-istio-data" pod "ad431a60-f82d-4b0b-928d-dec7ff77f067" (UID: "ad431a60-f82d-4b0b-928d-dec7ff77f067"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:56.225469 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.225378 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad431a60-f82d-4b0b-928d-dec7ff77f067-credential-socket" pod "ad431a60-f82d-4b0b-928d-dec7ff77f067" (UID: "ad431a60-f82d-4b0b-928d-dec7ff77f067"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:56.225798 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.225735 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad431a60-f82d-4b0b-928d-dec7ff77f067-workload-socket" pod "ad431a60-f82d-4b0b-928d-dec7ff77f067" (UID: "ad431a60-f82d-4b0b-928d-dec7ff77f067"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:56.225798 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.225744 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad431a60-f82d-4b0b-928d-dec7ff77f067-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:56.225798 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.225783 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad431a60-f82d-4b0b-928d-dec7ff77f067-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:56.226013 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.225801 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad431a60-f82d-4b0b-928d-dec7ff77f067-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:56.226190 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.226167 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad431a60-f82d-4b0b-928d-dec7ff77f067-istiod-ca-cert" pod "ad431a60-f82d-4b0b-928d-dec7ff77f067" (UID: "ad431a60-f82d-4b0b-928d-dec7ff77f067"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:59:56.228200 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.228152 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad431a60-f82d-4b0b-928d-dec7ff77f067-istio-envoy" pod "ad431a60-f82d-4b0b-928d-dec7ff77f067" (UID: "ad431a60-f82d-4b0b-928d-dec7ff77f067"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:56.228321 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.228280 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad431a60-f82d-4b0b-928d-dec7ff77f067-kube-api-access-pkmck" pod "ad431a60-f82d-4b0b-928d-dec7ff77f067" (UID: "ad431a60-f82d-4b0b-928d-dec7ff77f067"). InnerVolumeSpecName "kube-api-access-pkmck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:56.228894 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.228876 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad431a60-f82d-4b0b-928d-dec7ff77f067-istio-token" pod "ad431a60-f82d-4b0b-928d-dec7ff77f067" (UID: "ad431a60-f82d-4b0b-928d-dec7ff77f067"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:56.229085 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.229067 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad431a60-f82d-4b0b-928d-dec7ff77f067-istio-podinfo" pod "ad431a60-f82d-4b0b-928d-dec7ff77f067" (UID: "ad431a60-f82d-4b0b-928d-dec7ff77f067"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:59:56.266242 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.266205 6841 generic.go:356] "Generic (PLEG): container finished" podID="ad431a60-f82d-4b0b-928d-dec7ff77f067" containerID="9badce8a2785675a76d09d8fba11270834b28fa9a30d6a238d79f5792455d3bf" exitCode=0 Oct 31 12:59:56.266393 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.266260 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" Oct 31 12:59:56.266393 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.266277 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" event={"ID":"ad431a60-f82d-4b0b-928d-dec7ff77f067","Type":"ContainerDied","Data":"9badce8a2785675a76d09d8fba11270834b28fa9a30d6a238d79f5792455d3bf"} Oct 31 12:59:56.266393 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.266342 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf" event={"ID":"ad431a60-f82d-4b0b-928d-dec7ff77f067","Type":"ContainerDied","Data":"acddfcafba90047f1d7531c69526547217c6b69b62e6d0aef2cc8b9934a6753b"} Oct 31 12:59:56.266577 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.266374 6841 scope.go:122] "RemoveContainer" containerID="9badce8a2785675a76d09d8fba11270834b28fa9a30d6a238d79f5792455d3bf" Oct 31 12:59:56.281542 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.281516 6841 scope.go:122] "RemoveContainer" containerID="b1ee81c03b1d8adac98f61d130c9d997e94dc5767a755c0992b3c980210bb0e0" Oct 31 12:59:56.292327 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.292301 6841 scope.go:122] "RemoveContainer" containerID="9badce8a2785675a76d09d8fba11270834b28fa9a30d6a238d79f5792455d3bf" Oct 31 12:59:56.292636 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:59:56.292613 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9badce8a2785675a76d09d8fba11270834b28fa9a30d6a238d79f5792455d3bf\": container with ID starting with 9badce8a2785675a76d09d8fba11270834b28fa9a30d6a238d79f5792455d3bf not found: ID does not exist" containerID="9badce8a2785675a76d09d8fba11270834b28fa9a30d6a238d79f5792455d3bf" Oct 31 12:59:56.292720 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.292646 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9badce8a2785675a76d09d8fba11270834b28fa9a30d6a238d79f5792455d3bf"} err="failed to get container status \"9badce8a2785675a76d09d8fba11270834b28fa9a30d6a238d79f5792455d3bf\": rpc error: code = NotFound desc = could not find container \"9badce8a2785675a76d09d8fba11270834b28fa9a30d6a238d79f5792455d3bf\": container with ID starting with 9badce8a2785675a76d09d8fba11270834b28fa9a30d6a238d79f5792455d3bf not found: ID does not exist" Oct 31 12:59:56.292720 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.292669 6841 scope.go:122] "RemoveContainer" containerID="b1ee81c03b1d8adac98f61d130c9d997e94dc5767a755c0992b3c980210bb0e0" Oct 31 12:59:56.292942 ip-10-0-138-70 kubenswrapper[6841]: E1031 12:59:56.292924 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1ee81c03b1d8adac98f61d130c9d997e94dc5767a755c0992b3c980210bb0e0\": container with ID starting with b1ee81c03b1d8adac98f61d130c9d997e94dc5767a755c0992b3c980210bb0e0 not found: ID does not exist" containerID="b1ee81c03b1d8adac98f61d130c9d997e94dc5767a755c0992b3c980210bb0e0" Oct 31 12:59:56.293037 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.292947 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1ee81c03b1d8adac98f61d130c9d997e94dc5767a755c0992b3c980210bb0e0"} err="failed to get container status \"b1ee81c03b1d8adac98f61d130c9d997e94dc5767a755c0992b3c980210bb0e0\": rpc error: code = NotFound desc = could not find container \"b1ee81c03b1d8adac98f61d130c9d997e94dc5767a755c0992b3c980210bb0e0\": container with ID starting with b1ee81c03b1d8adac98f61d130c9d997e94dc5767a755c0992b3c980210bb0e0 not found: ID does not exist" Oct 31 12:59:56.301452 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.301431 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf"] Oct 31 12:59:56.319057 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.319027 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-21c556-openshift-default-66fdbf578c-2srxf"] Oct 31 12:59:56.328088 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.328061 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkmck\" (UniqueName: \"kubernetes.io/projected/ad431a60-f82d-4b0b-928d-dec7ff77f067-kube-api-access-pkmck\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:56.328189 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.328094 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad431a60-f82d-4b0b-928d-dec7ff77f067-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:56.328189 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.328112 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad431a60-f82d-4b0b-928d-dec7ff77f067-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:56.328189 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.328126 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad431a60-f82d-4b0b-928d-dec7ff77f067-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:56.328189 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.328141 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad431a60-f82d-4b0b-928d-dec7ff77f067-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:56.328189 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.328154 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad431a60-f82d-4b0b-928d-dec7ff77f067-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:56.597938 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:56.597906 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:57.313713 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:57.313651 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:57.643790 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:57.643713 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad431a60-f82d-4b0b-928d-dec7ff77f067" path="/var/lib/kubelet/pods/ad431a60-f82d-4b0b-928d-dec7ff77f067/volumes" Oct 31 12:59:58.011797 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:58.011763 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:58.745059 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:58.745023 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:58.974619 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:58.974564 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:59.265590 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.265556 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg"] Oct 31 12:59:59.266608 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.266583 6841 cpu_manager.go:414] "containerMap: removing container" podUID="c14527eb-216b-4d44-a5a0-c9de5de775b1" containerName="maas-api" Oct 31 12:59:59.266686 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.266610 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c14527eb-216b-4d44-a5a0-c9de5de775b1" containerName="maas-api" Oct 31 12:59:59.266686 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.266637 6841 cpu_manager.go:414] "containerMap: removing container" podUID="312077be-9bec-47bf-9479-d8710e4740c2" containerName="payload-pre-processing" Oct 31 12:59:59.266686 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.266645 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="312077be-9bec-47bf-9479-d8710e4740c2" containerName="payload-pre-processing" Oct 31 12:59:59.266686 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.266660 6841 cpu_manager.go:414] "containerMap: removing container" podUID="38fb7737-2348-460e-a5e6-a4e401c50b36" containerName="test-curl-11391-2b4b25" Oct 31 12:59:59.266686 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.266668 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38fb7737-2348-460e-a5e6-a4e401c50b36" containerName="test-curl-11391-2b4b25" Oct 31 12:59:59.266838 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.266711 6841 cpu_manager.go:414] "containerMap: removing container" podUID="c25646f3-cd9d-493c-bfef-692bf398d20f" containerName="maas-api" Oct 31 12:59:59.266838 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.266719 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c25646f3-cd9d-493c-bfef-692bf398d20f" containerName="maas-api" Oct 31 12:59:59.266838 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.266732 6841 cpu_manager.go:414] "containerMap: removing container" podUID="ad431a60-f82d-4b0b-928d-dec7ff77f067" containerName="istio-proxy" Oct 31 12:59:59.266838 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.266739 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad431a60-f82d-4b0b-928d-dec7ff77f067" containerName="istio-proxy" Oct 31 12:59:59.266838 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.266755 6841 cpu_manager.go:414] "containerMap: removing container" podUID="ad431a60-f82d-4b0b-928d-dec7ff77f067" containerName="istio-proxy" Oct 31 12:59:59.266838 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.266761 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad431a60-f82d-4b0b-928d-dec7ff77f067" containerName="istio-proxy" Oct 31 12:59:59.266838 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.266781 6841 cpu_manager.go:414] "containerMap: removing container" podUID="5a16b3c1-9c3f-4019-83c6-3ed3d085bc14" containerName="payload-pre-processing" Oct 31 12:59:59.266838 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.266788 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a16b3c1-9c3f-4019-83c6-3ed3d085bc14" containerName="payload-pre-processing" Oct 31 12:59:59.266838 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.266802 6841 cpu_manager.go:414] "containerMap: removing container" podUID="8efcc86a-0f1a-4e90-8763-fb436dfa7bfb" containerName="payload-processing" Oct 31 12:59:59.266838 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.266810 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8efcc86a-0f1a-4e90-8763-fb436dfa7bfb" containerName="payload-processing" Oct 31 12:59:59.267161 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.266958 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad431a60-f82d-4b0b-928d-dec7ff77f067" containerName="istio-proxy" Oct 31 12:59:59.267161 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.266971 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a16b3c1-9c3f-4019-83c6-3ed3d085bc14" containerName="payload-pre-processing" Oct 31 12:59:59.267161 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.267000 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="8efcc86a-0f1a-4e90-8763-fb436dfa7bfb" containerName="payload-processing" Oct 31 12:59:59.267161 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.267015 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="c14527eb-216b-4d44-a5a0-c9de5de775b1" containerName="maas-api" Oct 31 12:59:59.267161 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.267027 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="312077be-9bec-47bf-9479-d8710e4740c2" containerName="payload-pre-processing" Oct 31 12:59:59.267161 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.267037 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="38fb7737-2348-460e-a5e6-a4e401c50b36" containerName="test-curl-11391-2b4b25" Oct 31 12:59:59.267161 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.267054 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="c25646f3-cd9d-493c-bfef-692bf398d20f" containerName="maas-api" Oct 31 12:59:59.272066 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.272044 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 12:59:59.282841 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.282802 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-3128e5-openshift-default-dockercfg-chgx7\"" Oct 31 12:59:59.289329 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.289300 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg"] Oct 31 12:59:59.426013 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.424646 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35d138be-ff3b-43d9-ae6b-bae298aaec6b-istio-data\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg\" (UID: \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 12:59:59.426013 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.424717 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phqpc\" (UniqueName: \"kubernetes.io/projected/35d138be-ff3b-43d9-ae6b-bae298aaec6b-kube-api-access-phqpc\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg\" (UID: \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 12:59:59.426013 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.424759 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35d138be-ff3b-43d9-ae6b-bae298aaec6b-istio-envoy\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg\" (UID: \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 12:59:59.426013 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.425079 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35d138be-ff3b-43d9-ae6b-bae298aaec6b-istio-token\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg\" (UID: \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 12:59:59.426013 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.425119 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d138be-ff3b-43d9-ae6b-bae298aaec6b-credential-socket\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg\" (UID: \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 12:59:59.426013 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.425185 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d138be-ff3b-43d9-ae6b-bae298aaec6b-workload-socket\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg\" (UID: \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 12:59:59.426013 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.425227 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35d138be-ff3b-43d9-ae6b-bae298aaec6b-istiod-ca-cert\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg\" (UID: \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 12:59:59.426013 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.425253 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35d138be-ff3b-43d9-ae6b-bae298aaec6b-istio-podinfo\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg\" (UID: \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 12:59:59.426013 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.425290 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35d138be-ff3b-43d9-ae6b-bae298aaec6b-workload-certs\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg\" (UID: \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 12:59:59.452003 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.446216 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:59.537260 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.537165 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35d138be-ff3b-43d9-ae6b-bae298aaec6b-istio-token\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg\" (UID: \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 12:59:59.537845 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.537823 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d138be-ff3b-43d9-ae6b-bae298aaec6b-credential-socket\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg\" (UID: \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 12:59:59.538033 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.538018 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d138be-ff3b-43d9-ae6b-bae298aaec6b-workload-socket\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg\" (UID: \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 12:59:59.538178 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.538162 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35d138be-ff3b-43d9-ae6b-bae298aaec6b-istiod-ca-cert\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg\" (UID: \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 12:59:59.538284 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.538270 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35d138be-ff3b-43d9-ae6b-bae298aaec6b-istio-podinfo\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg\" (UID: \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 12:59:59.538390 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.538375 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35d138be-ff3b-43d9-ae6b-bae298aaec6b-workload-certs\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg\" (UID: \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 12:59:59.538542 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.538524 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35d138be-ff3b-43d9-ae6b-bae298aaec6b-istio-data\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg\" (UID: \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 12:59:59.538690 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.538655 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phqpc\" (UniqueName: \"kubernetes.io/projected/35d138be-ff3b-43d9-ae6b-bae298aaec6b-kube-api-access-phqpc\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg\" (UID: \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 12:59:59.538823 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.538807 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35d138be-ff3b-43d9-ae6b-bae298aaec6b-istio-envoy\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg\" (UID: \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 12:59:59.545001 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.541663 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d138be-ff3b-43d9-ae6b-bae298aaec6b-credential-socket\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg\" (UID: \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 12:59:59.545001 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.542475 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35d138be-ff3b-43d9-ae6b-bae298aaec6b-istiod-ca-cert\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg\" (UID: \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 12:59:59.545001 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.542781 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35d138be-ff3b-43d9-ae6b-bae298aaec6b-istio-data\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg\" (UID: \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 12:59:59.545001 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.543089 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35d138be-ff3b-43d9-ae6b-bae298aaec6b-workload-certs\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg\" (UID: \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 12:59:59.545001 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.543576 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d138be-ff3b-43d9-ae6b-bae298aaec6b-workload-socket\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg\" (UID: \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 12:59:59.583457 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.566636 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35d138be-ff3b-43d9-ae6b-bae298aaec6b-istio-token\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg\" (UID: \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 12:59:59.583457 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.578943 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phqpc\" (UniqueName: \"kubernetes.io/projected/35d138be-ff3b-43d9-ae6b-bae298aaec6b-kube-api-access-phqpc\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg\" (UID: \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 12:59:59.606003 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.603836 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35d138be-ff3b-43d9-ae6b-bae298aaec6b-istio-podinfo\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg\" (UID: \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 12:59:59.606003 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.604107 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35d138be-ff3b-43d9-ae6b-bae298aaec6b-istio-envoy\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg\" (UID: \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 12:59:59.670509 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.670467 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:59.873002 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.872896 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 12:59:59.887051 ip-10-0-138-70 kubenswrapper[6841]: I1031 12:59:59.887027 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 13:00:00.200159 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:00.200125 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg"] Oct 31 13:00:00.203503 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:00:00.203478 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35d138be_ff3b_43d9_ae6b_bae298aaec6b.slice/crio-9b2475212121bce2a52d38e2a36eff760108531a8fc02d15790511bfe60c0d71 WatchSource:0}: Error finding container 9b2475212121bce2a52d38e2a36eff760108531a8fc02d15790511bfe60c0d71: Status 404 returned error can't find the container with id 9b2475212121bce2a52d38e2a36eff760108531a8fc02d15790511bfe60c0d71 Oct 31 13:00:00.206140 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:00.206109 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:00:00.206258 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:00.206175 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:00:00.206258 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:00.206226 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:00:00.214662 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:00.212704 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890860-2wfc7"] Oct 31 13:00:00.214662 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:00.213801 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad431a60-f82d-4b0b-928d-dec7ff77f067" containerName="istio-proxy" Oct 31 13:00:00.218724 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:00.218705 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890860-2wfc7" Oct 31 13:00:00.236604 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:00.236580 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890860-2wfc7"] Oct 31 13:00:00.250189 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:00.250161 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9277q\" (UniqueName: \"kubernetes.io/projected/f7617410-6818-4026-bec3-03a042bec96f-kube-api-access-9277q\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890860-2wfc7\" (UID: \"f7617410-6818-4026-bec3-03a042bec96f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890860-2wfc7" Oct 31 13:00:00.250421 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:00.250403 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7617410-6818-4026-bec3-03a042bec96f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890860-2wfc7\" (UID: \"f7617410-6818-4026-bec3-03a042bec96f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890860-2wfc7" Oct 31 13:00:00.289959 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:00.289931 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:00:00.339193 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:00.339142 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" event={"ID":"35d138be-ff3b-43d9-ae6b-bae298aaec6b","Type":"ContainerStarted","Data":"b37e75d6f82bc614102b7d3ac3f705c19eb68f5a289554cd5f76254127cfff9c"} Oct 31 13:00:00.339349 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:00.339215 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" event={"ID":"35d138be-ff3b-43d9-ae6b-bae298aaec6b","Type":"ContainerStarted","Data":"9b2475212121bce2a52d38e2a36eff760108531a8fc02d15790511bfe60c0d71"} Oct 31 13:00:00.351779 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:00.351749 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7617410-6818-4026-bec3-03a042bec96f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890860-2wfc7\" (UID: \"f7617410-6818-4026-bec3-03a042bec96f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890860-2wfc7" Oct 31 13:00:00.351902 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:00.351863 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9277q\" (UniqueName: \"kubernetes.io/projected/f7617410-6818-4026-bec3-03a042bec96f-kube-api-access-9277q\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890860-2wfc7\" (UID: \"f7617410-6818-4026-bec3-03a042bec96f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890860-2wfc7" Oct 31 13:00:00.352574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:00.352548 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7617410-6818-4026-bec3-03a042bec96f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890860-2wfc7\" (UID: \"f7617410-6818-4026-bec3-03a042bec96f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890860-2wfc7" Oct 31 13:00:00.365253 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:00.365205 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" podStartSLOduration=1.365189602 podStartE2EDuration="1.365189602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:00.362373264 +0000 UTC m=+1646.845496168" watchObservedRunningTime="2026-10-31 13:00:00.365189602 +0000 UTC m=+1646.848312505" Oct 31 13:00:00.365569 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:00.365546 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9277q\" (UniqueName: \"kubernetes.io/projected/f7617410-6818-4026-bec3-03a042bec96f-kube-api-access-9277q\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890860-2wfc7\" (UID: \"f7617410-6818-4026-bec3-03a042bec96f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890860-2wfc7" Oct 31 13:00:00.547077 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:00.547004 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890860-2wfc7" Oct 31 13:00:00.675178 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:00.675130 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:00.887202 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:00.887119 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 13:00:00.958102 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:00.958077 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890860-2wfc7"] Oct 31 13:00:00.960234 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:00:00.960204 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7617410_6818_4026_bec3_03a042bec96f.slice/crio-37cd7fa23a825022d870b86e1856c12b04e7ac99c49e7ea735f897a0ce5a0658 WatchSource:0}: Error finding container 37cd7fa23a825022d870b86e1856c12b04e7ac99c49e7ea735f897a0ce5a0658: Status 404 returned error can't find the container with id 37cd7fa23a825022d870b86e1856c12b04e7ac99c49e7ea735f897a0ce5a0658 Oct 31 13:00:01.347533 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:01.347499 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890860-2wfc7" event={"ID":"f7617410-6818-4026-bec3-03a042bec96f","Type":"ContainerStarted","Data":"37cd7fa23a825022d870b86e1856c12b04e7ac99c49e7ea735f897a0ce5a0658"} Oct 31 13:00:01.548384 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:01.548337 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:01.892397 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:01.892365 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 13:00:02.354253 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:02.354218 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 13:00:02.355196 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:02.355176 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 13:00:02.446007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:02.444972 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:02.824665 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:02.824615 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:03.047952 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:03.047903 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:03.221758 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:03.221723 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8"] Oct 31 13:00:03.227475 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:03.227450 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:03.246107 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:03.241946 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8"] Oct 31 13:00:03.246107 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:03.245146 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-bc9ae8-openshift-default-dockercfg-vqlv8\"" Oct 31 13:00:03.448008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:03.430598 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:03.448008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:03.432433 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40d2bf70-21e4-498f-ac76-23246675e3bf-istio-podinfo\") pod \"e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8\" (UID: \"40d2bf70-21e4-498f-ac76-23246675e3bf\") " pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:03.448008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:03.432529 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40d2bf70-21e4-498f-ac76-23246675e3bf-istio-envoy\") pod \"e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8\" (UID: \"40d2bf70-21e4-498f-ac76-23246675e3bf\") " pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:03.448008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:03.432561 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40d2bf70-21e4-498f-ac76-23246675e3bf-istio-data\") pod \"e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8\" (UID: \"40d2bf70-21e4-498f-ac76-23246675e3bf\") " pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:03.448008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:03.432593 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d2bf70-21e4-498f-ac76-23246675e3bf-workload-socket\") pod \"e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8\" (UID: \"40d2bf70-21e4-498f-ac76-23246675e3bf\") " pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:03.448008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:03.432649 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d2bf70-21e4-498f-ac76-23246675e3bf-credential-socket\") pod \"e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8\" (UID: \"40d2bf70-21e4-498f-ac76-23246675e3bf\") " pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:03.448008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:03.432730 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40d2bf70-21e4-498f-ac76-23246675e3bf-workload-certs\") pod \"e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8\" (UID: \"40d2bf70-21e4-498f-ac76-23246675e3bf\") " pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:03.448008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:03.432782 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40d2bf70-21e4-498f-ac76-23246675e3bf-istiod-ca-cert\") pod \"e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8\" (UID: \"40d2bf70-21e4-498f-ac76-23246675e3bf\") " pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:03.448008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:03.432864 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j97b\" (UniqueName: \"kubernetes.io/projected/40d2bf70-21e4-498f-ac76-23246675e3bf-kube-api-access-5j97b\") pod \"e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8\" (UID: \"40d2bf70-21e4-498f-ac76-23246675e3bf\") " pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:03.448008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:03.432970 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40d2bf70-21e4-498f-ac76-23246675e3bf-istio-token\") pod \"e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8\" (UID: \"40d2bf70-21e4-498f-ac76-23246675e3bf\") " pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:03.534700 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:03.534608 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40d2bf70-21e4-498f-ac76-23246675e3bf-workload-certs\") pod \"e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8\" (UID: \"40d2bf70-21e4-498f-ac76-23246675e3bf\") " pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:03.535505 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:03.535397 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40d2bf70-21e4-498f-ac76-23246675e3bf-workload-certs\") pod \"e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8\" (UID: \"40d2bf70-21e4-498f-ac76-23246675e3bf\") " pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:03.536502 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:03.536466 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40d2bf70-21e4-498f-ac76-23246675e3bf-istiod-ca-cert\") pod \"e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8\" (UID: \"40d2bf70-21e4-498f-ac76-23246675e3bf\") " pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:03.536728 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:03.535722 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40d2bf70-21e4-498f-ac76-23246675e3bf-istiod-ca-cert\") pod \"e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8\" (UID: \"40d2bf70-21e4-498f-ac76-23246675e3bf\") " pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:03.536941 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:03.536924 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j97b\" (UniqueName: \"kubernetes.io/projected/40d2bf70-21e4-498f-ac76-23246675e3bf-kube-api-access-5j97b\") pod \"e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8\" (UID: \"40d2bf70-21e4-498f-ac76-23246675e3bf\") " pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:03.537552 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:03.537521 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40d2bf70-21e4-498f-ac76-23246675e3bf-istio-token\") pod \"e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8\" (UID: \"40d2bf70-21e4-498f-ac76-23246675e3bf\") " pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:03.538105 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:03.538086 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40d2bf70-21e4-498f-ac76-23246675e3bf-istio-podinfo\") pod \"e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8\" (UID: \"40d2bf70-21e4-498f-ac76-23246675e3bf\") " pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:03.538770 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:03.538750 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40d2bf70-21e4-498f-ac76-23246675e3bf-istio-envoy\") pod \"e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8\" (UID: \"40d2bf70-21e4-498f-ac76-23246675e3bf\") " pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:03.539285 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:03.539267 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40d2bf70-21e4-498f-ac76-23246675e3bf-istio-data\") pod \"e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8\" (UID: \"40d2bf70-21e4-498f-ac76-23246675e3bf\") " pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:03.539930 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:03.539671 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40d2bf70-21e4-498f-ac76-23246675e3bf-istio-data\") pod \"e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8\" (UID: \"40d2bf70-21e4-498f-ac76-23246675e3bf\") " pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:03.540373 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:03.540327 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d2bf70-21e4-498f-ac76-23246675e3bf-workload-socket\") pod \"e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8\" (UID: \"40d2bf70-21e4-498f-ac76-23246675e3bf\") " pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:03.540487 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:03.539796 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d2bf70-21e4-498f-ac76-23246675e3bf-workload-socket\") pod \"e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8\" (UID: \"40d2bf70-21e4-498f-ac76-23246675e3bf\") " pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:03.540685 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:03.540664 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d2bf70-21e4-498f-ac76-23246675e3bf-credential-socket\") pod \"e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8\" (UID: \"40d2bf70-21e4-498f-ac76-23246675e3bf\") " pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:03.541340 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:03.541222 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d2bf70-21e4-498f-ac76-23246675e3bf-credential-socket\") pod \"e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8\" (UID: \"40d2bf70-21e4-498f-ac76-23246675e3bf\") " pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:03.541719 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:03.541682 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40d2bf70-21e4-498f-ac76-23246675e3bf-istio-envoy\") pod \"e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8\" (UID: \"40d2bf70-21e4-498f-ac76-23246675e3bf\") " pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:03.602248 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:03.601786 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j97b\" (UniqueName: \"kubernetes.io/projected/40d2bf70-21e4-498f-ac76-23246675e3bf-kube-api-access-5j97b\") pod \"e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8\" (UID: \"40d2bf70-21e4-498f-ac76-23246675e3bf\") " pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:03.604056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:03.603458 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40d2bf70-21e4-498f-ac76-23246675e3bf-istio-token\") pod \"e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8\" (UID: \"40d2bf70-21e4-498f-ac76-23246675e3bf\") " pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:03.615093 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:03.615057 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40d2bf70-21e4-498f-ac76-23246675e3bf-istio-podinfo\") pod \"e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8\" (UID: \"40d2bf70-21e4-498f-ac76-23246675e3bf\") " pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:03.649036 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:03.648974 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:03.852754 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:03.852670 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:03.966593 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:03.966557 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:04.166629 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:04.166598 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8"] Oct 31 13:00:04.167116 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:00:04.166850 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d2bf70_21e4_498f_ac76_23246675e3bf.slice/crio-f4261dbb1e9b169705265ccb74ee4d61f9e677a6a5d438a39941e6dd93882484 WatchSource:0}: Error finding container f4261dbb1e9b169705265ccb74ee4d61f9e677a6a5d438a39941e6dd93882484: Status 404 returned error can't find the container with id f4261dbb1e9b169705265ccb74ee4d61f9e677a6a5d438a39941e6dd93882484 Oct 31 13:00:04.170033 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:04.169977 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:00:04.170142 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:04.170071 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:00:04.170142 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:04.170111 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:00:04.334496 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:04.334443 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3128e5-85d8848459-shp4r"] Oct 31 13:00:04.381280 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:04.381210 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:04.384412 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:04.384389 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3128e5-85d8848459-shp4r"] Oct 31 13:00:04.384676 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:04.384658 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3128e5-85d8848459-shp4r" Oct 31 13:00:04.396727 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:04.391925 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-3128e5-dockercfg-r9ctc\"" Oct 31 13:00:04.429339 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:04.429276 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-3128e5\"" Oct 31 13:00:04.520011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:04.510326 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" event={"ID":"40d2bf70-21e4-498f-ac76-23246675e3bf","Type":"ContainerStarted","Data":"1a09d688c41bd1d2cbce3a2c6f137d38aa0ff6b46ca662008a8325740ec7dacc"} Oct 31 13:00:04.520011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:04.510381 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" event={"ID":"40d2bf70-21e4-498f-ac76-23246675e3bf","Type":"ContainerStarted","Data":"f4261dbb1e9b169705265ccb74ee4d61f9e677a6a5d438a39941e6dd93882484"} Oct 31 13:00:04.532004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:04.529014 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6gsv\" (UniqueName: \"kubernetes.io/projected/26e9a61c-78ec-459d-835c-f384d3a4c4bc-kube-api-access-v6gsv\") pod \"payload-processing-e2e-api-a-3128e5-85d8848459-shp4r\" (UID: \"26e9a61c-78ec-459d-835c-f384d3a4c4bc\") " pod="openshift-ingress/payload-processing-e2e-api-a-3128e5-85d8848459-shp4r" Oct 31 13:00:04.532004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:04.529144 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26e9a61c-78ec-459d-835c-f384d3a4c4bc-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3128e5-85d8848459-shp4r\" (UID: \"26e9a61c-78ec-459d-835c-f384d3a4c4bc\") " pod="openshift-ingress/payload-processing-e2e-api-a-3128e5-85d8848459-shp4r" Oct 31 13:00:04.609002 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:04.608290 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:04.635265 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:04.635221 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6gsv\" (UniqueName: \"kubernetes.io/projected/26e9a61c-78ec-459d-835c-f384d3a4c4bc-kube-api-access-v6gsv\") pod \"payload-processing-e2e-api-a-3128e5-85d8848459-shp4r\" (UID: \"26e9a61c-78ec-459d-835c-f384d3a4c4bc\") " pod="openshift-ingress/payload-processing-e2e-api-a-3128e5-85d8848459-shp4r" Oct 31 13:00:04.635659 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:04.635634 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26e9a61c-78ec-459d-835c-f384d3a4c4bc-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3128e5-85d8848459-shp4r\" (UID: \"26e9a61c-78ec-459d-835c-f384d3a4c4bc\") " pod="openshift-ingress/payload-processing-e2e-api-a-3128e5-85d8848459-shp4r" Oct 31 13:00:04.649290 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:04.649255 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26e9a61c-78ec-459d-835c-f384d3a4c4bc-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3128e5-85d8848459-shp4r\" (UID: \"26e9a61c-78ec-459d-835c-f384d3a4c4bc\") " pod="openshift-ingress/payload-processing-e2e-api-a-3128e5-85d8848459-shp4r" Oct 31 13:00:04.675849 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:04.675788 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6gsv\" (UniqueName: \"kubernetes.io/projected/26e9a61c-78ec-459d-835c-f384d3a4c4bc-kube-api-access-v6gsv\") pod \"payload-processing-e2e-api-a-3128e5-85d8848459-shp4r\" (UID: \"26e9a61c-78ec-459d-835c-f384d3a4c4bc\") " pod="openshift-ingress/payload-processing-e2e-api-a-3128e5-85d8848459-shp4r" Oct 31 13:00:04.708878 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:04.708842 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3128e5-85d8848459-shp4r" Oct 31 13:00:04.857194 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:04.857147 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:05.168708 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.168654 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" podStartSLOduration=2.16863995 podStartE2EDuration="2.16863995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:04.581955569 +0000 UTC m=+1651.065078487" watchObservedRunningTime="2026-10-31 13:00:05.16863995 +0000 UTC m=+1651.651762846" Oct 31 13:00:05.175757 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.175728 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3128e5-85d8848459-shp4r"] Oct 31 13:00:05.409167 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.407893 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2"] Oct 31 13:00:05.414873 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.414846 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:05.426666 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.426601 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2"] Oct 31 13:00:05.427459 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.427431 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-ccf963-openshift-default-dockercfg-xmvz7\"" Oct 31 13:00:05.472836 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.472793 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebcb215e-7f84-4653-96e6-3a13c449be73-credential-socket\") pod \"e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2\" (UID: \"ebcb215e-7f84-4653-96e6-3a13c449be73\") " pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:05.473059 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.472843 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebcb215e-7f84-4653-96e6-3a13c449be73-istio-envoy\") pod \"e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2\" (UID: \"ebcb215e-7f84-4653-96e6-3a13c449be73\") " pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:05.473059 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.473007 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkh5r\" (UniqueName: \"kubernetes.io/projected/ebcb215e-7f84-4653-96e6-3a13c449be73-kube-api-access-pkh5r\") pod \"e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2\" (UID: \"ebcb215e-7f84-4653-96e6-3a13c449be73\") " pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:05.473198 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.473145 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebcb215e-7f84-4653-96e6-3a13c449be73-workload-socket\") pod \"e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2\" (UID: \"ebcb215e-7f84-4653-96e6-3a13c449be73\") " pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:05.473264 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.473200 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebcb215e-7f84-4653-96e6-3a13c449be73-istio-token\") pod \"e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2\" (UID: \"ebcb215e-7f84-4653-96e6-3a13c449be73\") " pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:05.473319 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.473292 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebcb215e-7f84-4653-96e6-3a13c449be73-istio-data\") pod \"e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2\" (UID: \"ebcb215e-7f84-4653-96e6-3a13c449be73\") " pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:05.473377 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.473321 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebcb215e-7f84-4653-96e6-3a13c449be73-istio-podinfo\") pod \"e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2\" (UID: \"ebcb215e-7f84-4653-96e6-3a13c449be73\") " pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:05.473539 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.473433 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebcb215e-7f84-4653-96e6-3a13c449be73-workload-certs\") pod \"e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2\" (UID: \"ebcb215e-7f84-4653-96e6-3a13c449be73\") " pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:05.473539 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.473497 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebcb215e-7f84-4653-96e6-3a13c449be73-istiod-ca-cert\") pod \"e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2\" (UID: \"ebcb215e-7f84-4653-96e6-3a13c449be73\") " pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:05.627013 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.603410 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebcb215e-7f84-4653-96e6-3a13c449be73-credential-socket\") pod \"e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2\" (UID: \"ebcb215e-7f84-4653-96e6-3a13c449be73\") " pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:05.627013 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.603463 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebcb215e-7f84-4653-96e6-3a13c449be73-istio-envoy\") pod \"e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2\" (UID: \"ebcb215e-7f84-4653-96e6-3a13c449be73\") " pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:05.627013 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.603564 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkh5r\" (UniqueName: \"kubernetes.io/projected/ebcb215e-7f84-4653-96e6-3a13c449be73-kube-api-access-pkh5r\") pod \"e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2\" (UID: \"ebcb215e-7f84-4653-96e6-3a13c449be73\") " pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:05.627013 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.603682 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebcb215e-7f84-4653-96e6-3a13c449be73-workload-socket\") pod \"e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2\" (UID: \"ebcb215e-7f84-4653-96e6-3a13c449be73\") " pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:05.627013 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.603735 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebcb215e-7f84-4653-96e6-3a13c449be73-istio-token\") pod \"e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2\" (UID: \"ebcb215e-7f84-4653-96e6-3a13c449be73\") " pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:05.627013 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.603842 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebcb215e-7f84-4653-96e6-3a13c449be73-istio-data\") pod \"e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2\" (UID: \"ebcb215e-7f84-4653-96e6-3a13c449be73\") " pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:05.627013 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.603869 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebcb215e-7f84-4653-96e6-3a13c449be73-istio-podinfo\") pod \"e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2\" (UID: \"ebcb215e-7f84-4653-96e6-3a13c449be73\") " pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:05.627013 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.603950 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebcb215e-7f84-4653-96e6-3a13c449be73-workload-certs\") pod \"e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2\" (UID: \"ebcb215e-7f84-4653-96e6-3a13c449be73\") " pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:05.627013 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.604009 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebcb215e-7f84-4653-96e6-3a13c449be73-istiod-ca-cert\") pod \"e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2\" (UID: \"ebcb215e-7f84-4653-96e6-3a13c449be73\") " pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:05.627013 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.619104 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebcb215e-7f84-4653-96e6-3a13c449be73-workload-socket\") pod \"e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2\" (UID: \"ebcb215e-7f84-4653-96e6-3a13c449be73\") " pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:05.627013 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.619409 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebcb215e-7f84-4653-96e6-3a13c449be73-credential-socket\") pod \"e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2\" (UID: \"ebcb215e-7f84-4653-96e6-3a13c449be73\") " pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:05.659036 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.604973 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebcb215e-7f84-4653-96e6-3a13c449be73-istiod-ca-cert\") pod \"e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2\" (UID: \"ebcb215e-7f84-4653-96e6-3a13c449be73\") " pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:05.681877 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.661369 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebcb215e-7f84-4653-96e6-3a13c449be73-istio-envoy\") pod \"e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2\" (UID: \"ebcb215e-7f84-4653-96e6-3a13c449be73\") " pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:05.681877 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.666618 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebcb215e-7f84-4653-96e6-3a13c449be73-istio-podinfo\") pod \"e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2\" (UID: \"ebcb215e-7f84-4653-96e6-3a13c449be73\") " pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:05.681877 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.666952 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebcb215e-7f84-4653-96e6-3a13c449be73-istio-data\") pod \"e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2\" (UID: \"ebcb215e-7f84-4653-96e6-3a13c449be73\") " pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:05.720041 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.691286 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebcb215e-7f84-4653-96e6-3a13c449be73-workload-certs\") pod \"e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2\" (UID: \"ebcb215e-7f84-4653-96e6-3a13c449be73\") " pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:05.720041 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.698319 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebcb215e-7f84-4653-96e6-3a13c449be73-istio-token\") pod \"e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2\" (UID: \"ebcb215e-7f84-4653-96e6-3a13c449be73\") " pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:05.746001 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.745666 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkh5r\" (UniqueName: \"kubernetes.io/projected/ebcb215e-7f84-4653-96e6-3a13c449be73-kube-api-access-pkh5r\") pod \"e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2\" (UID: \"ebcb215e-7f84-4653-96e6-3a13c449be73\") " pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:05.759007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.756928 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:05.762914 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.762835 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-3128e5-85d8848459-shp4r" podStartSLOduration=1.762801419 podStartE2EDuration="1.762801419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:05.762343489 +0000 UTC m=+1652.245466390" watchObservedRunningTime="2026-10-31 13:00:05.762801419 +0000 UTC m=+1652.245924320" Oct 31 13:00:05.778761 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.778060 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-3128e5-85d8848459-shp4r" Oct 31 13:00:05.778761 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.778128 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3128e5-85d8848459-shp4r" event={"ID":"26e9a61c-78ec-459d-835c-f384d3a4c4bc","Type":"ContainerStarted","Data":"d7d20572b961d65aca53045eada19cc5117e47c6c52d092ef6d1338f92ab4c97"} Oct 31 13:00:05.778761 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.778157 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3128e5-85d8848459-shp4r" event={"ID":"26e9a61c-78ec-459d-835c-f384d3a4c4bc","Type":"ContainerStarted","Data":"7db5bb2a65c8c62961389bef95517d67ec0f7c2f29c8193d43454b6ec08db6ee"} Oct 31 13:00:05.859750 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:05.859612 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:06.134498 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:06.134398 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2"] Oct 31 13:00:06.192696 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:06.192650 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:06.195793 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:06.195760 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:00:06.195922 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:06.195848 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:00:06.195922 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:06.195893 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:00:06.395387 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:06.395290 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:06.713962 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:06.713929 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890860-2wfc7" event={"ID":"f7617410-6818-4026-bec3-03a042bec96f","Type":"ContainerStarted","Data":"6a90fe91d78f167684485c975f144297a3fa4b365d2b17d34b0892c81599cc22"} Oct 31 13:00:06.715711 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:06.715673 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" event={"ID":"ebcb215e-7f84-4653-96e6-3a13c449be73","Type":"ContainerStarted","Data":"f86a3b89d8a0a8abe80b4cab3de2a83e592196d5d72d48535306d7013fb4da73"} Oct 31 13:00:06.715859 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:06.715728 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" event={"ID":"ebcb215e-7f84-4653-96e6-3a13c449be73","Type":"ContainerStarted","Data":"234dd507286c3ed7eb92d571434469a7197c7d9258885a6b8925f7fc8caa63ee"} Oct 31 13:00:06.716156 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:06.716133 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:06.716971 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:06.716953 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:06.742903 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:06.742860 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890860-2wfc7" podStartSLOduration=1.4466626200000001 podStartE2EDuration="6.742847239s" totalImagesPullingTime="5.296184619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:06.741339883 +0000 UTC m=+1653.224462786" watchObservedRunningTime="2026-10-31 13:00:06.742847239 +0000 UTC m=+1653.225970140" Oct 31 13:00:06.757874 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:06.757849 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:06.767617 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:06.767578 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" podStartSLOduration=1.7675657949999999 podStartE2EDuration="1.767565795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:06.766860324 +0000 UTC m=+1653.249983226" watchObservedRunningTime="2026-10-31 13:00:06.767565795 +0000 UTC m=+1653.250688696" Oct 31 13:00:07.164069 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:07.160876 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:07.429929 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:07.429847 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:07.725028 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:07.724974 6841 generic.go:356] "Generic (PLEG): container finished" podID="f7617410-6818-4026-bec3-03a042bec96f" containerID="6a90fe91d78f167684485c975f144297a3fa4b365d2b17d34b0892c81599cc22" exitCode=60 Oct 31 13:00:07.725028 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:07.725016 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890860-2wfc7" event={"ID":"f7617410-6818-4026-bec3-03a042bec96f","Type":"ContainerDied","Data":"6a90fe91d78f167684485c975f144297a3fa4b365d2b17d34b0892c81599cc22"} Oct 31 13:00:07.725400 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:07.725378 6841 scope.go:122] "RemoveContainer" containerID="6a90fe91d78f167684485c975f144297a3fa4b365d2b17d34b0892c81599cc22" Oct 31 13:00:07.764804 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:07.764777 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:08.322129 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:08.321947 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:08.543950 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:08.543914 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:08.773353 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:08.773313 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:08.782592 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:08.782546 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890860-2wfc7" event={"ID":"f7617410-6818-4026-bec3-03a042bec96f","Type":"ContainerStarted","Data":"73b7e7dbef2672b6df61aee595bff99691e9086cd5d484e29738ff0bc84b6031"} Oct 31 13:00:08.782833 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:08.782817 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:08.802420 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:08.802387 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:09.712144 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:09.689101 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:09.915005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:09.913289 6841 generic.go:356] "Generic (PLEG): container finished" podID="f7617410-6818-4026-bec3-03a042bec96f" containerID="73b7e7dbef2672b6df61aee595bff99691e9086cd5d484e29738ff0bc84b6031" exitCode=60 Oct 31 13:00:09.915005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:09.914879 6841 scope.go:122] "RemoveContainer" containerID="73b7e7dbef2672b6df61aee595bff99691e9086cd5d484e29738ff0bc84b6031" Oct 31 13:00:09.915294 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:00:09.915258 6841 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890860-2wfc7_odh-ai-gateway-infra(f7617410-6818-4026-bec3-03a042bec96f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890860-2wfc7" podUID="f7617410-6818-4026-bec3-03a042bec96f" Oct 31 13:00:09.915489 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:09.915467 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890860-2wfc7" event={"ID":"f7617410-6818-4026-bec3-03a042bec96f","Type":"ContainerDied","Data":"73b7e7dbef2672b6df61aee595bff99691e9086cd5d484e29738ff0bc84b6031"} Oct 31 13:00:09.915639 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:09.915625 6841 scope.go:122] "RemoveContainer" containerID="6a90fe91d78f167684485c975f144297a3fa4b365d2b17d34b0892c81599cc22" Oct 31 13:00:09.923491 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:09.923450 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:11.167005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:11.166406 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:11.209722 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:11.202462 6841 scope.go:122] "RemoveContainer" containerID="73b7e7dbef2672b6df61aee595bff99691e9086cd5d484e29738ff0bc84b6031" Oct 31 13:00:11.219586 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:00:11.219539 6841 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890860-2wfc7_odh-ai-gateway-infra(f7617410-6818-4026-bec3-03a042bec96f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890860-2wfc7" podUID="f7617410-6818-4026-bec3-03a042bec96f" Oct 31 13:00:11.375008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:11.369964 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:11.695010 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:11.694390 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:11.893008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:11.890028 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-6b55c6c94c-44jfs"] Oct 31 13:00:12.098955 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:12.098871 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-6b55c6c94c-44jfs" Oct 31 13:00:12.154035 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:12.146508 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2kdq\" (UniqueName: \"kubernetes.io/projected/bf16598d-b76a-4dda-aba1-b801582fad71-kube-api-access-l2kdq\") pod \"payload-pre-processing-e2e-ipp-a-ccf963-6b55c6c94c-44jfs\" (UID: \"bf16598d-b76a-4dda-aba1-b801582fad71\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-6b55c6c94c-44jfs" Oct 31 13:00:12.154703 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:12.154663 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf16598d-b76a-4dda-aba1-b801582fad71-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ccf963-6b55c6c94c-44jfs\" (UID: \"bf16598d-b76a-4dda-aba1-b801582fad71\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-6b55c6c94c-44jfs" Oct 31 13:00:12.204802 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:12.193960 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ccf963\"" Oct 31 13:00:12.247030 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:12.220055 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ccf963-dockercfg-mmvcx\"" Oct 31 13:00:12.365061 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:12.354516 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:12.400063 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:12.367770 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fk2tx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:00:12.400063 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:12.368933 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" podUID="aa7e0a48-9dc5-4735-b1ae-cafd28ff982f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:00:12.559875 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:12.559839 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-6b55c6c94c-44jfs"] Oct 31 13:00:12.585613 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:12.583899 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2kdq\" (UniqueName: \"kubernetes.io/projected/bf16598d-b76a-4dda-aba1-b801582fad71-kube-api-access-l2kdq\") pod \"payload-pre-processing-e2e-ipp-a-ccf963-6b55c6c94c-44jfs\" (UID: \"bf16598d-b76a-4dda-aba1-b801582fad71\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-6b55c6c94c-44jfs" Oct 31 13:00:12.585613 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:12.584106 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf16598d-b76a-4dda-aba1-b801582fad71-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ccf963-6b55c6c94c-44jfs\" (UID: \"bf16598d-b76a-4dda-aba1-b801582fad71\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-6b55c6c94c-44jfs" Oct 31 13:00:12.589018 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:12.588848 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf16598d-b76a-4dda-aba1-b801582fad71-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ccf963-6b55c6c94c-44jfs\" (UID: \"bf16598d-b76a-4dda-aba1-b801582fad71\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-6b55c6c94c-44jfs" Oct 31 13:00:12.638591 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:12.638497 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:12.671044 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:12.670968 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2kdq\" (UniqueName: \"kubernetes.io/projected/bf16598d-b76a-4dda-aba1-b801582fad71-kube-api-access-l2kdq\") pod \"payload-pre-processing-e2e-ipp-a-ccf963-6b55c6c94c-44jfs\" (UID: \"bf16598d-b76a-4dda-aba1-b801582fad71\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-6b55c6c94c-44jfs" Oct 31 13:00:12.740510 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:12.740470 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ccf963-8645c8f5f7-x69xk"] Oct 31 13:00:12.751009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:12.747052 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-8645c8f5f7-x69xk" Oct 31 13:00:12.793014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:12.788540 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b40a85-2601-4987-8b1e-b4806dcada3a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ccf963-8645c8f5f7-x69xk\" (UID: \"f7b40a85-2601-4987-8b1e-b4806dcada3a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-8645c8f5f7-x69xk" Oct 31 13:00:12.793014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:12.788813 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj555\" (UniqueName: \"kubernetes.io/projected/f7b40a85-2601-4987-8b1e-b4806dcada3a-kube-api-access-vj555\") pod \"payload-processing-e2e-ipp-a-ccf963-8645c8f5f7-x69xk\" (UID: \"f7b40a85-2601-4987-8b1e-b4806dcada3a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-8645c8f5f7-x69xk" Oct 31 13:00:12.837782 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:12.837749 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ccf963-8645c8f5f7-x69xk"] Oct 31 13:00:12.890059 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:12.889954 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj555\" (UniqueName: \"kubernetes.io/projected/f7b40a85-2601-4987-8b1e-b4806dcada3a-kube-api-access-vj555\") pod \"payload-processing-e2e-ipp-a-ccf963-8645c8f5f7-x69xk\" (UID: \"f7b40a85-2601-4987-8b1e-b4806dcada3a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-8645c8f5f7-x69xk" Oct 31 13:00:12.890512 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:12.890485 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b40a85-2601-4987-8b1e-b4806dcada3a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ccf963-8645c8f5f7-x69xk\" (UID: \"f7b40a85-2601-4987-8b1e-b4806dcada3a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-8645c8f5f7-x69xk" Oct 31 13:00:12.891607 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:12.891582 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b40a85-2601-4987-8b1e-b4806dcada3a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ccf963-8645c8f5f7-x69xk\" (UID: \"f7b40a85-2601-4987-8b1e-b4806dcada3a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-8645c8f5f7-x69xk" Oct 31 13:00:12.892516 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:12.892302 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-6b55c6c94c-44jfs" Oct 31 13:00:12.908335 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:12.908303 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-x5snf"] Oct 31 13:00:12.940005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:12.932168 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-l6mb5"] Oct 31 13:00:12.986002 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:12.970813 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj555\" (UniqueName: \"kubernetes.io/projected/f7b40a85-2601-4987-8b1e-b4806dcada3a-kube-api-access-vj555\") pod \"payload-processing-e2e-ipp-a-ccf963-8645c8f5f7-x69xk\" (UID: \"f7b40a85-2601-4987-8b1e-b4806dcada3a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-8645c8f5f7-x69xk" Oct 31 13:00:13.113618 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:13.113584 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:13.151684 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:13.151606 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-8645c8f5f7-x69xk" Oct 31 13:00:13.348005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:13.342758 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:13.362128 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:13.362028 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-6b55c6c94c-44jfs"] Oct 31 13:00:13.573033 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:13.564280 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:13.679448 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:13.679417 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-6b55c6c94c-44jfs" event={"ID":"bf16598d-b76a-4dda-aba1-b801582fad71","Type":"ContainerStarted","Data":"979dbec31e6c3e3640b0ef6767bcf96acbce18478faae9a7e2edce6a5d846cdc"} Oct 31 13:00:13.844337 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:13.844305 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:13.861033 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:13.861005 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ccf963-8645c8f5f7-x69xk"] Oct 31 13:00:13.865800 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:00:13.865763 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b40a85_2601_4987_8b1e_b4806dcada3a.slice/crio-3cd758823b7c2ab8f8daf048b58a26f0ddadb0e30cc87b05ca13051c275d2bff WatchSource:0}: Error finding container 3cd758823b7c2ab8f8daf048b58a26f0ddadb0e30cc87b05ca13051c275d2bff: Status 404 returned error can't find the container with id 3cd758823b7c2ab8f8daf048b58a26f0ddadb0e30cc87b05ca13051c275d2bff Oct 31 13:00:14.428078 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:14.427103 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:14.688745 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:14.688662 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-6b55c6c94c-44jfs" event={"ID":"bf16598d-b76a-4dda-aba1-b801582fad71","Type":"ContainerStarted","Data":"e643d407546465bed47e05556bfd61678943f64bb5b28d315a8431a8c985cbf4"} Oct 31 13:00:14.688745 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:14.688731 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-6b55c6c94c-44jfs" Oct 31 13:00:14.690375 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:14.690346 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-8645c8f5f7-x69xk" event={"ID":"f7b40a85-2601-4987-8b1e-b4806dcada3a","Type":"ContainerStarted","Data":"bdde9b3c447a3acdf5a85ae890319c3bed374f870725712b45af04b9a35c8c07"} Oct 31 13:00:14.690535 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:14.690395 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-8645c8f5f7-x69xk" event={"ID":"f7b40a85-2601-4987-8b1e-b4806dcada3a","Type":"ContainerStarted","Data":"3cd758823b7c2ab8f8daf048b58a26f0ddadb0e30cc87b05ca13051c275d2bff"} Oct 31 13:00:14.690535 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:14.690426 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-8645c8f5f7-x69xk" Oct 31 13:00:14.709781 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:14.709738 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-6b55c6c94c-44jfs" podStartSLOduration=3.709724182 podStartE2EDuration="3.709724182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:14.707367063 +0000 UTC m=+1661.190489960" watchObservedRunningTime="2026-10-31 13:00:14.709724182 +0000 UTC m=+1661.192847083" Oct 31 13:00:14.729756 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:14.729711 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-8645c8f5f7-x69xk" podStartSLOduration=3.72969712 podStartE2EDuration="3.72969712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:14.727345679 +0000 UTC m=+1661.210468581" watchObservedRunningTime="2026-10-31 13:00:14.72969712 +0000 UTC m=+1661.212820043" Oct 31 13:00:15.129865 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:15.129832 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c"] Oct 31 13:00:15.134914 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:15.134885 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c" Oct 31 13:00:15.147880 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:15.147848 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c"] Oct 31 13:00:15.150696 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:15.150671 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-bc9ae8-serving-cert\"" Oct 31 13:00:15.216429 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:15.216400 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8afce0f-eb7a-45c5-b837-2024fd7c563a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c\" (UID: \"a8afce0f-eb7a-45c5-b837-2024fd7c563a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c" Oct 31 13:00:15.216617 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:15.216442 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8afce0f-eb7a-45c5-b837-2024fd7c563a-metrics-certs\") pod \"maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c\" (UID: \"a8afce0f-eb7a-45c5-b837-2024fd7c563a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c" Oct 31 13:00:15.216617 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:15.216471 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfcdc\" (UniqueName: \"kubernetes.io/projected/a8afce0f-eb7a-45c5-b837-2024fd7c563a-kube-api-access-zfcdc\") pod \"maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c\" (UID: \"a8afce0f-eb7a-45c5-b837-2024fd7c563a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c" Oct 31 13:00:15.216755 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:15.216647 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8afce0f-eb7a-45c5-b837-2024fd7c563a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c\" (UID: \"a8afce0f-eb7a-45c5-b837-2024fd7c563a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c" Oct 31 13:00:15.320005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:15.298412 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:15.327013 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:15.323146 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8afce0f-eb7a-45c5-b837-2024fd7c563a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c\" (UID: \"a8afce0f-eb7a-45c5-b837-2024fd7c563a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c" Oct 31 13:00:15.327013 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:15.323206 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8afce0f-eb7a-45c5-b837-2024fd7c563a-metrics-certs\") pod \"maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c\" (UID: \"a8afce0f-eb7a-45c5-b837-2024fd7c563a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c" Oct 31 13:00:15.327013 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:15.323239 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfcdc\" (UniqueName: \"kubernetes.io/projected/a8afce0f-eb7a-45c5-b837-2024fd7c563a-kube-api-access-zfcdc\") pod \"maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c\" (UID: \"a8afce0f-eb7a-45c5-b837-2024fd7c563a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c" Oct 31 13:00:15.327013 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:15.323383 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8afce0f-eb7a-45c5-b837-2024fd7c563a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c\" (UID: \"a8afce0f-eb7a-45c5-b837-2024fd7c563a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c" Oct 31 13:00:15.333911 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:15.333874 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8afce0f-eb7a-45c5-b837-2024fd7c563a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c\" (UID: \"a8afce0f-eb7a-45c5-b837-2024fd7c563a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c" Oct 31 13:00:15.390232 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:15.390145 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8afce0f-eb7a-45c5-b837-2024fd7c563a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c\" (UID: \"a8afce0f-eb7a-45c5-b837-2024fd7c563a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c" Oct 31 13:00:15.391573 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:15.391538 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8afce0f-eb7a-45c5-b837-2024fd7c563a-metrics-certs\") pod \"maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c\" (UID: \"a8afce0f-eb7a-45c5-b837-2024fd7c563a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c" Oct 31 13:00:15.393107 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:15.393081 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfcdc\" (UniqueName: \"kubernetes.io/projected/a8afce0f-eb7a-45c5-b837-2024fd7c563a-kube-api-access-zfcdc\") pod \"maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c\" (UID: \"a8afce0f-eb7a-45c5-b837-2024fd7c563a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c" Oct 31 13:00:15.457645 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:15.457618 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c" Oct 31 13:00:15.561425 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:15.561381 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:15.742821 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:15.742796 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c"] Oct 31 13:00:15.744112 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:00:15.744086 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8afce0f_eb7a_45c5_b837_2024fd7c563a.slice/crio-e2abec19384b30f4f10517110aba0812eccd17f2c496b1d6342a8aa4ae9a271e WatchSource:0}: Error finding container e2abec19384b30f4f10517110aba0812eccd17f2c496b1d6342a8aa4ae9a271e: Status 404 returned error can't find the container with id e2abec19384b30f4f10517110aba0812eccd17f2c496b1d6342a8aa4ae9a271e Oct 31 13:00:15.968554 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:15.968490 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-z959n"] Oct 31 13:00:15.969112 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:15.969018 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-z959n" podUID="ceab3693-ea13-4097-a44c-37561dd698d4" containerName="maas-api" containerID="cri-o://a1fd1bb3bc4cd3d712566013da5c3088749c965f1639de82a98f58a2ea1ea3de" gracePeriod=30 Oct 31 13:00:16.193635 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:16.193543 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:16.600014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:16.598850 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:16.729501 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:16.729474 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-3128e5-85d8848459-shp4r" Oct 31 13:00:16.732706 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:16.732680 6841 generic.go:356] "Generic (PLEG): container finished" podID="ceab3693-ea13-4097-a44c-37561dd698d4" containerID="a1fd1bb3bc4cd3d712566013da5c3088749c965f1639de82a98f58a2ea1ea3de" exitCode=0 Oct 31 13:00:16.732807 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:16.732789 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-z959n" event={"ID":"ceab3693-ea13-4097-a44c-37561dd698d4","Type":"ContainerDied","Data":"a1fd1bb3bc4cd3d712566013da5c3088749c965f1639de82a98f58a2ea1ea3de"} Oct 31 13:00:16.734380 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:16.734357 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c" event={"ID":"a8afce0f-eb7a-45c5-b837-2024fd7c563a","Type":"ContainerStarted","Data":"e2abec19384b30f4f10517110aba0812eccd17f2c496b1d6342a8aa4ae9a271e"} Oct 31 13:00:16.883517 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:16.883436 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-z959n" Oct 31 13:00:16.915970 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:16.915934 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ceab3693-ea13-4097-a44c-37561dd698d4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ceab3693-ea13-4097-a44c-37561dd698d4-maas-api-tls\") pod \"ceab3693-ea13-4097-a44c-37561dd698d4\" (UID: \"ceab3693-ea13-4097-a44c-37561dd698d4\") " Oct 31 13:00:16.916345 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:16.916319 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceab3693-ea13-4097-a44c-37561dd698d4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceab3693-ea13-4097-a44c-37561dd698d4-openshift-service-ca\") pod \"ceab3693-ea13-4097-a44c-37561dd698d4\" (UID: \"ceab3693-ea13-4097-a44c-37561dd698d4\") " Oct 31 13:00:16.916698 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:16.916656 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceab3693-ea13-4097-a44c-37561dd698d4-openshift-service-ca" pod "ceab3693-ea13-4097-a44c-37561dd698d4" (UID: "ceab3693-ea13-4097-a44c-37561dd698d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:00:16.917208 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:16.917179 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceab3693-ea13-4097-a44c-37561dd698d4-kube-api-access-xrf9v\" (UniqueName: \"kubernetes.io/projected/ceab3693-ea13-4097-a44c-37561dd698d4-kube-api-access-xrf9v\") pod \"ceab3693-ea13-4097-a44c-37561dd698d4\" (UID: \"ceab3693-ea13-4097-a44c-37561dd698d4\") " Oct 31 13:00:16.917352 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:16.917327 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ceab3693-ea13-4097-a44c-37561dd698d4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceab3693-ea13-4097-a44c-37561dd698d4-metrics-certs\") pod \"ceab3693-ea13-4097-a44c-37561dd698d4\" (UID: \"ceab3693-ea13-4097-a44c-37561dd698d4\") " Oct 31 13:00:16.918141 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:16.918116 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceab3693-ea13-4097-a44c-37561dd698d4-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:16.919742 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:16.919719 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceab3693-ea13-4097-a44c-37561dd698d4-kube-api-access-xrf9v" pod "ceab3693-ea13-4097-a44c-37561dd698d4" (UID: "ceab3693-ea13-4097-a44c-37561dd698d4"). InnerVolumeSpecName "kube-api-access-xrf9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:16.919965 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:16.919946 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceab3693-ea13-4097-a44c-37561dd698d4-maas-api-tls" pod "ceab3693-ea13-4097-a44c-37561dd698d4" (UID: "ceab3693-ea13-4097-a44c-37561dd698d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:00:16.921003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:16.920961 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceab3693-ea13-4097-a44c-37561dd698d4-metrics-certs" pod "ceab3693-ea13-4097-a44c-37561dd698d4" (UID: "ceab3693-ea13-4097-a44c-37561dd698d4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:00:17.019694 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:17.019661 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceab3693-ea13-4097-a44c-37561dd698d4-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:17.019912 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:17.019894 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ceab3693-ea13-4097-a44c-37561dd698d4-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:17.020041 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:17.020022 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrf9v\" (UniqueName: \"kubernetes.io/projected/ceab3693-ea13-4097-a44c-37561dd698d4-kube-api-access-xrf9v\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:17.241863 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:17.241831 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z"] Oct 31 13:00:17.243010 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:17.242970 6841 cpu_manager.go:414] "containerMap: removing container" podUID="ceab3693-ea13-4097-a44c-37561dd698d4" containerName="maas-api" Oct 31 13:00:17.243010 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:17.243007 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceab3693-ea13-4097-a44c-37561dd698d4" containerName="maas-api" Oct 31 13:00:17.243287 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:17.243260 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceab3693-ea13-4097-a44c-37561dd698d4" containerName="maas-api" Oct 31 13:00:17.247158 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:17.247136 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z" Oct 31 13:00:17.260840 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:17.260819 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-ccf963-serving-cert\"" Oct 31 13:00:17.267076 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:17.267045 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z"] Oct 31 13:00:17.324532 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:17.324504 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlbpf\" (UniqueName: \"kubernetes.io/projected/4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585-kube-api-access-wlbpf\") pod \"maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z\" (UID: \"4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z" Oct 31 13:00:17.324694 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:17.324674 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585-metrics-certs\") pod \"maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z\" (UID: \"4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z" Oct 31 13:00:17.324828 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:17.324809 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z\" (UID: \"4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z" Oct 31 13:00:17.324887 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:17.324863 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z\" (UID: \"4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z" Oct 31 13:00:17.460007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:17.440399 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z\" (UID: \"4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z" Oct 31 13:00:17.460007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:17.440511 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z\" (UID: \"4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z" Oct 31 13:00:17.460007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:17.440606 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlbpf\" (UniqueName: \"kubernetes.io/projected/4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585-kube-api-access-wlbpf\") pod \"maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z\" (UID: \"4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z" Oct 31 13:00:17.460007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:17.440771 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585-metrics-certs\") pod \"maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z\" (UID: \"4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z" Oct 31 13:00:17.460007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:17.449929 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z\" (UID: \"4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z" Oct 31 13:00:17.483014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:17.469744 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:17.483014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:17.480909 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585-metrics-certs\") pod \"maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z\" (UID: \"4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z" Oct 31 13:00:17.489016 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:17.485479 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z\" (UID: \"4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z" Oct 31 13:00:17.507711 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:17.507638 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlbpf\" (UniqueName: \"kubernetes.io/projected/4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585-kube-api-access-wlbpf\") pod \"maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z\" (UID: \"4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z" Oct 31 13:00:17.571067 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:17.570766 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z" Oct 31 13:00:17.672238 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:17.672142 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:17.745715 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:17.745679 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-z959n" event={"ID":"ceab3693-ea13-4097-a44c-37561dd698d4","Type":"ContainerDied","Data":"c9eefd5db859722b084b035101fba43af31d2b4b672a8d830332a8b5c049dece"} Oct 31 13:00:17.745923 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:17.745757 6841 scope.go:122] "RemoveContainer" containerID="a1fd1bb3bc4cd3d712566013da5c3088749c965f1639de82a98f58a2ea1ea3de" Oct 31 13:00:17.745923 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:17.745691 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-z959n" Oct 31 13:00:17.751073 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:17.751039 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c" event={"ID":"a8afce0f-eb7a-45c5-b837-2024fd7c563a","Type":"ContainerStarted","Data":"188b352140aa98b636eb865382c17d8c072f0f5397e802a79cb0756e412c1bd1"} Oct 31 13:00:17.751375 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:17.751356 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c" Oct 31 13:00:17.767774 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:17.767750 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-z959n"] Oct 31 13:00:17.773007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:17.772967 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c4167730-85cb9464bc-z959n"] Oct 31 13:00:17.793725 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:17.793679 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c" podStartSLOduration=1.96058733 podStartE2EDuration="2.793664923s" totalImagesPullingTime="833.077593ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:17.791260089 +0000 UTC m=+1664.274383016" watchObservedRunningTime="2026-10-31 13:00:17.793664923 +0000 UTC m=+1664.276787823" Oct 31 13:00:17.865305 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:17.865278 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z"] Oct 31 13:00:17.865666 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:00:17.865642 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d3e1a2f_5dfd_4a0a_bf35_4aeaeeeac585.slice/crio-da81ec7edf4862c08370cfb0043d7b781f7b1ef16f652089b78b2651037549b5 WatchSource:0}: Error finding container da81ec7edf4862c08370cfb0043d7b781f7b1ef16f652089b78b2651037549b5: Status 404 returned error can't find the container with id da81ec7edf4862c08370cfb0043d7b781f7b1ef16f652089b78b2651037549b5 Oct 31 13:00:17.909441 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:17.909407 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-x5snf" podUID="fddad4ae-9da5-4956-a2ae-fa2d0b075f2c" containerName="payload-processing" containerID="cri-o://c51d581a359187bb0416ffa2b2f206769b7197fd8bd9559a736d54efb2b0e319" gracePeriod=25 Oct 31 13:00:17.933453 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:17.933408 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-l6mb5" podUID="0db0ab8b-39fe-485d-9d81-c75676f13374" containerName="payload-pre-processing" containerID="cri-o://816e01486b7c1fbd7fbf964ef937c3e2acad92fafa0920a5691a16f4217f5c57" gracePeriod=25 Oct 31 13:00:18.309041 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.308822 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:18.379508 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.379413 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-x5snf" Oct 31 13:00:18.462126 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.462099 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-l6mb5" Oct 31 13:00:18.512292 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.512264 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fddad4ae-9da5-4956-a2ae-fa2d0b075f2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fddad4ae-9da5-4956-a2ae-fa2d0b075f2c-plugins-config-volume\") pod \"fddad4ae-9da5-4956-a2ae-fa2d0b075f2c\" (UID: \"fddad4ae-9da5-4956-a2ae-fa2d0b075f2c\") " Oct 31 13:00:18.512492 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.512471 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fddad4ae-9da5-4956-a2ae-fa2d0b075f2c-kube-api-access-fs8dk\" (UniqueName: \"kubernetes.io/projected/fddad4ae-9da5-4956-a2ae-fa2d0b075f2c-kube-api-access-fs8dk\") pod \"fddad4ae-9da5-4956-a2ae-fa2d0b075f2c\" (UID: \"fddad4ae-9da5-4956-a2ae-fa2d0b075f2c\") " Oct 31 13:00:18.512723 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.512693 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fddad4ae-9da5-4956-a2ae-fa2d0b075f2c-plugins-config-volume" pod "fddad4ae-9da5-4956-a2ae-fa2d0b075f2c" (UID: "fddad4ae-9da5-4956-a2ae-fa2d0b075f2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:00:18.513153 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.513129 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fddad4ae-9da5-4956-a2ae-fa2d0b075f2c-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:18.515014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.514957 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fddad4ae-9da5-4956-a2ae-fa2d0b075f2c-kube-api-access-fs8dk" pod "fddad4ae-9da5-4956-a2ae-fa2d0b075f2c" (UID: "fddad4ae-9da5-4956-a2ae-fa2d0b075f2c"). InnerVolumeSpecName "kube-api-access-fs8dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:18.613946 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.613919 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0db0ab8b-39fe-485d-9d81-c75676f13374-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0db0ab8b-39fe-485d-9d81-c75676f13374-plugins-config-volume\") pod \"0db0ab8b-39fe-485d-9d81-c75676f13374\" (UID: \"0db0ab8b-39fe-485d-9d81-c75676f13374\") " Oct 31 13:00:18.614208 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.614190 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0db0ab8b-39fe-485d-9d81-c75676f13374-kube-api-access-gvnh4\" (UniqueName: \"kubernetes.io/projected/0db0ab8b-39fe-485d-9d81-c75676f13374-kube-api-access-gvnh4\") pod \"0db0ab8b-39fe-485d-9d81-c75676f13374\" (UID: \"0db0ab8b-39fe-485d-9d81-c75676f13374\") " Oct 31 13:00:18.614303 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.614280 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0db0ab8b-39fe-485d-9d81-c75676f13374-plugins-config-volume" pod "0db0ab8b-39fe-485d-9d81-c75676f13374" (UID: "0db0ab8b-39fe-485d-9d81-c75676f13374"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:00:18.614611 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.614593 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs8dk\" (UniqueName: \"kubernetes.io/projected/fddad4ae-9da5-4956-a2ae-fa2d0b075f2c-kube-api-access-fs8dk\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:18.614654 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.614614 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0db0ab8b-39fe-485d-9d81-c75676f13374-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:18.616134 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.616114 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0db0ab8b-39fe-485d-9d81-c75676f13374-kube-api-access-gvnh4" pod "0db0ab8b-39fe-485d-9d81-c75676f13374" (UID: "0db0ab8b-39fe-485d-9d81-c75676f13374"). InnerVolumeSpecName "kube-api-access-gvnh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:18.716109 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.716070 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvnh4\" (UniqueName: \"kubernetes.io/projected/0db0ab8b-39fe-485d-9d81-c75676f13374-kube-api-access-gvnh4\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:18.760671 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.760633 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z" event={"ID":"4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585","Type":"ContainerStarted","Data":"52c869b4451aace43501bd7ead4968df9fcd0f8a179ab092c608bfef344bc546"} Oct 31 13:00:18.761295 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.761271 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z" Oct 31 13:00:18.761429 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.761318 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z" event={"ID":"4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585","Type":"ContainerStarted","Data":"da81ec7edf4862c08370cfb0043d7b781f7b1ef16f652089b78b2651037549b5"} Oct 31 13:00:18.763213 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.763189 6841 generic.go:356] "Generic (PLEG): container finished" podID="fddad4ae-9da5-4956-a2ae-fa2d0b075f2c" containerID="c51d581a359187bb0416ffa2b2f206769b7197fd8bd9559a736d54efb2b0e319" exitCode=0 Oct 31 13:00:18.763301 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.763231 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-x5snf" Oct 31 13:00:18.763301 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.763264 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-x5snf" event={"ID":"fddad4ae-9da5-4956-a2ae-fa2d0b075f2c","Type":"ContainerDied","Data":"c51d581a359187bb0416ffa2b2f206769b7197fd8bd9559a736d54efb2b0e319"} Oct 31 13:00:18.763381 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.763307 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-x5snf" event={"ID":"fddad4ae-9da5-4956-a2ae-fa2d0b075f2c","Type":"ContainerDied","Data":"13263f03d22ad9ac1d7c77719f264f6a91c85700b38a24b83493a407d9c95e52"} Oct 31 13:00:18.763381 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.763338 6841 scope.go:122] "RemoveContainer" containerID="c51d581a359187bb0416ffa2b2f206769b7197fd8bd9559a736d54efb2b0e319" Oct 31 13:00:18.765010 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.764971 6841 generic.go:356] "Generic (PLEG): container finished" podID="0db0ab8b-39fe-485d-9d81-c75676f13374" containerID="816e01486b7c1fbd7fbf964ef937c3e2acad92fafa0920a5691a16f4217f5c57" exitCode=0 Oct 31 13:00:18.765126 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.765065 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-l6mb5" event={"ID":"0db0ab8b-39fe-485d-9d81-c75676f13374","Type":"ContainerDied","Data":"816e01486b7c1fbd7fbf964ef937c3e2acad92fafa0920a5691a16f4217f5c57"} Oct 31 13:00:18.765196 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.765135 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-l6mb5" event={"ID":"0db0ab8b-39fe-485d-9d81-c75676f13374","Type":"ContainerDied","Data":"05531e978f9e9d2d5e79cb48d8b2aa69f2a67e4185315941a483d50ebf654d0b"} Oct 31 13:00:18.765196 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.765077 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-l6mb5" Oct 31 13:00:18.775572 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.775545 6841 scope.go:122] "RemoveContainer" containerID="c51d581a359187bb0416ffa2b2f206769b7197fd8bd9559a736d54efb2b0e319" Oct 31 13:00:18.775885 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:00:18.775857 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c51d581a359187bb0416ffa2b2f206769b7197fd8bd9559a736d54efb2b0e319\": container with ID starting with c51d581a359187bb0416ffa2b2f206769b7197fd8bd9559a736d54efb2b0e319 not found: ID does not exist" containerID="c51d581a359187bb0416ffa2b2f206769b7197fd8bd9559a736d54efb2b0e319" Oct 31 13:00:18.775972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.775895 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c51d581a359187bb0416ffa2b2f206769b7197fd8bd9559a736d54efb2b0e319"} err="failed to get container status \"c51d581a359187bb0416ffa2b2f206769b7197fd8bd9559a736d54efb2b0e319\": rpc error: code = NotFound desc = could not find container \"c51d581a359187bb0416ffa2b2f206769b7197fd8bd9559a736d54efb2b0e319\": container with ID starting with c51d581a359187bb0416ffa2b2f206769b7197fd8bd9559a736d54efb2b0e319 not found: ID does not exist" Oct 31 13:00:18.775972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.775918 6841 scope.go:122] "RemoveContainer" containerID="816e01486b7c1fbd7fbf964ef937c3e2acad92fafa0920a5691a16f4217f5c57" Oct 31 13:00:18.786172 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.786157 6841 scope.go:122] "RemoveContainer" containerID="816e01486b7c1fbd7fbf964ef937c3e2acad92fafa0920a5691a16f4217f5c57" Oct 31 13:00:18.786553 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:00:18.786522 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"816e01486b7c1fbd7fbf964ef937c3e2acad92fafa0920a5691a16f4217f5c57\": container with ID starting with 816e01486b7c1fbd7fbf964ef937c3e2acad92fafa0920a5691a16f4217f5c57 not found: ID does not exist" containerID="816e01486b7c1fbd7fbf964ef937c3e2acad92fafa0920a5691a16f4217f5c57" Oct 31 13:00:18.786664 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.786560 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"816e01486b7c1fbd7fbf964ef937c3e2acad92fafa0920a5691a16f4217f5c57"} err="failed to get container status \"816e01486b7c1fbd7fbf964ef937c3e2acad92fafa0920a5691a16f4217f5c57\": rpc error: code = NotFound desc = could not find container \"816e01486b7c1fbd7fbf964ef937c3e2acad92fafa0920a5691a16f4217f5c57\": container with ID starting with 816e01486b7c1fbd7fbf964ef937c3e2acad92fafa0920a5691a16f4217f5c57 not found: ID does not exist" Oct 31 13:00:18.789107 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.789062 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z" podStartSLOduration=1.163118336 podStartE2EDuration="1.789049382s" totalImagesPullingTime="625.931046ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:18.787179259 +0000 UTC m=+1665.270302162" watchObservedRunningTime="2026-10-31 13:00:18.789049382 +0000 UTC m=+1665.272172284" Oct 31 13:00:18.807619 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.807591 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-l6mb5"] Oct 31 13:00:18.812227 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.812203 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c4167730-6cb6676695-l6mb5"] Oct 31 13:00:18.824342 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.824320 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-x5snf"] Oct 31 13:00:18.828521 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:18.828502 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c4167730-758cb7b74f-x5snf"] Oct 31 13:00:19.243205 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:19.243167 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:19.644574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:19.644504 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0db0ab8b-39fe-485d-9d81-c75676f13374" path="/var/lib/kubelet/pods/0db0ab8b-39fe-485d-9d81-c75676f13374/volumes" Oct 31 13:00:19.645021 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:19.645006 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceab3693-ea13-4097-a44c-37561dd698d4" path="/var/lib/kubelet/pods/ceab3693-ea13-4097-a44c-37561dd698d4/volumes" Oct 31 13:00:19.645391 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:19.645377 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fddad4ae-9da5-4956-a2ae-fa2d0b075f2c" path="/var/lib/kubelet/pods/fddad4ae-9da5-4956-a2ae-fa2d0b075f2c/volumes" Oct 31 13:00:20.054421 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:20.054380 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:20.101962 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:00:20.101913 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:00:21.000010 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:20.999328 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:21.480748 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:21.480711 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:21.717623 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:21.717581 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:22.187033 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:22.183031 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:22.369943 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:22.369907 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd"] Oct 31 13:00:22.370205 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:22.370173 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" podUID="6b1dbe25-ef6f-43bb-a04a-6e497ecbf216" containerName="istio-proxy" containerID="cri-o://a1e3bd6f94001f1595270cece6f834825385fb8995cfa9260fde377b24d14a3e" gracePeriod=30 Oct 31 13:00:22.396721 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:22.396685 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:22.637008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:22.636770 6841 scope.go:122] "RemoveContainer" containerID="73b7e7dbef2672b6df61aee595bff99691e9086cd5d484e29738ff0bc84b6031" Oct 31 13:00:22.637504 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:22.637049 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:22.991173 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:22.991132 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:23.474403 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:23.474360 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:23.734601 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:23.734531 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:23.771333 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:23.771310 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c" Oct 31 13:00:23.901210 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:23.901181 6841 generic.go:356] "Generic (PLEG): container finished" podID="f7617410-6818-4026-bec3-03a042bec96f" containerID="5543d569d47a7c74a7a88a5bb7cc7c22fecba8ec12cbae96cd1ef3f602dc7a2b" exitCode=60 Oct 31 13:00:23.901371 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:23.901249 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890860-2wfc7" event={"ID":"f7617410-6818-4026-bec3-03a042bec96f","Type":"ContainerDied","Data":"5543d569d47a7c74a7a88a5bb7cc7c22fecba8ec12cbae96cd1ef3f602dc7a2b"} Oct 31 13:00:23.901371 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:23.901309 6841 scope.go:122] "RemoveContainer" containerID="73b7e7dbef2672b6df61aee595bff99691e9086cd5d484e29738ff0bc84b6031" Oct 31 13:00:23.901766 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:23.901747 6841 scope.go:122] "RemoveContainer" containerID="5543d569d47a7c74a7a88a5bb7cc7c22fecba8ec12cbae96cd1ef3f602dc7a2b" Oct 31 13:00:23.902198 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:00:23.902172 6841 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890860-2wfc7_odh-ai-gateway-infra(f7617410-6818-4026-bec3-03a042bec96f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890860-2wfc7" podUID="f7617410-6818-4026-bec3-03a042bec96f" Oct 31 13:00:24.338027 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:24.331576 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:24.545461 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:24.545380 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:24.802509 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:24.802447 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:24.812216 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:24.812188 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z" Oct 31 13:00:24.955932 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:24.955899 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890860-2wfc7"] Oct 31 13:00:25.157476 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:25.157448 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890860-2wfc7" Oct 31 13:00:25.260131 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:25.260085 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7617410-6818-4026-bec3-03a042bec96f-kube-api-access-9277q\" (UniqueName: \"kubernetes.io/projected/f7617410-6818-4026-bec3-03a042bec96f-kube-api-access-9277q\") pod \"f7617410-6818-4026-bec3-03a042bec96f\" (UID: \"f7617410-6818-4026-bec3-03a042bec96f\") " Oct 31 13:00:25.260570 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:25.260219 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7617410-6818-4026-bec3-03a042bec96f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7617410-6818-4026-bec3-03a042bec96f-maas-api-service-ca\") pod \"f7617410-6818-4026-bec3-03a042bec96f\" (UID: \"f7617410-6818-4026-bec3-03a042bec96f\") " Oct 31 13:00:25.261300 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:25.261264 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7617410-6818-4026-bec3-03a042bec96f-maas-api-service-ca" pod "f7617410-6818-4026-bec3-03a042bec96f" (UID: "f7617410-6818-4026-bec3-03a042bec96f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:00:25.263067 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:25.263038 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7617410-6818-4026-bec3-03a042bec96f-kube-api-access-9277q" pod "f7617410-6818-4026-bec3-03a042bec96f" (UID: "f7617410-6818-4026-bec3-03a042bec96f"). InnerVolumeSpecName "kube-api-access-9277q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:25.362701 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:25.362669 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9277q\" (UniqueName: \"kubernetes.io/projected/f7617410-6818-4026-bec3-03a042bec96f-kube-api-access-9277q\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:25.362701 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:25.362695 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7617410-6818-4026-bec3-03a042bec96f-maas-api-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:25.735566 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:25.732093 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-8645c8f5f7-x69xk" Oct 31 13:00:25.739115 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:25.738181 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-6b55c6c94c-44jfs" Oct 31 13:00:25.919548 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:25.919523 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890860-2wfc7" Oct 31 13:00:25.919966 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:25.919558 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890860-2wfc7" event={"ID":"f7617410-6818-4026-bec3-03a042bec96f","Type":"ContainerDied","Data":"37cd7fa23a825022d870b86e1856c12b04e7ac99c49e7ea735f897a0ce5a0658"} Oct 31 13:00:25.919966 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:25.919618 6841 scope.go:122] "RemoveContainer" containerID="5543d569d47a7c74a7a88a5bb7cc7c22fecba8ec12cbae96cd1ef3f602dc7a2b" Oct 31 13:00:26.011037 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:26.010898 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890860-2wfc7"] Oct 31 13:00:26.032292 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:26.032246 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890860-2wfc7"] Oct 31 13:00:26.068482 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:26.068441 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:26.868559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:26.868520 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:27.251514 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:27.251475 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:27.547036 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:27.546913 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:27.708470 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:27.697178 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7617410-6818-4026-bec3-03a042bec96f" path="/var/lib/kubelet/pods/f7617410-6818-4026-bec3-03a042bec96f/volumes" Oct 31 13:00:27.810532 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:27.810222 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:28.005143 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.005113 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 13:00:28.009378 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.009349 6841 generic.go:356] "Generic (PLEG): container finished" podID="6b1dbe25-ef6f-43bb-a04a-6e497ecbf216" containerID="a1e3bd6f94001f1595270cece6f834825385fb8995cfa9260fde377b24d14a3e" exitCode=0 Oct 31 13:00:28.009513 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.009396 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" Oct 31 13:00:28.009513 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.009430 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" event={"ID":"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216","Type":"ContainerDied","Data":"a1e3bd6f94001f1595270cece6f834825385fb8995cfa9260fde377b24d14a3e"} Oct 31 13:00:28.009513 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.009482 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd" event={"ID":"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216","Type":"ContainerDied","Data":"ba04af26eae0d1aa9e9401e9f2b86125419446d5b05c5cb721163591b2e2b0e5"} Oct 31 13:00:28.009513 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.009507 6841 scope.go:122] "RemoveContainer" containerID="a1e3bd6f94001f1595270cece6f834825385fb8995cfa9260fde377b24d14a3e" Oct 31 13:00:28.019936 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.019916 6841 scope.go:122] "RemoveContainer" containerID="ec182338bc3277355f454c782942169d58e8ab71578976ef8999c1c47e490a03" Oct 31 13:00:28.029659 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.029624 6841 scope.go:122] "RemoveContainer" containerID="a1e3bd6f94001f1595270cece6f834825385fb8995cfa9260fde377b24d14a3e" Oct 31 13:00:28.029940 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:00:28.029921 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1e3bd6f94001f1595270cece6f834825385fb8995cfa9260fde377b24d14a3e\": container with ID starting with a1e3bd6f94001f1595270cece6f834825385fb8995cfa9260fde377b24d14a3e not found: ID does not exist" containerID="a1e3bd6f94001f1595270cece6f834825385fb8995cfa9260fde377b24d14a3e" Oct 31 13:00:28.030039 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.029952 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1e3bd6f94001f1595270cece6f834825385fb8995cfa9260fde377b24d14a3e"} err="failed to get container status \"a1e3bd6f94001f1595270cece6f834825385fb8995cfa9260fde377b24d14a3e\": rpc error: code = NotFound desc = could not find container \"a1e3bd6f94001f1595270cece6f834825385fb8995cfa9260fde377b24d14a3e\": container with ID starting with a1e3bd6f94001f1595270cece6f834825385fb8995cfa9260fde377b24d14a3e not found: ID does not exist" Oct 31 13:00:28.030039 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.029969 6841 scope.go:122] "RemoveContainer" containerID="ec182338bc3277355f454c782942169d58e8ab71578976ef8999c1c47e490a03" Oct 31 13:00:28.030263 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:00:28.030240 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec182338bc3277355f454c782942169d58e8ab71578976ef8999c1c47e490a03\": container with ID starting with ec182338bc3277355f454c782942169d58e8ab71578976ef8999c1c47e490a03 not found: ID does not exist" containerID="ec182338bc3277355f454c782942169d58e8ab71578976ef8999c1c47e490a03" Oct 31 13:00:28.030306 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.030269 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec182338bc3277355f454c782942169d58e8ab71578976ef8999c1c47e490a03"} err="failed to get container status \"ec182338bc3277355f454c782942169d58e8ab71578976ef8999c1c47e490a03\": rpc error: code = NotFound desc = could not find container \"ec182338bc3277355f454c782942169d58e8ab71578976ef8999c1c47e490a03\": container with ID starting with ec182338bc3277355f454c782942169d58e8ab71578976ef8999c1c47e490a03 not found: ID does not exist" Oct 31 13:00:28.091966 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.091870 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:28.104110 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.104079 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-istio-token\" (UniqueName: \"kubernetes.io/projected/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-istio-token\") pod \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\" (UID: \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\") " Oct 31 13:00:28.104276 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.104144 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-istiod-ca-cert\") pod \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\" (UID: \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\") " Oct 31 13:00:28.104276 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.104205 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-istio-podinfo\") pod \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\" (UID: \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\") " Oct 31 13:00:28.104382 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.104363 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-istio-data\") pod \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\" (UID: \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\") " Oct 31 13:00:28.104458 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.104432 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-workload-certs\") pod \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\" (UID: \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\") " Oct 31 13:00:28.104536 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.104500 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-kube-api-access-z66l7\" (UniqueName: \"kubernetes.io/projected/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-kube-api-access-z66l7\") pod \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\" (UID: \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\") " Oct 31 13:00:28.104602 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.104546 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-istio-envoy\") pod \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\" (UID: \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\") " Oct 31 13:00:28.104602 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.104576 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-workload-socket\") pod \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\" (UID: \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\") " Oct 31 13:00:28.104712 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.104692 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-credential-socket\") pod \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\" (UID: \"6b1dbe25-ef6f-43bb-a04a-6e497ecbf216\") " Oct 31 13:00:28.105060 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.104938 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-workload-socket" pod "6b1dbe25-ef6f-43bb-a04a-6e497ecbf216" (UID: "6b1dbe25-ef6f-43bb-a04a-6e497ecbf216"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:28.105167 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.105129 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-workload-certs" pod "6b1dbe25-ef6f-43bb-a04a-6e497ecbf216" (UID: "6b1dbe25-ef6f-43bb-a04a-6e497ecbf216"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:28.105563 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.105322 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-credential-socket" pod "6b1dbe25-ef6f-43bb-a04a-6e497ecbf216" (UID: "6b1dbe25-ef6f-43bb-a04a-6e497ecbf216"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:28.105563 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.105435 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:28.105563 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.105458 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:28.105563 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.105474 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:28.105563 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.105525 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-istio-data" pod "6b1dbe25-ef6f-43bb-a04a-6e497ecbf216" (UID: "6b1dbe25-ef6f-43bb-a04a-6e497ecbf216"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:28.105871 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.105779 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-istiod-ca-cert" pod "6b1dbe25-ef6f-43bb-a04a-6e497ecbf216" (UID: "6b1dbe25-ef6f-43bb-a04a-6e497ecbf216"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:00:28.107021 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.106931 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-istio-podinfo" pod "6b1dbe25-ef6f-43bb-a04a-6e497ecbf216" (UID: "6b1dbe25-ef6f-43bb-a04a-6e497ecbf216"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:00:28.112728 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.108756 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-istio-envoy" pod "6b1dbe25-ef6f-43bb-a04a-6e497ecbf216" (UID: "6b1dbe25-ef6f-43bb-a04a-6e497ecbf216"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:28.112728 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.111213 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-istio-token" pod "6b1dbe25-ef6f-43bb-a04a-6e497ecbf216" (UID: "6b1dbe25-ef6f-43bb-a04a-6e497ecbf216"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:28.112898 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.112804 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-kube-api-access-z66l7" pod "6b1dbe25-ef6f-43bb-a04a-6e497ecbf216" (UID: "6b1dbe25-ef6f-43bb-a04a-6e497ecbf216"). InnerVolumeSpecName "kube-api-access-z66l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:28.207120 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.207081 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:28.207120 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.207108 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:28.207120 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.207118 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:28.207120 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.207129 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:28.207467 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.207139 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z66l7\" (UniqueName: \"kubernetes.io/projected/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-kube-api-access-z66l7\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:28.207467 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.207149 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:28.349054 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.348708 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:28.437107 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.437068 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd"] Oct 31 13:00:28.445955 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:28.445893 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c4167730-openshift-default-578df84889-42vvd"] Oct 31 13:00:29.173621 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:29.173580 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ccf963-8645c8f5f7-x69xk"] Oct 31 13:00:29.193929 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:29.193896 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-6b55c6c94c-44jfs"] Oct 31 13:00:29.319964 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:29.319254 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:29.566589 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:29.566500 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:29.720541 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:29.720504 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b1dbe25-ef6f-43bb-a04a-6e497ecbf216" path="/var/lib/kubelet/pods/6b1dbe25-ef6f-43bb-a04a-6e497ecbf216/volumes" Oct 31 13:00:30.295039 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:30.293277 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:30.439111 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:30.437764 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-56a83a-7d476bc98c-mxvqr"] Oct 31 13:00:30.447278 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:30.440158 6841 cpu_manager.go:414] "containerMap: removing container" podUID="f7617410-6818-4026-bec3-03a042bec96f" containerName="cleanup" Oct 31 13:00:30.541171 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:30.530977 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7617410-6818-4026-bec3-03a042bec96f" containerName="cleanup" Oct 31 13:00:30.541503 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:30.541481 6841 cpu_manager.go:414] "containerMap: removing container" podUID="f7617410-6818-4026-bec3-03a042bec96f" containerName="cleanup" Oct 31 13:00:30.541632 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:30.541617 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7617410-6818-4026-bec3-03a042bec96f" containerName="cleanup" Oct 31 13:00:30.541780 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:30.541765 6841 cpu_manager.go:414] "containerMap: removing container" podUID="0db0ab8b-39fe-485d-9d81-c75676f13374" containerName="payload-pre-processing" Oct 31 13:00:30.541904 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:30.541889 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0db0ab8b-39fe-485d-9d81-c75676f13374" containerName="payload-pre-processing" Oct 31 13:00:30.542137 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:30.542121 6841 cpu_manager.go:414] "containerMap: removing container" podUID="6b1dbe25-ef6f-43bb-a04a-6e497ecbf216" containerName="istio-proxy" Oct 31 13:00:30.542269 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:30.542255 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b1dbe25-ef6f-43bb-a04a-6e497ecbf216" containerName="istio-proxy" Oct 31 13:00:30.542418 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:30.542406 6841 cpu_manager.go:414] "containerMap: removing container" podUID="6b1dbe25-ef6f-43bb-a04a-6e497ecbf216" containerName="istio-proxy" Oct 31 13:00:30.542534 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:30.542521 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b1dbe25-ef6f-43bb-a04a-6e497ecbf216" containerName="istio-proxy" Oct 31 13:00:30.542660 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:30.542648 6841 cpu_manager.go:414] "containerMap: removing container" podUID="fddad4ae-9da5-4956-a2ae-fa2d0b075f2c" containerName="payload-processing" Oct 31 13:00:30.542756 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:30.542744 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fddad4ae-9da5-4956-a2ae-fa2d0b075f2c" containerName="payload-processing" Oct 31 13:00:30.543507 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:30.543485 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b1dbe25-ef6f-43bb-a04a-6e497ecbf216" containerName="istio-proxy" Oct 31 13:00:30.543684 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:30.543670 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b1dbe25-ef6f-43bb-a04a-6e497ecbf216" containerName="istio-proxy" Oct 31 13:00:30.708007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:30.700964 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="fddad4ae-9da5-4956-a2ae-fa2d0b075f2c" containerName="payload-processing" Oct 31 13:00:30.708007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:30.701096 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7617410-6818-4026-bec3-03a042bec96f" containerName="cleanup" Oct 31 13:00:30.708007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:30.701122 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7617410-6818-4026-bec3-03a042bec96f" containerName="cleanup" Oct 31 13:00:30.708007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:30.701138 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7617410-6818-4026-bec3-03a042bec96f" containerName="cleanup" Oct 31 13:00:30.708007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:30.701150 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="0db0ab8b-39fe-485d-9d81-c75676f13374" containerName="payload-pre-processing" Oct 31 13:00:30.720310 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:30.589930 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:30.773111 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:30.773058 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-56a83a-7865d4d4f7-ggdfx"] Oct 31 13:00:30.779388 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:30.774167 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56a83a-7d476bc98c-mxvqr" Oct 31 13:00:30.779388 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:30.775211 6841 cpu_manager.go:414] "containerMap: removing container" podUID="f7617410-6818-4026-bec3-03a042bec96f" containerName="cleanup" Oct 31 13:00:30.779388 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:30.775232 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7617410-6818-4026-bec3-03a042bec96f" containerName="cleanup" Oct 31 13:00:30.791638 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:30.786325 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-56a83a\"" Oct 31 13:00:30.821275 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:30.821242 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-56a83a-7d476bc98c-mxvqr"] Oct 31 13:00:30.821684 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:30.821667 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-56a83a-7865d4d4f7-ggdfx"] Oct 31 13:00:30.821838 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:30.821599 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-56a83a-7865d4d4f7-ggdfx" Oct 31 13:00:30.875365 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:30.875330 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-56a83a-dockercfg-j9jbg\"" Oct 31 13:00:30.921446 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:30.921063 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc662639-7616-4ebe-88cc-ef4026b066c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-56a83a-7d476bc98c-mxvqr\" (UID: \"bc662639-7616-4ebe-88cc-ef4026b066c8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-56a83a-7d476bc98c-mxvqr" Oct 31 13:00:30.921446 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:30.921220 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjlxw\" (UniqueName: \"kubernetes.io/projected/6324318a-fb95-45af-a04d-8c73774e8c84-kube-api-access-kjlxw\") pod \"payload-processing-e2e-api-b-56a83a-7865d4d4f7-ggdfx\" (UID: \"6324318a-fb95-45af-a04d-8c73774e8c84\") " pod="openshift-ingress/payload-processing-e2e-api-b-56a83a-7865d4d4f7-ggdfx" Oct 31 13:00:30.921446 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:30.921342 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shvb6\" (UniqueName: \"kubernetes.io/projected/bc662639-7616-4ebe-88cc-ef4026b066c8-kube-api-access-shvb6\") pod \"payload-pre-processing-e2e-api-b-56a83a-7d476bc98c-mxvqr\" (UID: \"bc662639-7616-4ebe-88cc-ef4026b066c8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-56a83a-7d476bc98c-mxvqr" Oct 31 13:00:30.921783 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:30.921680 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6324318a-fb95-45af-a04d-8c73774e8c84-plugins-config-volume\") pod \"payload-processing-e2e-api-b-56a83a-7865d4d4f7-ggdfx\" (UID: \"6324318a-fb95-45af-a04d-8c73774e8c84\") " pod="openshift-ingress/payload-processing-e2e-api-b-56a83a-7865d4d4f7-ggdfx" Oct 31 13:00:30.964188 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:30.964093 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:31.034070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:31.030046 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc662639-7616-4ebe-88cc-ef4026b066c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-56a83a-7d476bc98c-mxvqr\" (UID: \"bc662639-7616-4ebe-88cc-ef4026b066c8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-56a83a-7d476bc98c-mxvqr" Oct 31 13:00:31.034070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:31.030205 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjlxw\" (UniqueName: \"kubernetes.io/projected/6324318a-fb95-45af-a04d-8c73774e8c84-kube-api-access-kjlxw\") pod \"payload-processing-e2e-api-b-56a83a-7865d4d4f7-ggdfx\" (UID: \"6324318a-fb95-45af-a04d-8c73774e8c84\") " pod="openshift-ingress/payload-processing-e2e-api-b-56a83a-7865d4d4f7-ggdfx" Oct 31 13:00:31.034070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:31.030313 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shvb6\" (UniqueName: \"kubernetes.io/projected/bc662639-7616-4ebe-88cc-ef4026b066c8-kube-api-access-shvb6\") pod \"payload-pre-processing-e2e-api-b-56a83a-7d476bc98c-mxvqr\" (UID: \"bc662639-7616-4ebe-88cc-ef4026b066c8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-56a83a-7d476bc98c-mxvqr" Oct 31 13:00:31.034070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:31.030413 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6324318a-fb95-45af-a04d-8c73774e8c84-plugins-config-volume\") pod \"payload-processing-e2e-api-b-56a83a-7865d4d4f7-ggdfx\" (UID: \"6324318a-fb95-45af-a04d-8c73774e8c84\") " pod="openshift-ingress/payload-processing-e2e-api-b-56a83a-7865d4d4f7-ggdfx" Oct 31 13:00:31.034070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:31.031532 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6324318a-fb95-45af-a04d-8c73774e8c84-plugins-config-volume\") pod \"payload-processing-e2e-api-b-56a83a-7865d4d4f7-ggdfx\" (UID: \"6324318a-fb95-45af-a04d-8c73774e8c84\") " pod="openshift-ingress/payload-processing-e2e-api-b-56a83a-7865d4d4f7-ggdfx" Oct 31 13:00:31.034070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:31.032126 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc662639-7616-4ebe-88cc-ef4026b066c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-56a83a-7d476bc98c-mxvqr\" (UID: \"bc662639-7616-4ebe-88cc-ef4026b066c8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-56a83a-7d476bc98c-mxvqr" Oct 31 13:00:31.072839 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:31.072805 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjlxw\" (UniqueName: \"kubernetes.io/projected/6324318a-fb95-45af-a04d-8c73774e8c84-kube-api-access-kjlxw\") pod \"payload-processing-e2e-api-b-56a83a-7865d4d4f7-ggdfx\" (UID: \"6324318a-fb95-45af-a04d-8c73774e8c84\") " pod="openshift-ingress/payload-processing-e2e-api-b-56a83a-7865d4d4f7-ggdfx" Oct 31 13:00:31.081018 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:31.078156 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shvb6\" (UniqueName: \"kubernetes.io/projected/bc662639-7616-4ebe-88cc-ef4026b066c8-kube-api-access-shvb6\") pod \"payload-pre-processing-e2e-api-b-56a83a-7d476bc98c-mxvqr\" (UID: \"bc662639-7616-4ebe-88cc-ef4026b066c8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-56a83a-7d476bc98c-mxvqr" Oct 31 13:00:31.139005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:31.138311 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56a83a-7d476bc98c-mxvqr" Oct 31 13:00:31.221111 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:31.214413 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-56a83a-7865d4d4f7-ggdfx" Oct 31 13:00:31.226082 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:31.225365 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:31.544782 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:31.544748 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:32.037971 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:32.036605 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-56a83a-7d476bc98c-mxvqr"] Oct 31 13:00:32.085798 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:00:32.085756 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc662639_7616_4ebe_88cc_ef4026b066c8.slice/crio-e2abfebf65d21c8d12a9cc62e0a992761d79b27186d0e1eb3027411cf053ca27 WatchSource:0}: Error finding container e2abfebf65d21c8d12a9cc62e0a992761d79b27186d0e1eb3027411cf053ca27: Status 404 returned error can't find the container with id e2abfebf65d21c8d12a9cc62e0a992761d79b27186d0e1eb3027411cf053ca27 Oct 31 13:00:32.138705 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:32.138640 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-56a83a-7865d4d4f7-ggdfx"] Oct 31 13:00:32.254004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:32.244683 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:32.981049 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:32.979564 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-56a83a-7865d4d4f7-ggdfx" event={"ID":"6324318a-fb95-45af-a04d-8c73774e8c84","Type":"ContainerStarted","Data":"619cfe6ee46c8be893a6f30ce6bbdd1d9f3017f2a35b2116465cb9166c56c4e0"} Oct 31 13:00:32.981049 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:32.979650 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-56a83a-7865d4d4f7-ggdfx" event={"ID":"6324318a-fb95-45af-a04d-8c73774e8c84","Type":"ContainerStarted","Data":"968d518175e1e017894ac90aafa469a80e83dae4ba83d482064c0c0c03d158d9"} Oct 31 13:00:33.000014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:32.994297 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-56a83a-7865d4d4f7-ggdfx" Oct 31 13:00:33.000014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:32.996949 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56a83a-7d476bc98c-mxvqr" event={"ID":"bc662639-7616-4ebe-88cc-ef4026b066c8","Type":"ContainerStarted","Data":"f8149ee2c7f053495790d73a1a36258eb6d3c42b1e3db6925e1a454f68105d28"} Oct 31 13:00:33.000014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:32.997032 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56a83a-7d476bc98c-mxvqr" event={"ID":"bc662639-7616-4ebe-88cc-ef4026b066c8","Type":"ContainerStarted","Data":"e2abfebf65d21c8d12a9cc62e0a992761d79b27186d0e1eb3027411cf053ca27"} Oct 31 13:00:33.000014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:32.997938 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56a83a-7d476bc98c-mxvqr" Oct 31 13:00:33.024005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:33.021810 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:33.070502 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:33.070440 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-56a83a-7865d4d4f7-ggdfx" podStartSLOduration=4.070417147 podStartE2EDuration="4.070417147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:33.038495798 +0000 UTC m=+1679.521618701" watchObservedRunningTime="2026-10-31 13:00:33.070417147 +0000 UTC m=+1679.553540049" Oct 31 13:00:33.289062 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:33.287194 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56a83a-7d476bc98c-mxvqr" podStartSLOduration=4.287172964 podStartE2EDuration="4.287172964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:33.067542691 +0000 UTC m=+1679.550665593" watchObservedRunningTime="2026-10-31 13:00:33.287172964 +0000 UTC m=+1679.770295866" Oct 31 13:00:33.978168 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:33.974427 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:34.182627 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:34.182106 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-8645c8f5f7-x69xk" podUID="f7b40a85-2601-4987-8b1e-b4806dcada3a" containerName="payload-processing" containerID="cri-o://bdde9b3c447a3acdf5a85ae890319c3bed374f870725712b45af04b9a35c8c07" gracePeriod=25 Oct 31 13:00:34.213526 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:34.211918 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-6b55c6c94c-44jfs" podUID="bf16598d-b76a-4dda-aba1-b801582fad71" containerName="payload-pre-processing" containerID="cri-o://e643d407546465bed47e05556bfd61678943f64bb5b28d315a8431a8c985cbf4" gracePeriod=25 Oct 31 13:00:34.539225 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:34.538395 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:34.636317 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:34.636176 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-5d88b777bb-dnwdj"] Oct 31 13:00:34.708008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:34.657739 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-5d88b777bb-dnwdj" Oct 31 13:00:34.745008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:34.742242 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67n9q\" (UniqueName: \"kubernetes.io/projected/e3fee6c8-5e84-4011-a626-7a3da7239b06-kube-api-access-67n9q\") pod \"payload-processing-e2e-ipp-a-bc9ae8-5d88b777bb-dnwdj\" (UID: \"e3fee6c8-5e84-4011-a626-7a3da7239b06\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-5d88b777bb-dnwdj" Oct 31 13:00:34.745008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:34.742305 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3fee6c8-5e84-4011-a626-7a3da7239b06-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bc9ae8-5d88b777bb-dnwdj\" (UID: \"e3fee6c8-5e84-4011-a626-7a3da7239b06\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-5d88b777bb-dnwdj" Oct 31 13:00:34.927422 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:34.857456 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bc9ae8\"" Oct 31 13:00:34.927422 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:34.857838 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bc9ae8-dockercfg-gw9sc\"" Oct 31 13:00:34.927422 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:34.862316 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67n9q\" (UniqueName: \"kubernetes.io/projected/e3fee6c8-5e84-4011-a626-7a3da7239b06-kube-api-access-67n9q\") pod \"payload-processing-e2e-ipp-a-bc9ae8-5d88b777bb-dnwdj\" (UID: \"e3fee6c8-5e84-4011-a626-7a3da7239b06\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-5d88b777bb-dnwdj" Oct 31 13:00:34.927422 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:34.862371 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3fee6c8-5e84-4011-a626-7a3da7239b06-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bc9ae8-5d88b777bb-dnwdj\" (UID: \"e3fee6c8-5e84-4011-a626-7a3da7239b06\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-5d88b777bb-dnwdj" Oct 31 13:00:34.958269 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:34.958182 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:35.024008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:35.016937 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3fee6c8-5e84-4011-a626-7a3da7239b06-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bc9ae8-5d88b777bb-dnwdj\" (UID: \"e3fee6c8-5e84-4011-a626-7a3da7239b06\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-5d88b777bb-dnwdj" Oct 31 13:00:35.191050 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:35.168834 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-5d88b777bb-dnwdj"] Oct 31 13:00:35.228768 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:35.214515 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:35.273019 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:35.263580 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67n9q\" (UniqueName: \"kubernetes.io/projected/e3fee6c8-5e84-4011-a626-7a3da7239b06-kube-api-access-67n9q\") pod \"payload-processing-e2e-ipp-a-bc9ae8-5d88b777bb-dnwdj\" (UID: \"e3fee6c8-5e84-4011-a626-7a3da7239b06\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-5d88b777bb-dnwdj" Oct 31 13:00:35.303576 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:35.303353 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-5d88b777bb-dnwdj" Oct 31 13:00:35.303949 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:35.303924 6841 generic.go:356] "Generic (PLEG): container finished" podID="bf16598d-b76a-4dda-aba1-b801582fad71" containerID="e643d407546465bed47e05556bfd61678943f64bb5b28d315a8431a8c985cbf4" exitCode=0 Oct 31 13:00:35.314210 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:35.308316 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-6b55c6c94c-44jfs" event={"ID":"bf16598d-b76a-4dda-aba1-b801582fad71","Type":"ContainerDied","Data":"e643d407546465bed47e05556bfd61678943f64bb5b28d315a8431a8c985cbf4"} Oct 31 13:00:35.382164 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:35.354616 6841 generic.go:356] "Generic (PLEG): container finished" podID="f7b40a85-2601-4987-8b1e-b4806dcada3a" containerID="bdde9b3c447a3acdf5a85ae890319c3bed374f870725712b45af04b9a35c8c07" exitCode=0 Oct 31 13:00:35.382164 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:35.355660 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-8645c8f5f7-x69xk" event={"ID":"f7b40a85-2601-4987-8b1e-b4806dcada3a","Type":"ContainerDied","Data":"bdde9b3c447a3acdf5a85ae890319c3bed374f870725712b45af04b9a35c8c07"} Oct 31 13:00:36.160416 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:36.142555 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:36.238048 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:36.230236 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-8645c8f5f7-x69xk" Oct 31 13:00:36.266856 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:36.262030 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7b40a85-2601-4987-8b1e-b4806dcada3a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b40a85-2601-4987-8b1e-b4806dcada3a-plugins-config-volume\") pod \"f7b40a85-2601-4987-8b1e-b4806dcada3a\" (UID: \"f7b40a85-2601-4987-8b1e-b4806dcada3a\") " Oct 31 13:00:36.266856 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:36.262156 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7b40a85-2601-4987-8b1e-b4806dcada3a-kube-api-access-vj555\" (UniqueName: \"kubernetes.io/projected/f7b40a85-2601-4987-8b1e-b4806dcada3a-kube-api-access-vj555\") pod \"f7b40a85-2601-4987-8b1e-b4806dcada3a\" (UID: \"f7b40a85-2601-4987-8b1e-b4806dcada3a\") " Oct 31 13:00:36.266856 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:36.266172 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7b40a85-2601-4987-8b1e-b4806dcada3a-plugins-config-volume" pod "f7b40a85-2601-4987-8b1e-b4806dcada3a" (UID: "f7b40a85-2601-4987-8b1e-b4806dcada3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:00:36.275261 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:36.275197 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b40a85-2601-4987-8b1e-b4806dcada3a-kube-api-access-vj555" pod "f7b40a85-2601-4987-8b1e-b4806dcada3a" (UID: "f7b40a85-2601-4987-8b1e-b4806dcada3a"). InnerVolumeSpecName "kube-api-access-vj555". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:36.432006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:36.430616 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b40a85-2601-4987-8b1e-b4806dcada3a-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:36.432006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:36.430673 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj555\" (UniqueName: \"kubernetes.io/projected/f7b40a85-2601-4987-8b1e-b4806dcada3a-kube-api-access-vj555\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:36.507954 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:36.507671 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:36.522957 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:36.522924 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-6b55c6c94c-44jfs" event={"ID":"bf16598d-b76a-4dda-aba1-b801582fad71","Type":"ContainerDied","Data":"979dbec31e6c3e3640b0ef6767bcf96acbce18478faae9a7e2edce6a5d846cdc"} Oct 31 13:00:36.523171 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:36.523011 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="979dbec31e6c3e3640b0ef6767bcf96acbce18478faae9a7e2edce6a5d846cdc" Oct 31 13:00:36.523797 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:36.523414 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-6b55c6c94c-44jfs" Oct 31 13:00:36.529014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:36.525313 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-8645c8f5f7-x69xk" Oct 31 13:00:36.529014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:36.525344 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-8645c8f5f7-x69xk" event={"ID":"f7b40a85-2601-4987-8b1e-b4806dcada3a","Type":"ContainerDied","Data":"3cd758823b7c2ab8f8daf048b58a26f0ddadb0e30cc87b05ca13051c275d2bff"} Oct 31 13:00:36.529014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:36.525416 6841 scope.go:122] "RemoveContainer" containerID="bdde9b3c447a3acdf5a85ae890319c3bed374f870725712b45af04b9a35c8c07" Oct 31 13:00:36.620531 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:36.620493 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ccf963-8645c8f5f7-x69xk"] Oct 31 13:00:36.626777 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:36.626747 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ccf963-8645c8f5f7-x69xk"] Oct 31 13:00:36.633435 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:36.633406 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf16598d-b76a-4dda-aba1-b801582fad71-kube-api-access-l2kdq\" (UniqueName: \"kubernetes.io/projected/bf16598d-b76a-4dda-aba1-b801582fad71-kube-api-access-l2kdq\") pod \"bf16598d-b76a-4dda-aba1-b801582fad71\" (UID: \"bf16598d-b76a-4dda-aba1-b801582fad71\") " Oct 31 13:00:36.633628 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:36.633610 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf16598d-b76a-4dda-aba1-b801582fad71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf16598d-b76a-4dda-aba1-b801582fad71-plugins-config-volume\") pod \"bf16598d-b76a-4dda-aba1-b801582fad71\" (UID: \"bf16598d-b76a-4dda-aba1-b801582fad71\") " Oct 31 13:00:36.676540 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:36.635849 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf16598d-b76a-4dda-aba1-b801582fad71-plugins-config-volume" pod "bf16598d-b76a-4dda-aba1-b801582fad71" (UID: "bf16598d-b76a-4dda-aba1-b801582fad71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:00:36.676540 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:36.640547 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf16598d-b76a-4dda-aba1-b801582fad71-kube-api-access-l2kdq" pod "bf16598d-b76a-4dda-aba1-b801582fad71" (UID: "bf16598d-b76a-4dda-aba1-b801582fad71"). InnerVolumeSpecName "kube-api-access-l2kdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:36.719905 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:36.719603 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-8645c8f5f7-x69xk" podUID="f7b40a85-2601-4987-8b1e-b4806dcada3a" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.99:50052\" within 1s: context deadline exceeded" Oct 31 13:00:36.719905 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:00:36.719690 6841 logging.go:55] [core] [Channel #953 SubChannel #954]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.99:50052", ServerName: "10.134.0.99:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.99:50052: operation was canceled" Oct 31 13:00:36.735916 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:36.735541 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2kdq\" (UniqueName: \"kubernetes.io/projected/bf16598d-b76a-4dda-aba1-b801582fad71-kube-api-access-l2kdq\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:36.735916 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:36.735571 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf16598d-b76a-4dda-aba1-b801582fad71-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:36.758891 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:36.757359 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-6b55c6c94c-44jfs" podUID="bf16598d-b76a-4dda-aba1-b801582fad71" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.98:50052\" within 1s: context deadline exceeded" Oct 31 13:00:36.758891 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:00:36.757714 6841 logging.go:55] [core] [Channel #955 SubChannel #956]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.98:50052", ServerName: "10.134.0.98:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.98:50052: operation was canceled" Oct 31 13:00:36.800022 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:36.799953 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:36.909371 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:36.909342 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-5d88b777bb-dnwdj"] Oct 31 13:00:37.077063 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.076954 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:37.172013 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.171949 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-875df4b95-k59g5"] Oct 31 13:00:37.172710 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.172688 6841 cpu_manager.go:414] "containerMap: removing container" podUID="f7b40a85-2601-4987-8b1e-b4806dcada3a" containerName="payload-processing" Oct 31 13:00:37.172710 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.172708 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7b40a85-2601-4987-8b1e-b4806dcada3a" containerName="payload-processing" Oct 31 13:00:37.172891 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.172750 6841 cpu_manager.go:414] "containerMap: removing container" podUID="bf16598d-b76a-4dda-aba1-b801582fad71" containerName="payload-pre-processing" Oct 31 13:00:37.172891 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.172756 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf16598d-b76a-4dda-aba1-b801582fad71" containerName="payload-pre-processing" Oct 31 13:00:37.172891 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.172880 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf16598d-b76a-4dda-aba1-b801582fad71" containerName="payload-pre-processing" Oct 31 13:00:37.173099 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.172893 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7b40a85-2601-4987-8b1e-b4806dcada3a" containerName="payload-processing" Oct 31 13:00:37.180437 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.180415 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-875df4b95-k59g5" Oct 31 13:00:37.186227 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.185314 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ccf963-dockercfg-9fqq2\"" Oct 31 13:00:37.186357 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.186307 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-875df4b95-k59g5"] Oct 31 13:00:37.325012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.291516 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl87g\" (UniqueName: \"kubernetes.io/projected/399ac99e-25a1-44d3-b774-04991fbe4f47-kube-api-access-vl87g\") pod \"payload-pre-processing-e2e-ipp-a-ccf963-875df4b95-k59g5\" (UID: \"399ac99e-25a1-44d3-b774-04991fbe4f47\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-875df4b95-k59g5" Oct 31 13:00:37.325012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.291618 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/399ac99e-25a1-44d3-b774-04991fbe4f47-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ccf963-875df4b95-k59g5\" (UID: \"399ac99e-25a1-44d3-b774-04991fbe4f47\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-875df4b95-k59g5" Oct 31 13:00:37.369132 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.364491 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:37.391078 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.390936 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ccf963-cff4fd7fd-dgj5x"] Oct 31 13:00:37.396591 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.393398 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl87g\" (UniqueName: \"kubernetes.io/projected/399ac99e-25a1-44d3-b774-04991fbe4f47-kube-api-access-vl87g\") pod \"payload-pre-processing-e2e-ipp-a-ccf963-875df4b95-k59g5\" (UID: \"399ac99e-25a1-44d3-b774-04991fbe4f47\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-875df4b95-k59g5" Oct 31 13:00:37.396591 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.393531 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/399ac99e-25a1-44d3-b774-04991fbe4f47-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ccf963-875df4b95-k59g5\" (UID: \"399ac99e-25a1-44d3-b774-04991fbe4f47\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-875df4b95-k59g5" Oct 31 13:00:37.398350 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.397873 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/399ac99e-25a1-44d3-b774-04991fbe4f47-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ccf963-875df4b95-k59g5\" (UID: \"399ac99e-25a1-44d3-b774-04991fbe4f47\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-875df4b95-k59g5" Oct 31 13:00:37.424650 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.420972 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-cff4fd7fd-dgj5x" Oct 31 13:00:37.436373 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.436336 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ccf963-cff4fd7fd-dgj5x"] Oct 31 13:00:37.455043 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.454956 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl87g\" (UniqueName: \"kubernetes.io/projected/399ac99e-25a1-44d3-b774-04991fbe4f47-kube-api-access-vl87g\") pod \"payload-pre-processing-e2e-ipp-a-ccf963-875df4b95-k59g5\" (UID: \"399ac99e-25a1-44d3-b774-04991fbe4f47\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-875df4b95-k59g5" Oct 31 13:00:37.500645 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.499646 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ccf963-cff4fd7fd-dgj5x\" (UID: \"e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-cff4fd7fd-dgj5x" Oct 31 13:00:37.500645 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.499743 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4jbx\" (UniqueName: \"kubernetes.io/projected/e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9-kube-api-access-w4jbx\") pod \"payload-processing-e2e-ipp-a-ccf963-cff4fd7fd-dgj5x\" (UID: \"e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-cff4fd7fd-dgj5x" Oct 31 13:00:37.543721 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.543333 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-875df4b95-k59g5" Oct 31 13:00:37.555251 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.555223 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-6b55c6c94c-44jfs" Oct 31 13:00:37.556970 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.556923 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-5d88b777bb-dnwdj" event={"ID":"e3fee6c8-5e84-4011-a626-7a3da7239b06","Type":"ContainerStarted","Data":"0402498952dd9678242763e3355efecbbe94ee61d06dfb23c1641e164e555e87"} Oct 31 13:00:37.601678 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.601643 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ccf963-cff4fd7fd-dgj5x\" (UID: \"e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-cff4fd7fd-dgj5x" Oct 31 13:00:37.601840 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.601704 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4jbx\" (UniqueName: \"kubernetes.io/projected/e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9-kube-api-access-w4jbx\") pod \"payload-processing-e2e-ipp-a-ccf963-cff4fd7fd-dgj5x\" (UID: \"e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-cff4fd7fd-dgj5x" Oct 31 13:00:37.603693 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.602746 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ccf963-cff4fd7fd-dgj5x\" (UID: \"e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-cff4fd7fd-dgj5x" Oct 31 13:00:37.655746 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.655710 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4jbx\" (UniqueName: \"kubernetes.io/projected/e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9-kube-api-access-w4jbx\") pod \"payload-processing-e2e-ipp-a-ccf963-cff4fd7fd-dgj5x\" (UID: \"e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-cff4fd7fd-dgj5x" Oct 31 13:00:37.669412 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.664632 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7b40a85-2601-4987-8b1e-b4806dcada3a" path="/var/lib/kubelet/pods/f7b40a85-2601-4987-8b1e-b4806dcada3a/volumes" Oct 31 13:00:37.669412 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.665104 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-6b55c6c94c-44jfs"] Oct 31 13:00:37.669412 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.665142 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-6b55c6c94c-44jfs"] Oct 31 13:00:37.758162 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.758118 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:00:37.758352 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.758185 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" podUID="40d2bf70-21e4-498f-ac76-23246675e3bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:00:37.773311 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.773278 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-cff4fd7fd-dgj5x" Oct 31 13:00:37.898379 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.898199 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:37.990136 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:37.990083 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-875df4b95-k59g5"] Oct 31 13:00:38.117146 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:38.117116 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:38.496404 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:00:38.496366 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c41d9c_82e3_440b_82fd_dbe2e5ab8bd9.slice/crio-277c7644d48379121f6a97ae942f771e7f37986cc09f89bf1f991419807d0147 WatchSource:0}: Error finding container 277c7644d48379121f6a97ae942f771e7f37986cc09f89bf1f991419807d0147: Status 404 returned error can't find the container with id 277c7644d48379121f6a97ae942f771e7f37986cc09f89bf1f991419807d0147 Oct 31 13:00:38.504070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:38.504041 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ccf963-cff4fd7fd-dgj5x"] Oct 31 13:00:38.594516 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:38.594470 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-875df4b95-k59g5" event={"ID":"399ac99e-25a1-44d3-b774-04991fbe4f47","Type":"ContainerStarted","Data":"c9a6b77cb9469d3e827f1ae69f0dd1b7b58922c973b2c6f879ddc8d4f4115e4b"} Oct 31 13:00:38.598817 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:38.598735 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-cff4fd7fd-dgj5x" event={"ID":"e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9","Type":"ContainerStarted","Data":"277c7644d48379121f6a97ae942f771e7f37986cc09f89bf1f991419807d0147"} Oct 31 13:00:38.976773 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:38.976736 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:39.514905 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:39.514681 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:39.662179 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:39.659460 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf16598d-b76a-4dda-aba1-b801582fad71" path="/var/lib/kubelet/pods/bf16598d-b76a-4dda-aba1-b801582fad71/volumes" Oct 31 13:00:39.740837 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:39.740738 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:39.823226 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:39.823130 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:00:39.823226 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:39.823201 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" podUID="ebcb215e-7f84-4653-96e6-3a13c449be73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:00:40.221414 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:40.221380 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:40.635262 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:40.634380 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-875df4b95-k59g5" event={"ID":"399ac99e-25a1-44d3-b774-04991fbe4f47","Type":"ContainerStarted","Data":"21a16ed537f35e4190ec4314338b709b228a3c074a92bd8f82e3767ae1c03261"} Oct 31 13:00:40.635262 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:40.634607 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-875df4b95-k59g5" Oct 31 13:00:40.639309 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:40.639273 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-5d88b777bb-dnwdj" event={"ID":"e3fee6c8-5e84-4011-a626-7a3da7239b06","Type":"ContainerStarted","Data":"489586f21464e62c9f1a33658064165dcad7af3027cd37708d8393c9ea65e31c"} Oct 31 13:00:40.640155 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:40.640127 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-5d88b777bb-dnwdj" Oct 31 13:00:40.748004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:40.747332 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-5d88b777bb-dnwdj" podStartSLOduration=3.79779686 podStartE2EDuration="6.747312132s" totalImagesPullingTime="2.949515272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:40.740487771 +0000 UTC m=+1687.223610671" watchObservedRunningTime="2026-10-31 13:00:40.747312132 +0000 UTC m=+1687.230435035" Oct 31 13:00:40.748004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:40.747452 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-875df4b95-k59g5" podStartSLOduration=1.6002854260000001 podStartE2EDuration="3.747443883s" totalImagesPullingTime="2.147158457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:40.684664869 +0000 UTC m=+1687.167787770" watchObservedRunningTime="2026-10-31 13:00:40.747443883 +0000 UTC m=+1687.230566786" Oct 31 13:00:40.831971 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:40.831940 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:40.847660 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:40.847636 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-958f54b4c-d8bt2"] Oct 31 13:00:40.879295 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:40.879270 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-958f54b4c-d8bt2" Oct 31 13:00:40.883098 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:40.882048 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-958f54b4c-d8bt2"] Oct 31 13:00:40.891735 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:40.891690 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-adfd11\"" Oct 31 13:00:41.049021 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:41.018433 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-adfd11-dockercfg-t2zxm\"" Oct 31 13:00:41.187897 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:41.187815 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-adfd11-958f54b4c-d8bt2\" (UID: \"fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-958f54b4c-d8bt2" Oct 31 13:00:41.201067 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:41.188129 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfzq8\" (UniqueName: \"kubernetes.io/projected/fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3-kube-api-access-gfzq8\") pod \"payload-pre-processing-e2e-ipp-a-adfd11-958f54b4c-d8bt2\" (UID: \"fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-958f54b4c-d8bt2" Oct 31 13:00:41.201609 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:41.201575 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:41.307510 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:41.299056 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-adfd11-958f54b4c-d8bt2\" (UID: \"fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-958f54b4c-d8bt2" Oct 31 13:00:41.307510 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:41.299120 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfzq8\" (UniqueName: \"kubernetes.io/projected/fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3-kube-api-access-gfzq8\") pod \"payload-pre-processing-e2e-ipp-a-adfd11-958f54b4c-d8bt2\" (UID: \"fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-958f54b4c-d8bt2" Oct 31 13:00:41.307510 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:41.300554 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-adfd11-958f54b4c-d8bt2\" (UID: \"fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-958f54b4c-d8bt2" Oct 31 13:00:41.337085 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:41.337027 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfzq8\" (UniqueName: \"kubernetes.io/projected/fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3-kube-api-access-gfzq8\") pod \"payload-pre-processing-e2e-ipp-a-adfd11-958f54b4c-d8bt2\" (UID: \"fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-958f54b4c-d8bt2" Oct 31 13:00:41.538006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:41.537301 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-958f54b4c-d8bt2" Oct 31 13:00:41.645121 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:41.642485 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:41.676861 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:41.675574 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-cff4fd7fd-dgj5x" event={"ID":"e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9","Type":"ContainerStarted","Data":"75ac6696050ce370e2cc573e1ff405d72e03cd5e5156be3f54f57bca703dbb51"} Oct 31 13:00:41.676861 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:41.676225 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-cff4fd7fd-dgj5x" Oct 31 13:00:41.711422 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:41.711367 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-cff4fd7fd-dgj5x" podStartSLOduration=2.489256973 podStartE2EDuration="4.711344965s" totalImagesPullingTime="2.222087992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:41.707855558 +0000 UTC m=+1688.190978472" watchObservedRunningTime="2026-10-31 13:00:41.711344965 +0000 UTC m=+1688.194467867" Oct 31 13:00:41.994008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:41.984256 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-958f54b4c-d8bt2"] Oct 31 13:00:42.624045 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:42.620936 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:42.686358 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:42.686309 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-958f54b4c-d8bt2" event={"ID":"fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3","Type":"ContainerStarted","Data":"4c5d9c146f3aad611eaa216d5c0969f05e3e19e84a0fc1061c5b0a9a74d4ee9e"} Oct 31 13:00:42.686896 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:42.686388 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-958f54b4c-d8bt2" event={"ID":"fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3","Type":"ContainerStarted","Data":"2e83780fca5071cb2c424f1f1154249e16931ae3b48f19ce8aa067b7c6bc8718"} Oct 31 13:00:42.686896 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:42.686422 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-958f54b4c-d8bt2" Oct 31 13:00:42.708755 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:42.708706 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-958f54b4c-d8bt2" podStartSLOduration=2.708693781 podStartE2EDuration="2.708693781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:42.705226944 +0000 UTC m=+1689.188349846" watchObservedRunningTime="2026-10-31 13:00:42.708693781 +0000 UTC m=+1689.191816681" Oct 31 13:00:42.908330 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:42.908184 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:43.266113 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:43.266079 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr"] Oct 31 13:00:43.286038 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:43.286010 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:00:43.319376 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:43.319168 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-dc2db4-openshift-default-dockercfg-bmdxk\"" Oct 31 13:00:43.323633 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:43.323607 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr"] Oct 31 13:00:43.427016 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:43.426131 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c502702-da0b-45d4-ba77-5ae972c1bd16-istiod-ca-cert\") pod \"e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr\" (UID: \"1c502702-da0b-45d4-ba77-5ae972c1bd16\") " pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:00:43.427016 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:43.426209 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c502702-da0b-45d4-ba77-5ae972c1bd16-istio-token\") pod \"e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr\" (UID: \"1c502702-da0b-45d4-ba77-5ae972c1bd16\") " pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:00:43.427016 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:43.426260 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mzx6\" (UniqueName: \"kubernetes.io/projected/1c502702-da0b-45d4-ba77-5ae972c1bd16-kube-api-access-4mzx6\") pod \"e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr\" (UID: \"1c502702-da0b-45d4-ba77-5ae972c1bd16\") " pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:00:43.427016 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:43.426293 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c502702-da0b-45d4-ba77-5ae972c1bd16-istio-data\") pod \"e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr\" (UID: \"1c502702-da0b-45d4-ba77-5ae972c1bd16\") " pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:00:43.427016 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:43.426372 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c502702-da0b-45d4-ba77-5ae972c1bd16-workload-certs\") pod \"e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr\" (UID: \"1c502702-da0b-45d4-ba77-5ae972c1bd16\") " pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:00:43.427016 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:43.426410 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c502702-da0b-45d4-ba77-5ae972c1bd16-workload-socket\") pod \"e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr\" (UID: \"1c502702-da0b-45d4-ba77-5ae972c1bd16\") " pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:00:43.427016 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:43.426538 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c502702-da0b-45d4-ba77-5ae972c1bd16-istio-envoy\") pod \"e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr\" (UID: \"1c502702-da0b-45d4-ba77-5ae972c1bd16\") " pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:00:43.427016 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:43.426590 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c502702-da0b-45d4-ba77-5ae972c1bd16-credential-socket\") pod \"e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr\" (UID: \"1c502702-da0b-45d4-ba77-5ae972c1bd16\") " pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:00:43.427016 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:43.426615 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c502702-da0b-45d4-ba77-5ae972c1bd16-istio-podinfo\") pod \"e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr\" (UID: \"1c502702-da0b-45d4-ba77-5ae972c1bd16\") " pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:00:43.465362 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:43.465255 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:43.571056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:43.570178 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c502702-da0b-45d4-ba77-5ae972c1bd16-credential-socket\") pod \"e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr\" (UID: \"1c502702-da0b-45d4-ba77-5ae972c1bd16\") " pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:00:43.571056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:43.570233 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c502702-da0b-45d4-ba77-5ae972c1bd16-istio-podinfo\") pod \"e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr\" (UID: \"1c502702-da0b-45d4-ba77-5ae972c1bd16\") " pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:00:43.571056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:43.570406 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c502702-da0b-45d4-ba77-5ae972c1bd16-istiod-ca-cert\") pod \"e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr\" (UID: \"1c502702-da0b-45d4-ba77-5ae972c1bd16\") " pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:00:43.571056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:43.570447 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c502702-da0b-45d4-ba77-5ae972c1bd16-istio-token\") pod \"e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr\" (UID: \"1c502702-da0b-45d4-ba77-5ae972c1bd16\") " pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:00:43.571056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:43.570491 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mzx6\" (UniqueName: \"kubernetes.io/projected/1c502702-da0b-45d4-ba77-5ae972c1bd16-kube-api-access-4mzx6\") pod \"e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr\" (UID: \"1c502702-da0b-45d4-ba77-5ae972c1bd16\") " pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:00:43.571056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:43.570520 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c502702-da0b-45d4-ba77-5ae972c1bd16-istio-data\") pod \"e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr\" (UID: \"1c502702-da0b-45d4-ba77-5ae972c1bd16\") " pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:00:43.571056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:43.570595 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c502702-da0b-45d4-ba77-5ae972c1bd16-workload-certs\") pod \"e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr\" (UID: \"1c502702-da0b-45d4-ba77-5ae972c1bd16\") " pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:00:43.571056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:43.570629 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c502702-da0b-45d4-ba77-5ae972c1bd16-workload-socket\") pod \"e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr\" (UID: \"1c502702-da0b-45d4-ba77-5ae972c1bd16\") " pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:00:43.571056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:43.570756 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c502702-da0b-45d4-ba77-5ae972c1bd16-istio-envoy\") pod \"e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr\" (UID: \"1c502702-da0b-45d4-ba77-5ae972c1bd16\") " pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:00:43.571668 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:43.571556 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c502702-da0b-45d4-ba77-5ae972c1bd16-istio-data\") pod \"e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr\" (UID: \"1c502702-da0b-45d4-ba77-5ae972c1bd16\") " pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:00:43.576002 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:43.572126 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c502702-da0b-45d4-ba77-5ae972c1bd16-workload-certs\") pod \"e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr\" (UID: \"1c502702-da0b-45d4-ba77-5ae972c1bd16\") " pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:00:43.577646 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:43.577311 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c502702-da0b-45d4-ba77-5ae972c1bd16-workload-socket\") pod \"e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr\" (UID: \"1c502702-da0b-45d4-ba77-5ae972c1bd16\") " pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:00:43.577646 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:43.577559 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c502702-da0b-45d4-ba77-5ae972c1bd16-credential-socket\") pod \"e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr\" (UID: \"1c502702-da0b-45d4-ba77-5ae972c1bd16\") " pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:00:43.591006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:43.586021 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c502702-da0b-45d4-ba77-5ae972c1bd16-istiod-ca-cert\") pod \"e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr\" (UID: \"1c502702-da0b-45d4-ba77-5ae972c1bd16\") " pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:00:43.638003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:43.633625 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mzx6\" (UniqueName: \"kubernetes.io/projected/1c502702-da0b-45d4-ba77-5ae972c1bd16-kube-api-access-4mzx6\") pod \"e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr\" (UID: \"1c502702-da0b-45d4-ba77-5ae972c1bd16\") " pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:00:43.638003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:43.634384 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c502702-da0b-45d4-ba77-5ae972c1bd16-istio-podinfo\") pod \"e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr\" (UID: \"1c502702-da0b-45d4-ba77-5ae972c1bd16\") " pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:00:43.679059 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:43.646843 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c502702-da0b-45d4-ba77-5ae972c1bd16-istio-envoy\") pod \"e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr\" (UID: \"1c502702-da0b-45d4-ba77-5ae972c1bd16\") " pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:00:43.689009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:43.680856 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c502702-da0b-45d4-ba77-5ae972c1bd16-istio-token\") pod \"e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr\" (UID: \"1c502702-da0b-45d4-ba77-5ae972c1bd16\") " pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:00:43.764842 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:43.764800 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:43.921546 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:43.921457 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:00:44.249115 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.249083 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr"] Oct 31 13:00:44.249398 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:00:44.249376 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c502702_da0b_45d4_ba77_5ae972c1bd16.slice/crio-45129d45580b3e372b04db17c9aa512dbe3ad544836a2af65251543bde9e1943 WatchSource:0}: Error finding container 45129d45580b3e372b04db17c9aa512dbe3ad544836a2af65251543bde9e1943: Status 404 returned error can't find the container with id 45129d45580b3e372b04db17c9aa512dbe3ad544836a2af65251543bde9e1943 Oct 31 13:00:44.252536 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.252496 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:00:44.252645 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.252580 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:00:44.252645 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.252625 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:00:44.463264 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.463227 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9"] Oct 31 13:00:44.467478 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.467453 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" Oct 31 13:00:44.500103 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.496184 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e512c4f-871f-49c6-8114-13073e883e1a-credential-socket\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9\" (UID: \"6e512c4f-871f-49c6-8114-13073e883e1a\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" Oct 31 13:00:44.500103 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.496270 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e512c4f-871f-49c6-8114-13073e883e1a-istio-token\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9\" (UID: \"6e512c4f-871f-49c6-8114-13073e883e1a\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" Oct 31 13:00:44.500103 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.496368 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e512c4f-871f-49c6-8114-13073e883e1a-workload-socket\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9\" (UID: \"6e512c4f-871f-49c6-8114-13073e883e1a\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" Oct 31 13:00:44.500103 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.496437 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz5b4\" (UniqueName: \"kubernetes.io/projected/6e512c4f-871f-49c6-8114-13073e883e1a-kube-api-access-vz5b4\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9\" (UID: \"6e512c4f-871f-49c6-8114-13073e883e1a\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" Oct 31 13:00:44.500103 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.496481 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e512c4f-871f-49c6-8114-13073e883e1a-istio-data\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9\" (UID: \"6e512c4f-871f-49c6-8114-13073e883e1a\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" Oct 31 13:00:44.500103 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.496512 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e512c4f-871f-49c6-8114-13073e883e1a-istio-envoy\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9\" (UID: \"6e512c4f-871f-49c6-8114-13073e883e1a\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" Oct 31 13:00:44.500103 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.496532 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e512c4f-871f-49c6-8114-13073e883e1a-istio-podinfo\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9\" (UID: \"6e512c4f-871f-49c6-8114-13073e883e1a\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" Oct 31 13:00:44.500103 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.496570 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e512c4f-871f-49c6-8114-13073e883e1a-istiod-ca-cert\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9\" (UID: \"6e512c4f-871f-49c6-8114-13073e883e1a\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" Oct 31 13:00:44.500103 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.496606 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e512c4f-871f-49c6-8114-13073e883e1a-workload-certs\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9\" (UID: \"6e512c4f-871f-49c6-8114-13073e883e1a\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" Oct 31 13:00:44.560010 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.550517 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9"] Oct 31 13:00:44.632161 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.630963 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e512c4f-871f-49c6-8114-13073e883e1a-credential-socket\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9\" (UID: \"6e512c4f-871f-49c6-8114-13073e883e1a\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" Oct 31 13:00:44.632161 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.631066 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e512c4f-871f-49c6-8114-13073e883e1a-istio-token\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9\" (UID: \"6e512c4f-871f-49c6-8114-13073e883e1a\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" Oct 31 13:00:44.632161 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.631187 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e512c4f-871f-49c6-8114-13073e883e1a-workload-socket\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9\" (UID: \"6e512c4f-871f-49c6-8114-13073e883e1a\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" Oct 31 13:00:44.632161 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.631273 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz5b4\" (UniqueName: \"kubernetes.io/projected/6e512c4f-871f-49c6-8114-13073e883e1a-kube-api-access-vz5b4\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9\" (UID: \"6e512c4f-871f-49c6-8114-13073e883e1a\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" Oct 31 13:00:44.632161 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.631328 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e512c4f-871f-49c6-8114-13073e883e1a-istio-data\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9\" (UID: \"6e512c4f-871f-49c6-8114-13073e883e1a\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" Oct 31 13:00:44.632161 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.631363 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e512c4f-871f-49c6-8114-13073e883e1a-istio-envoy\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9\" (UID: \"6e512c4f-871f-49c6-8114-13073e883e1a\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" Oct 31 13:00:44.632161 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.631387 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e512c4f-871f-49c6-8114-13073e883e1a-istio-podinfo\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9\" (UID: \"6e512c4f-871f-49c6-8114-13073e883e1a\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" Oct 31 13:00:44.632161 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.631470 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e512c4f-871f-49c6-8114-13073e883e1a-istiod-ca-cert\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9\" (UID: \"6e512c4f-871f-49c6-8114-13073e883e1a\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" Oct 31 13:00:44.632161 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.631513 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e512c4f-871f-49c6-8114-13073e883e1a-workload-certs\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9\" (UID: \"6e512c4f-871f-49c6-8114-13073e883e1a\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" Oct 31 13:00:44.632161 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.632139 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e512c4f-871f-49c6-8114-13073e883e1a-workload-certs\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9\" (UID: \"6e512c4f-871f-49c6-8114-13073e883e1a\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" Oct 31 13:00:44.634373 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.632382 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e512c4f-871f-49c6-8114-13073e883e1a-credential-socket\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9\" (UID: \"6e512c4f-871f-49c6-8114-13073e883e1a\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" Oct 31 13:00:44.634373 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.633032 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e512c4f-871f-49c6-8114-13073e883e1a-workload-socket\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9\" (UID: \"6e512c4f-871f-49c6-8114-13073e883e1a\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" Oct 31 13:00:44.634373 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.633498 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e512c4f-871f-49c6-8114-13073e883e1a-istio-data\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9\" (UID: \"6e512c4f-871f-49c6-8114-13073e883e1a\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" Oct 31 13:00:44.635725 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.635693 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e512c4f-871f-49c6-8114-13073e883e1a-istiod-ca-cert\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9\" (UID: \"6e512c4f-871f-49c6-8114-13073e883e1a\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" Oct 31 13:00:44.637934 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.637910 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e512c4f-871f-49c6-8114-13073e883e1a-istio-envoy\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9\" (UID: \"6e512c4f-871f-49c6-8114-13073e883e1a\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" Oct 31 13:00:44.640306 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.640282 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e512c4f-871f-49c6-8114-13073e883e1a-istio-podinfo\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9\" (UID: \"6e512c4f-871f-49c6-8114-13073e883e1a\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" Oct 31 13:00:44.645841 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.645800 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e512c4f-871f-49c6-8114-13073e883e1a-istio-token\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9\" (UID: \"6e512c4f-871f-49c6-8114-13073e883e1a\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" Oct 31 13:00:44.646963 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.646924 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz5b4\" (UniqueName: \"kubernetes.io/projected/6e512c4f-871f-49c6-8114-13073e883e1a-kube-api-access-vz5b4\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9\" (UID: \"6e512c4f-871f-49c6-8114-13073e883e1a\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" Oct 31 13:00:44.655257 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.655230 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:44.719616 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.719588 6841 generic.go:356] "Generic (PLEG): container finished" podID="40d2bf70-21e4-498f-ac76-23246675e3bf" containerID="1a09d688c41bd1d2cbce3a2c6f137d38aa0ff6b46ca662008a8325740ec7dacc" exitCode=0 Oct 31 13:00:44.720083 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.719662 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" event={"ID":"40d2bf70-21e4-498f-ac76-23246675e3bf","Type":"ContainerDied","Data":"1a09d688c41bd1d2cbce3a2c6f137d38aa0ff6b46ca662008a8325740ec7dacc"} Oct 31 13:00:44.720248 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.720228 6841 scope.go:122] "RemoveContainer" containerID="1a09d688c41bd1d2cbce3a2c6f137d38aa0ff6b46ca662008a8325740ec7dacc" Oct 31 13:00:44.721791 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.721768 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" event={"ID":"1c502702-da0b-45d4-ba77-5ae972c1bd16","Type":"ContainerStarted","Data":"062d41d9f05c95d634be86d75b50e597350d8238c78dac954f601aaccfaf0996"} Oct 31 13:00:44.721954 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.721818 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" event={"ID":"1c502702-da0b-45d4-ba77-5ae972c1bd16","Type":"ContainerStarted","Data":"45129d45580b3e372b04db17c9aa512dbe3ad544836a2af65251543bde9e1943"} Oct 31 13:00:44.722196 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.722161 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:00:44.722289 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.722260 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:00:44.722361 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.722309 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:00:44.724435 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.724414 6841 generic.go:356] "Generic (PLEG): container finished" podID="ebcb215e-7f84-4653-96e6-3a13c449be73" containerID="f86a3b89d8a0a8abe80b4cab3de2a83e592196d5d72d48535306d7013fb4da73" exitCode=0 Oct 31 13:00:44.724535 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.724484 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" event={"ID":"ebcb215e-7f84-4653-96e6-3a13c449be73","Type":"ContainerDied","Data":"f86a3b89d8a0a8abe80b4cab3de2a83e592196d5d72d48535306d7013fb4da73"} Oct 31 13:00:44.724931 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.724911 6841 scope.go:122] "RemoveContainer" containerID="f86a3b89d8a0a8abe80b4cab3de2a83e592196d5d72d48535306d7013fb4da73" Oct 31 13:00:44.726499 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.726466 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:00:44.726599 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.726533 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:00:44.726599 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.726553 6841 generic.go:356] "Generic (PLEG): container finished" podID="35d138be-ff3b-43d9-ae6b-bae298aaec6b" containerID="b37e75d6f82bc614102b7d3ac3f705c19eb68f5a289554cd5f76254127cfff9c" exitCode=0 Oct 31 13:00:44.726599 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.726566 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:00:44.726756 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.726604 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" event={"ID":"35d138be-ff3b-43d9-ae6b-bae298aaec6b","Type":"ContainerDied","Data":"b37e75d6f82bc614102b7d3ac3f705c19eb68f5a289554cd5f76254127cfff9c"} Oct 31 13:00:44.727051 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.726920 6841 scope.go:122] "RemoveContainer" containerID="b37e75d6f82bc614102b7d3ac3f705c19eb68f5a289554cd5f76254127cfff9c" Oct 31 13:00:44.728252 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.728228 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:00:44.728355 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.728281 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:00:44.728355 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.728320 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:00:44.838574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.838545 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" Oct 31 13:00:44.921756 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:44.921574 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:00:45.208997 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:45.208923 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" podStartSLOduration=2.208891848 podStartE2EDuration="2.208891848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:44.871335929 +0000 UTC m=+1691.354458830" watchObservedRunningTime="2026-10-31 13:00:45.208891848 +0000 UTC m=+1691.692014750" Oct 31 13:00:45.222120 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:45.222093 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9"] Oct 31 13:00:45.240381 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:45.240336 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:00:45.241011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:45.240960 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:00:45.241200 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:45.241178 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:00:45.360004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:45.358849 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56a83a-7d476bc98c-mxvqr" Oct 31 13:00:45.360004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:45.359290 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-56a83a-7865d4d4f7-ggdfx" Oct 31 13:00:45.735015 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:45.734550 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:45.754015 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:45.750522 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" event={"ID":"40d2bf70-21e4-498f-ac76-23246675e3bf","Type":"ContainerStarted","Data":"e353905af1081da1a8522088240c7514e5ae6e2633109e62d7dbad60a8ccb3c5"} Oct 31 13:00:45.764897 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:45.764864 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" event={"ID":"ebcb215e-7f84-4653-96e6-3a13c449be73","Type":"ContainerStarted","Data":"54c965bd8bde25b4e49d2af75f2e8bfdd691ab7c0dbf680572bf5e32ca966e7e"} Oct 31 13:00:45.769107 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:45.769075 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" event={"ID":"35d138be-ff3b-43d9-ae6b-bae298aaec6b","Type":"ContainerStarted","Data":"535a8c90bfaf4cf2f66bfc858f198361bdf1911ba87c30a088d5522f570df7f3"} Oct 31 13:00:45.774229 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:45.774193 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" event={"ID":"6e512c4f-871f-49c6-8114-13073e883e1a","Type":"ContainerStarted","Data":"7f91f31b7a034a1727465e671ddf923eeeeef2bb6031b640b7285dfe2e6195c2"} Oct 31 13:00:45.774438 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:45.774249 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" event={"ID":"6e512c4f-871f-49c6-8114-13073e883e1a","Type":"ContainerStarted","Data":"c485ed3572bf7d6ac3cc06a01dfc21b9c7b124014e1052753838705e7091843c"} Oct 31 13:00:45.822387 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:45.820142 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" podStartSLOduration=1.8201211160000001 podStartE2EDuration="1.820121116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:45.815561665 +0000 UTC m=+1692.298684596" watchObservedRunningTime="2026-10-31 13:00:45.820121116 +0000 UTC m=+1692.303244032" Oct 31 13:00:45.846248 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:45.846203 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" Oct 31 13:00:45.859024 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:45.857111 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:45.859024 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:45.857247 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 31 13:00:45.859024 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:45.857288 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" podUID="40d2bf70-21e4-498f-ac76-23246675e3bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 31 13:00:45.888230 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:45.888193 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 13:00:45.889232 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:45.889197 6841 patch_prober.go:29] interesting pod/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 13:00:45.889381 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:45.889257 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" podUID="35d138be-ff3b-43d9-ae6b-bae298aaec6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 13:00:45.928370 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:45.928339 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:00:46.762162 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:46.762048 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:46.766510 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:46.766107 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 31 13:00:46.766510 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:46.766189 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" podUID="ebcb215e-7f84-4653-96e6-3a13c449be73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 31 13:00:46.862007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:46.861038 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 31 13:00:46.862007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:46.861113 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" podUID="40d2bf70-21e4-498f-ac76-23246675e3bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 31 13:00:46.885009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:46.883685 6841 patch_prober.go:29] interesting pod/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 31 13:00:46.885009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:46.883758 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" podUID="6e512c4f-871f-49c6-8114-13073e883e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 31 13:00:46.908006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:46.898830 6841 patch_prober.go:29] interesting pod/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 13:00:46.908006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:46.898888 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" podUID="35d138be-ff3b-43d9-ae6b-bae298aaec6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 13:00:46.955167 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:46.955134 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:00:46.995583 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:46.995242 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:00:47.369129 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:47.366089 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 13:00:47.395671 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:47.395586 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:47.680204 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:47.671854 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:47.760010 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:47.758671 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 31 13:00:47.760010 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:47.758742 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" podUID="ebcb215e-7f84-4653-96e6-3a13c449be73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 31 13:00:47.845456 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:47.845421 6841 patch_prober.go:29] interesting pod/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 31 13:00:47.846369 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:47.846338 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" podUID="6e512c4f-871f-49c6-8114-13073e883e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 31 13:00:47.854735 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:47.854712 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 31 13:00:47.854916 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:47.854897 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" podUID="40d2bf70-21e4-498f-ac76-23246675e3bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 31 13:00:47.893032 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:47.892162 6841 patch_prober.go:29] interesting pod/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 13:00:47.893032 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:47.892222 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" podUID="35d138be-ff3b-43d9-ae6b-bae298aaec6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 13:00:48.254597 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:48.254559 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:48.779763 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:48.776547 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 31 13:00:48.779763 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:48.776606 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" podUID="ebcb215e-7f84-4653-96e6-3a13c449be73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 31 13:00:48.782713 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:48.782681 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:48.915289 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:48.880367 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 31 13:00:48.935277 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:48.935221 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" podUID="40d2bf70-21e4-498f-ac76-23246675e3bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 31 13:00:48.953462 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:48.881817 6841 patch_prober.go:29] interesting pod/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 31 13:00:49.013120 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:49.009539 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" podUID="6e512c4f-871f-49c6-8114-13073e883e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 31 13:00:49.123950 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:49.094119 6841 patch_prober.go:29] interesting pod/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 13:00:49.123950 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:49.094185 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" podUID="35d138be-ff3b-43d9-ae6b-bae298aaec6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 13:00:49.310899 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:49.310848 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:49.761269 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:49.761213 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 31 13:00:49.761571 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:49.761536 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" podUID="ebcb215e-7f84-4653-96e6-3a13c449be73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 31 13:00:49.842266 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:49.842229 6841 patch_prober.go:29] interesting pod/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 31 13:00:49.842533 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:49.842506 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" podUID="6e512c4f-871f-49c6-8114-13073e883e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 31 13:00:49.856647 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:49.856609 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 31 13:00:49.856881 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:49.856852 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" podUID="40d2bf70-21e4-498f-ac76-23246675e3bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 31 13:00:49.893083 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:49.893046 6841 patch_prober.go:29] interesting pod/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 13:00:49.893384 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:49.893353 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" podUID="35d138be-ff3b-43d9-ae6b-bae298aaec6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 13:00:49.972924 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:49.972880 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:50.206008 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:00:50.204704 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:00:50.774140 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:50.774099 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 31 13:00:50.774446 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:50.774416 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" podUID="ebcb215e-7f84-4653-96e6-3a13c449be73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 31 13:00:50.846410 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:50.846373 6841 patch_prober.go:29] interesting pod/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 31 13:00:50.846720 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:50.846684 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" podUID="6e512c4f-871f-49c6-8114-13073e883e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 31 13:00:50.857387 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:50.857357 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 31 13:00:50.857642 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:50.857618 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" podUID="40d2bf70-21e4-498f-ac76-23246675e3bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 31 13:00:50.888387 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:50.888349 6841 patch_prober.go:29] interesting pod/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 13:00:50.888671 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:50.888640 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" podUID="35d138be-ff3b-43d9-ae6b-bae298aaec6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 13:00:51.215007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:51.207151 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:51.457334 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:51.457297 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:51.715007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:51.701173 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-875df4b95-k59g5" Oct 31 13:00:51.719001 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:51.718869 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:51.784030 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:51.779546 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 31 13:00:51.784030 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:51.779610 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" podUID="ebcb215e-7f84-4653-96e6-3a13c449be73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 31 13:00:51.919033 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:51.916480 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 31 13:00:51.919033 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:51.916602 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" podUID="40d2bf70-21e4-498f-ac76-23246675e3bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 31 13:00:51.928143 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:51.920207 6841 patch_prober.go:29] interesting pod/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 31 13:00:51.928143 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:51.920467 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" podUID="6e512c4f-871f-49c6-8114-13073e883e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 31 13:00:52.041089 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:52.033059 6841 patch_prober.go:29] interesting pod/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 13:00:52.041089 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:52.033155 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" podUID="35d138be-ff3b-43d9-ae6b-bae298aaec6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 13:00:52.290171 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:52.290120 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" podUID="da83bd57-52af-48d3-bd76-42e36c18112f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:00:52.675762 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:52.675727 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:52.695118 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:52.688145 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-5d88b777bb-dnwdj" Oct 31 13:00:52.956067 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:52.931456 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 31 13:00:52.957192 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:52.957152 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" podUID="40d2bf70-21e4-498f-ac76-23246675e3bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 31 13:00:52.961008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:52.949733 6841 patch_prober.go:29] interesting pod/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 31 13:00:52.961205 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:52.961177 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" podUID="6e512c4f-871f-49c6-8114-13073e883e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 31 13:00:52.965063 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:52.949801 6841 patch_prober.go:29] interesting pod/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 13:00:52.965227 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:52.965205 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" podUID="35d138be-ff3b-43d9-ae6b-bae298aaec6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 13:00:52.965342 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:52.949839 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 31 13:00:52.965464 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:52.965429 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" podUID="ebcb215e-7f84-4653-96e6-3a13c449be73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 31 13:00:53.642027 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:53.637418 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:53.731283 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:53.731097 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-cff4fd7fd-dgj5x" Oct 31 13:00:53.731283 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:53.731184 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-958f54b4c-d8bt2" Oct 31 13:00:53.880030 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:53.853579 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:53.880030 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:53.855724 6841 patch_prober.go:29] interesting pod/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 31 13:00:53.880030 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:53.855788 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" podUID="6e512c4f-871f-49c6-8114-13073e883e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 31 13:00:53.880030 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:53.856232 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 31 13:00:53.880030 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:53.856268 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" podUID="ebcb215e-7f84-4653-96e6-3a13c449be73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 31 13:00:53.880030 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:53.857004 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 31 13:00:53.880030 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:53.857038 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" podUID="40d2bf70-21e4-498f-ac76-23246675e3bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 31 13:00:53.917736 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:53.892207 6841 patch_prober.go:29] interesting pod/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 13:00:53.917736 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:53.892263 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" podUID="35d138be-ff3b-43d9-ae6b-bae298aaec6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 13:00:54.007004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:54.003493 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:54.475016 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:54.458056 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:54.780864 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:54.780767 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:54.818003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:54.808136 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 31 13:00:54.818003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:54.808215 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" podUID="ebcb215e-7f84-4653-96e6-3a13c449be73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 31 13:00:54.860038 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:54.845126 6841 patch_prober.go:29] interesting pod/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 31 13:00:54.860038 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:54.845202 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" podUID="6e512c4f-871f-49c6-8114-13073e883e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 31 13:00:54.865135 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:54.865102 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 31 13:00:54.865464 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:54.865425 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" podUID="40d2bf70-21e4-498f-ac76-23246675e3bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 31 13:00:54.881133 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:54.881102 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj"] Oct 31 13:00:54.954446 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:54.954409 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:00:54.956099 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:54.956054 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj"] Oct 31 13:00:54.956216 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:54.956194 6841 patch_prober.go:29] interesting pod/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 13:00:54.956287 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:54.956261 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" podUID="35d138be-ff3b-43d9-ae6b-bae298aaec6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 13:00:55.009006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.000773 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-3c8c51-openshift-default-dockercfg-gn78c\"" Oct 31 13:00:55.062412 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.061296 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:55.096444 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.096405 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba9a674c-9aec-46ad-9600-56952a8128c0-istiod-ca-cert\") pod \"e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj\" (UID: \"ba9a674c-9aec-46ad-9600-56952a8128c0\") " pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:00:55.096734 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.096715 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba9a674c-9aec-46ad-9600-56952a8128c0-istio-envoy\") pod \"e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj\" (UID: \"ba9a674c-9aec-46ad-9600-56952a8128c0\") " pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:00:55.096885 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.096867 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnzhn\" (UniqueName: \"kubernetes.io/projected/ba9a674c-9aec-46ad-9600-56952a8128c0-kube-api-access-pnzhn\") pod \"e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj\" (UID: \"ba9a674c-9aec-46ad-9600-56952a8128c0\") " pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:00:55.097141 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.097118 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba9a674c-9aec-46ad-9600-56952a8128c0-istio-token\") pod \"e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj\" (UID: \"ba9a674c-9aec-46ad-9600-56952a8128c0\") " pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:00:55.097448 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.097428 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba9a674c-9aec-46ad-9600-56952a8128c0-workload-socket\") pod \"e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj\" (UID: \"ba9a674c-9aec-46ad-9600-56952a8128c0\") " pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:00:55.112760 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.112728 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba9a674c-9aec-46ad-9600-56952a8128c0-istio-podinfo\") pod \"e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj\" (UID: \"ba9a674c-9aec-46ad-9600-56952a8128c0\") " pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:00:55.113044 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.113022 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba9a674c-9aec-46ad-9600-56952a8128c0-istio-data\") pod \"e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj\" (UID: \"ba9a674c-9aec-46ad-9600-56952a8128c0\") " pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:00:55.113213 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.113195 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba9a674c-9aec-46ad-9600-56952a8128c0-workload-certs\") pod \"e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj\" (UID: \"ba9a674c-9aec-46ad-9600-56952a8128c0\") " pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:00:55.113379 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.113362 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba9a674c-9aec-46ad-9600-56952a8128c0-credential-socket\") pod \"e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj\" (UID: \"ba9a674c-9aec-46ad-9600-56952a8128c0\") " pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:00:55.218025 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.217040 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba9a674c-9aec-46ad-9600-56952a8128c0-workload-socket\") pod \"e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj\" (UID: \"ba9a674c-9aec-46ad-9600-56952a8128c0\") " pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:00:55.218025 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.217128 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba9a674c-9aec-46ad-9600-56952a8128c0-istio-podinfo\") pod \"e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj\" (UID: \"ba9a674c-9aec-46ad-9600-56952a8128c0\") " pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:00:55.218025 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.217180 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba9a674c-9aec-46ad-9600-56952a8128c0-istio-data\") pod \"e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj\" (UID: \"ba9a674c-9aec-46ad-9600-56952a8128c0\") " pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:00:55.218025 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.217226 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba9a674c-9aec-46ad-9600-56952a8128c0-workload-certs\") pod \"e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj\" (UID: \"ba9a674c-9aec-46ad-9600-56952a8128c0\") " pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:00:55.218025 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.217281 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba9a674c-9aec-46ad-9600-56952a8128c0-credential-socket\") pod \"e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj\" (UID: \"ba9a674c-9aec-46ad-9600-56952a8128c0\") " pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:00:55.218025 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.217434 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba9a674c-9aec-46ad-9600-56952a8128c0-istiod-ca-cert\") pod \"e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj\" (UID: \"ba9a674c-9aec-46ad-9600-56952a8128c0\") " pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:00:55.218025 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.217515 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba9a674c-9aec-46ad-9600-56952a8128c0-istio-envoy\") pod \"e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj\" (UID: \"ba9a674c-9aec-46ad-9600-56952a8128c0\") " pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:00:55.218025 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.217566 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnzhn\" (UniqueName: \"kubernetes.io/projected/ba9a674c-9aec-46ad-9600-56952a8128c0-kube-api-access-pnzhn\") pod \"e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj\" (UID: \"ba9a674c-9aec-46ad-9600-56952a8128c0\") " pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:00:55.218025 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.217635 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba9a674c-9aec-46ad-9600-56952a8128c0-istio-token\") pod \"e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj\" (UID: \"ba9a674c-9aec-46ad-9600-56952a8128c0\") " pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:00:55.219692 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.219269 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba9a674c-9aec-46ad-9600-56952a8128c0-workload-socket\") pod \"e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj\" (UID: \"ba9a674c-9aec-46ad-9600-56952a8128c0\") " pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:00:55.223069 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.220196 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba9a674c-9aec-46ad-9600-56952a8128c0-istio-data\") pod \"e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj\" (UID: \"ba9a674c-9aec-46ad-9600-56952a8128c0\") " pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:00:55.223069 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.220931 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba9a674c-9aec-46ad-9600-56952a8128c0-workload-certs\") pod \"e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj\" (UID: \"ba9a674c-9aec-46ad-9600-56952a8128c0\") " pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:00:55.223448 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.223398 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba9a674c-9aec-46ad-9600-56952a8128c0-istiod-ca-cert\") pod \"e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj\" (UID: \"ba9a674c-9aec-46ad-9600-56952a8128c0\") " pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:00:55.223814 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.223793 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba9a674c-9aec-46ad-9600-56952a8128c0-credential-socket\") pod \"e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj\" (UID: \"ba9a674c-9aec-46ad-9600-56952a8128c0\") " pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:00:55.251993 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.251901 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba9a674c-9aec-46ad-9600-56952a8128c0-istio-podinfo\") pod \"e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj\" (UID: \"ba9a674c-9aec-46ad-9600-56952a8128c0\") " pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:00:55.260952 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.260897 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnzhn\" (UniqueName: \"kubernetes.io/projected/ba9a674c-9aec-46ad-9600-56952a8128c0-kube-api-access-pnzhn\") pod \"e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj\" (UID: \"ba9a674c-9aec-46ad-9600-56952a8128c0\") " pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:00:55.283244 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.263584 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba9a674c-9aec-46ad-9600-56952a8128c0-istio-envoy\") pod \"e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj\" (UID: \"ba9a674c-9aec-46ad-9600-56952a8128c0\") " pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:00:55.292440 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.292388 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba9a674c-9aec-46ad-9600-56952a8128c0-istio-token\") pod \"e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj\" (UID: \"ba9a674c-9aec-46ad-9600-56952a8128c0\") " pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:00:55.393823 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.393641 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:55.455669 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.452740 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:00:55.765028 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.763595 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:55.765195 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.765159 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:00:55.863398 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.863366 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:55.864758 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.864690 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:00:55.883048 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.882740 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" Oct 31 13:00:55.883916 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.883761 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" Oct 31 13:00:55.885227 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.884767 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" Oct 31 13:00:55.898847 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.896566 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 13:00:55.898847 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:55.898391 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 13:00:56.011087 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:56.011044 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:00:56.011210 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:56.011161 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:00:56.011268 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:56.011208 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:00:56.026165 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:56.026139 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj"] Oct 31 13:00:56.409159 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:56.406017 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:56.665377 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:56.665288 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:57.020547 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:56.967082 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" event={"ID":"ba9a674c-9aec-46ad-9600-56952a8128c0","Type":"ContainerStarted","Data":"c94e46db8065457675b759a07fa1b7b20fc687e393dd3c812bd1dab9da8264bd"} Oct 31 13:00:57.020547 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:56.967229 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" event={"ID":"ba9a674c-9aec-46ad-9600-56952a8128c0","Type":"ContainerStarted","Data":"96546c1ecac358b406fc03d5309147c5f83157ef7ebbb95b9b70b62e662488e7"} Oct 31 13:00:57.023026 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:57.022929 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:57.150018 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:57.149292 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" podStartSLOduration=3.149268311 podStartE2EDuration="3.149268311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:57.096337872 +0000 UTC m=+1703.579460775" watchObservedRunningTime="2026-10-31 13:00:57.149268311 +0000 UTC m=+1703.632391204" Oct 31 13:00:57.386080 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:57.378584 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:57.457326 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:57.457297 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:00:57.462779 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:57.462752 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:00:57.702875 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:57.702835 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:57.974327 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:57.974244 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:00:57.975258 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:57.975238 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:00:58.919373 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:58.919336 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:00:59.741288 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:00:59.741252 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:00.077212 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:00.077121 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:00.496175 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:00.496136 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:00.866082 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:00.865965 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:01.168770 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:01.168693 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-76774864d5-q9b96"] Oct 31 13:01:01.173427 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:01.173410 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-76774864d5-q9b96" Oct 31 13:01:01.238016 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:01.237971 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3c8c51\"" Oct 31 13:01:01.243181 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:01.243157 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-76774864d5-q9b96"] Oct 31 13:01:01.307051 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:01.306978 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:01.324797 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:01.324767 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81517822-a07f-45ea-a5ca-119633d02dc3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3c8c51-76774864d5-q9b96\" (UID: \"81517822-a07f-45ea-a5ca-119633d02dc3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-76774864d5-q9b96" Oct 31 13:01:01.324946 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:01.324810 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vxcp\" (UniqueName: \"kubernetes.io/projected/81517822-a07f-45ea-a5ca-119633d02dc3-kube-api-access-9vxcp\") pod \"payload-processing-e2e-ipp-b-3c8c51-76774864d5-q9b96\" (UID: \"81517822-a07f-45ea-a5ca-119633d02dc3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-76774864d5-q9b96" Oct 31 13:01:01.432879 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:01.429107 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81517822-a07f-45ea-a5ca-119633d02dc3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3c8c51-76774864d5-q9b96\" (UID: \"81517822-a07f-45ea-a5ca-119633d02dc3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-76774864d5-q9b96" Oct 31 13:01:01.432879 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:01.429166 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vxcp\" (UniqueName: \"kubernetes.io/projected/81517822-a07f-45ea-a5ca-119633d02dc3-kube-api-access-9vxcp\") pod \"payload-processing-e2e-ipp-b-3c8c51-76774864d5-q9b96\" (UID: \"81517822-a07f-45ea-a5ca-119633d02dc3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-76774864d5-q9b96" Oct 31 13:01:01.432879 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:01.430644 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81517822-a07f-45ea-a5ca-119633d02dc3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3c8c51-76774864d5-q9b96\" (UID: \"81517822-a07f-45ea-a5ca-119633d02dc3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-76774864d5-q9b96" Oct 31 13:01:01.503008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:01.490625 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vxcp\" (UniqueName: \"kubernetes.io/projected/81517822-a07f-45ea-a5ca-119633d02dc3-kube-api-access-9vxcp\") pod \"payload-processing-e2e-ipp-b-3c8c51-76774864d5-q9b96\" (UID: \"81517822-a07f-45ea-a5ca-119633d02dc3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-76774864d5-q9b96" Oct 31 13:01:01.788676 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:01.788643 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-76774864d5-q9b96" Oct 31 13:01:02.103186 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:02.103066 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-76774864d5-q9b96"] Oct 31 13:01:02.106807 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:01:02.106778 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81517822_a07f_45ea_a5ca_119633d02dc3.slice/crio-0fbda5faf4c3b0bd839f190b35d1a2621acda1a8c1c6b7616b0c86372db061f1 WatchSource:0}: Error finding container 0fbda5faf4c3b0bd839f190b35d1a2621acda1a8c1c6b7616b0c86372db061f1: Status 404 returned error can't find the container with id 0fbda5faf4c3b0bd839f190b35d1a2621acda1a8c1c6b7616b0c86372db061f1 Oct 31 13:01:02.797006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:02.795578 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:03.018768 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:03.018730 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-76774864d5-q9b96" event={"ID":"81517822-a07f-45ea-a5ca-119633d02dc3","Type":"ContainerStarted","Data":"29e3e2f488e29cff7eb133b99bb5978754a7a7c48d203d6396d4b46f899c441f"} Oct 31 13:01:03.018942 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:03.018790 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-76774864d5-q9b96" event={"ID":"81517822-a07f-45ea-a5ca-119633d02dc3","Type":"ContainerStarted","Data":"0fbda5faf4c3b0bd839f190b35d1a2621acda1a8c1c6b7616b0c86372db061f1"} Oct 31 13:01:03.018942 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:03.018821 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-76774864d5-q9b96" Oct 31 13:01:03.049900 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:03.049789 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-76774864d5-q9b96" podStartSLOduration=2.049774479 podStartE2EDuration="2.049774479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:01:03.047714797 +0000 UTC m=+1709.530837688" watchObservedRunningTime="2026-10-31 13:01:03.049774479 +0000 UTC m=+1709.532897379" Oct 31 13:01:03.087387 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:03.087352 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:04.221340 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:04.221094 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:04.435707 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:04.435671 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3128e5-85d8848459-shp4r"] Oct 31 13:01:04.675304 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:04.675217 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:04.959065 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:04.955755 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:05.988684 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:05.983072 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:06.284062 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:06.276625 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:07.176794 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:07.176745 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:07.992494 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:07.992410 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:09.099261 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:09.099215 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:09.436184 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:09.436101 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-3128e5-85d8848459-shp4r" podUID="26e9a61c-78ec-459d-835c-f384d3a4c4bc" containerName="payload-processing" containerID="cri-o://d7d20572b961d65aca53045eada19cc5117e47c6c52d092ef6d1338f92ab4c97" gracePeriod=25 Oct 31 13:01:09.730420 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:09.730392 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3128e5-85d8848459-shp4r" Oct 31 13:01:09.750066 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:09.750032 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26e9a61c-78ec-459d-835c-f384d3a4c4bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26e9a61c-78ec-459d-835c-f384d3a4c4bc-plugins-config-volume\") pod \"26e9a61c-78ec-459d-835c-f384d3a4c4bc\" (UID: \"26e9a61c-78ec-459d-835c-f384d3a4c4bc\") " Oct 31 13:01:09.750066 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:09.750070 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26e9a61c-78ec-459d-835c-f384d3a4c4bc-kube-api-access-v6gsv\" (UniqueName: \"kubernetes.io/projected/26e9a61c-78ec-459d-835c-f384d3a4c4bc-kube-api-access-v6gsv\") pod \"26e9a61c-78ec-459d-835c-f384d3a4c4bc\" (UID: \"26e9a61c-78ec-459d-835c-f384d3a4c4bc\") " Oct 31 13:01:09.750322 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:09.750287 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26e9a61c-78ec-459d-835c-f384d3a4c4bc-plugins-config-volume" pod "26e9a61c-78ec-459d-835c-f384d3a4c4bc" (UID: "26e9a61c-78ec-459d-835c-f384d3a4c4bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:01:09.751429 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:09.751106 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26e9a61c-78ec-459d-835c-f384d3a4c4bc-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:09.753107 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:09.753083 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26e9a61c-78ec-459d-835c-f384d3a4c4bc-kube-api-access-v6gsv" pod "26e9a61c-78ec-459d-835c-f384d3a4c4bc" (UID: "26e9a61c-78ec-459d-835c-f384d3a4c4bc"). InnerVolumeSpecName "kube-api-access-v6gsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:01:09.852648 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:09.852610 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6gsv\" (UniqueName: \"kubernetes.io/projected/26e9a61c-78ec-459d-835c-f384d3a4c4bc-kube-api-access-v6gsv\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:09.862409 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:09.862375 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-958f54b4c-d8bt2"] Oct 31 13:01:10.090407 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:10.090319 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:10.184203 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:10.184171 6841 generic.go:356] "Generic (PLEG): container finished" podID="26e9a61c-78ec-459d-835c-f384d3a4c4bc" containerID="d7d20572b961d65aca53045eada19cc5117e47c6c52d092ef6d1338f92ab4c97" exitCode=0 Oct 31 13:01:10.184608 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:10.184219 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3128e5-85d8848459-shp4r" Oct 31 13:01:10.184608 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:10.184251 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3128e5-85d8848459-shp4r" event={"ID":"26e9a61c-78ec-459d-835c-f384d3a4c4bc","Type":"ContainerDied","Data":"d7d20572b961d65aca53045eada19cc5117e47c6c52d092ef6d1338f92ab4c97"} Oct 31 13:01:10.184608 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:10.184310 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3128e5-85d8848459-shp4r" event={"ID":"26e9a61c-78ec-459d-835c-f384d3a4c4bc","Type":"ContainerDied","Data":"7db5bb2a65c8c62961389bef95517d67ec0f7c2f29c8193d43454b6ec08db6ee"} Oct 31 13:01:10.184608 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:10.184338 6841 scope.go:122] "RemoveContainer" containerID="d7d20572b961d65aca53045eada19cc5117e47c6c52d092ef6d1338f92ab4c97" Oct 31 13:01:10.194489 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:10.194471 6841 scope.go:122] "RemoveContainer" containerID="d7d20572b961d65aca53045eada19cc5117e47c6c52d092ef6d1338f92ab4c97" Oct 31 13:01:10.194766 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:01:10.194749 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7d20572b961d65aca53045eada19cc5117e47c6c52d092ef6d1338f92ab4c97\": container with ID starting with d7d20572b961d65aca53045eada19cc5117e47c6c52d092ef6d1338f92ab4c97 not found: ID does not exist" containerID="d7d20572b961d65aca53045eada19cc5117e47c6c52d092ef6d1338f92ab4c97" Oct 31 13:01:10.194826 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:10.194776 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7d20572b961d65aca53045eada19cc5117e47c6c52d092ef6d1338f92ab4c97"} err="failed to get container status \"d7d20572b961d65aca53045eada19cc5117e47c6c52d092ef6d1338f92ab4c97\": rpc error: code = NotFound desc = could not find container \"d7d20572b961d65aca53045eada19cc5117e47c6c52d092ef6d1338f92ab4c97\": container with ID starting with d7d20572b961d65aca53045eada19cc5117e47c6c52d092ef6d1338f92ab4c97 not found: ID does not exist" Oct 31 13:01:10.265682 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:10.265031 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3128e5-85d8848459-shp4r"] Oct 31 13:01:10.304006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:10.292499 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3128e5-85d8848459-shp4r"] Oct 31 13:01:10.541942 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:10.541902 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:11.110459 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:11.110425 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:11.678547 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:11.678516 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26e9a61c-78ec-459d-835c-f384d3a4c4bc" path="/var/lib/kubelet/pods/26e9a61c-78ec-459d-835c-f384d3a4c4bc/volumes" Oct 31 13:01:11.680666 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:11.680640 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:12.811446 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:12.811400 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:14.077539 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:14.077510 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-76774864d5-q9b96" Oct 31 13:01:14.222975 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:14.222930 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:14.865897 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:14.865129 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-958f54b4c-d8bt2" podUID="fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3" containerName="payload-pre-processing" containerID="cri-o://4c5d9c146f3aad611eaa216d5c0969f05e3e19e84a0fc1061c5b0a9a74d4ee9e" gracePeriod=25 Oct 31 13:01:14.976161 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:14.975777 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:15.421026 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:15.420950 6841 generic.go:356] "Generic (PLEG): container finished" podID="fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3" containerID="4c5d9c146f3aad611eaa216d5c0969f05e3e19e84a0fc1061c5b0a9a74d4ee9e" exitCode=0 Oct 31 13:01:15.421410 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:15.421070 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-958f54b4c-d8bt2" event={"ID":"fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3","Type":"ContainerDied","Data":"4c5d9c146f3aad611eaa216d5c0969f05e3e19e84a0fc1061c5b0a9a74d4ee9e"} Oct 31 13:01:15.436097 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:15.436055 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-958f54b4c-d8bt2" Oct 31 13:01:15.569839 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:15.569785 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3-kube-api-access-gfzq8\" (UniqueName: \"kubernetes.io/projected/fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3-kube-api-access-gfzq8\") pod \"fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3\" (UID: \"fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3\") " Oct 31 13:01:15.570232 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:15.569868 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3-plugins-config-volume\") pod \"fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3\" (UID: \"fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3\") " Oct 31 13:01:15.570832 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:15.570806 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3-plugins-config-volume" pod "fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3" (UID: "fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:01:15.572353 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:15.572323 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3-kube-api-access-gfzq8" pod "fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3" (UID: "fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3"). InnerVolumeSpecName "kube-api-access-gfzq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:01:15.672026 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:15.671903 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfzq8\" (UniqueName: \"kubernetes.io/projected/fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3-kube-api-access-gfzq8\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:15.672026 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:15.671939 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:15.927920 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:15.927806 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:16.431743 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:16.431714 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-958f54b4c-d8bt2" Oct 31 13:01:16.431743 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:16.431733 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-958f54b4c-d8bt2" event={"ID":"fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3","Type":"ContainerDied","Data":"2e83780fca5071cb2c424f1f1154249e16931ae3b48f19ce8aa067b7c6bc8718"} Oct 31 13:01:16.432224 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:16.431817 6841 scope.go:122] "RemoveContainer" containerID="4c5d9c146f3aad611eaa216d5c0969f05e3e19e84a0fc1061c5b0a9a74d4ee9e" Oct 31 13:01:16.481898 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:16.481866 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-958f54b4c-d8bt2"] Oct 31 13:01:16.485560 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:16.485539 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-958f54b4c-d8bt2"] Oct 31 13:01:17.008997 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:17.008946 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln"] Oct 31 13:01:17.009864 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:17.009839 6841 cpu_manager.go:414] "containerMap: removing container" podUID="26e9a61c-78ec-459d-835c-f384d3a4c4bc" containerName="payload-processing" Oct 31 13:01:17.009864 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:17.009861 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26e9a61c-78ec-459d-835c-f384d3a4c4bc" containerName="payload-processing" Oct 31 13:01:17.010140 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:17.009900 6841 cpu_manager.go:414] "containerMap: removing container" podUID="fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3" containerName="payload-pre-processing" Oct 31 13:01:17.010140 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:17.009907 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3" containerName="payload-pre-processing" Oct 31 13:01:17.010140 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:17.010036 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3" containerName="payload-pre-processing" Oct 31 13:01:17.010140 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:17.010060 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="26e9a61c-78ec-459d-835c-f384d3a4c4bc" containerName="payload-processing" Oct 31 13:01:17.015236 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:17.015210 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln" Oct 31 13:01:17.067408 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:17.067377 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-48dcc1-serving-cert\"" Oct 31 13:01:17.074813 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:17.074731 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln"] Oct 31 13:01:17.170025 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:17.169842 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:17.180003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:17.177034 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09c52086-c0c4-4429-9d45-4cc93d125c77-maas-api-tls\") pod \"maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln\" (UID: \"09c52086-c0c4-4429-9d45-4cc93d125c77\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln" Oct 31 13:01:17.192006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:17.186833 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w6dh\" (UniqueName: \"kubernetes.io/projected/09c52086-c0c4-4429-9d45-4cc93d125c77-kube-api-access-9w6dh\") pod \"maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln\" (UID: \"09c52086-c0c4-4429-9d45-4cc93d125c77\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln" Oct 31 13:01:17.192006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:17.186916 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09c52086-c0c4-4429-9d45-4cc93d125c77-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln\" (UID: \"09c52086-c0c4-4429-9d45-4cc93d125c77\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln" Oct 31 13:01:17.192006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:17.186969 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09c52086-c0c4-4429-9d45-4cc93d125c77-metrics-certs\") pod \"maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln\" (UID: \"09c52086-c0c4-4429-9d45-4cc93d125c77\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln" Oct 31 13:01:17.302051 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:17.300287 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09c52086-c0c4-4429-9d45-4cc93d125c77-maas-api-tls\") pod \"maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln\" (UID: \"09c52086-c0c4-4429-9d45-4cc93d125c77\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln" Oct 31 13:01:17.302051 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:17.300492 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w6dh\" (UniqueName: \"kubernetes.io/projected/09c52086-c0c4-4429-9d45-4cc93d125c77-kube-api-access-9w6dh\") pod \"maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln\" (UID: \"09c52086-c0c4-4429-9d45-4cc93d125c77\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln" Oct 31 13:01:17.302051 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:17.300538 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09c52086-c0c4-4429-9d45-4cc93d125c77-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln\" (UID: \"09c52086-c0c4-4429-9d45-4cc93d125c77\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln" Oct 31 13:01:17.302051 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:17.300581 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09c52086-c0c4-4429-9d45-4cc93d125c77-metrics-certs\") pod \"maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln\" (UID: \"09c52086-c0c4-4429-9d45-4cc93d125c77\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln" Oct 31 13:01:17.338532 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:17.337896 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w6dh\" (UniqueName: \"kubernetes.io/projected/09c52086-c0c4-4429-9d45-4cc93d125c77-kube-api-access-9w6dh\") pod \"maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln\" (UID: \"09c52086-c0c4-4429-9d45-4cc93d125c77\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln" Oct 31 13:01:17.338532 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:17.338276 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09c52086-c0c4-4429-9d45-4cc93d125c77-metrics-certs\") pod \"maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln\" (UID: \"09c52086-c0c4-4429-9d45-4cc93d125c77\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln" Oct 31 13:01:17.338892 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:17.302680 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09c52086-c0c4-4429-9d45-4cc93d125c77-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln\" (UID: \"09c52086-c0c4-4429-9d45-4cc93d125c77\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln" Oct 31 13:01:17.373368 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:17.373332 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09c52086-c0c4-4429-9d45-4cc93d125c77-maas-api-tls\") pod \"maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln\" (UID: \"09c52086-c0c4-4429-9d45-4cc93d125c77\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln" Oct 31 13:01:17.578340 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:17.578244 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:17.633225 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:17.633195 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln" Oct 31 13:01:17.675672 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:17.675641 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3" path="/var/lib/kubelet/pods/fbb95a64-0c06-4c4d-ab61-40fb5db9e4e3/volumes" Oct 31 13:01:17.888122 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:17.888035 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:17.980884 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:17.980861 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln"] Oct 31 13:01:18.177147 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:18.177052 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg"] Oct 31 13:01:18.177299 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:18.177277 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" podUID="35d138be-ff3b-43d9-ae6b-bae298aaec6b" containerName="istio-proxy" containerID="cri-o://535a8c90bfaf4cf2f66bfc858f198361bdf1911ba87c30a088d5522f570df7f3" gracePeriod=30 Oct 31 13:01:18.182483 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:18.182453 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9"] Oct 31 13:01:18.182694 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:18.182672 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" podUID="6e512c4f-871f-49c6-8114-13073e883e1a" containerName="istio-proxy" containerID="cri-o://7f91f31b7a034a1727465e671ddf923eeeeef2bb6031b640b7285dfe2e6195c2" gracePeriod=30 Oct 31 13:01:18.366008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:18.365256 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:18.573377 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:18.573331 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln" event={"ID":"09c52086-c0c4-4429-9d45-4cc93d125c77","Type":"ContainerStarted","Data":"97fd92bb7318d66664db444e1ea0caf7044ca48a7ca6fd343ff5e093e711ddab"} Oct 31 13:01:19.077858 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:19.077818 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:19.349062 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:19.341443 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:19.583040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:19.582974 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln" event={"ID":"09c52086-c0c4-4429-9d45-4cc93d125c77","Type":"ContainerStarted","Data":"185ebe897c682dd925c7365389e784a6cd418b4ba8a9ac0722a6ea54e1a7bd4f"} Oct 31 13:01:19.583194 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:19.583063 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln" Oct 31 13:01:19.613741 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:19.613656 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln" podStartSLOduration=2.986434799 podStartE2EDuration="3.61364209s" totalImagesPullingTime="627.207291ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:01:19.612229961 +0000 UTC m=+1726.095352870" watchObservedRunningTime="2026-10-31 13:01:19.61364209 +0000 UTC m=+1726.096765047" Oct 31 13:01:20.035796 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:01:20.035760 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:01:20.698093 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:20.698042 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:21.733566 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:21.733535 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:22.700229 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:22.700189 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:22.957723 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:22.957634 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:23.477493 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.477422 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 13:01:23.544916 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.544880 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" Oct 31 13:01:23.582865 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.582835 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35d138be-ff3b-43d9-ae6b-bae298aaec6b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35d138be-ff3b-43d9-ae6b-bae298aaec6b-workload-certs\") pod \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\" (UID: \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\") " Oct 31 13:01:23.583037 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.582968 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35d138be-ff3b-43d9-ae6b-bae298aaec6b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35d138be-ff3b-43d9-ae6b-bae298aaec6b-istio-envoy\") pod \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\" (UID: \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\") " Oct 31 13:01:23.583117 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.583095 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35d138be-ff3b-43d9-ae6b-bae298aaec6b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d138be-ff3b-43d9-ae6b-bae298aaec6b-credential-socket\") pod \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\" (UID: \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\") " Oct 31 13:01:23.583169 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.583116 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35d138be-ff3b-43d9-ae6b-bae298aaec6b-workload-certs" pod "35d138be-ff3b-43d9-ae6b-bae298aaec6b" (UID: "35d138be-ff3b-43d9-ae6b-bae298aaec6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:01:23.583169 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.583145 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35d138be-ff3b-43d9-ae6b-bae298aaec6b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d138be-ff3b-43d9-ae6b-bae298aaec6b-workload-socket\") pod \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\" (UID: \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\") " Oct 31 13:01:23.583319 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.583302 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35d138be-ff3b-43d9-ae6b-bae298aaec6b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35d138be-ff3b-43d9-ae6b-bae298aaec6b-istiod-ca-cert\") pod \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\" (UID: \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\") " Oct 31 13:01:23.583395 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.583372 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35d138be-ff3b-43d9-ae6b-bae298aaec6b-kube-api-access-phqpc\" (UniqueName: \"kubernetes.io/projected/35d138be-ff3b-43d9-ae6b-bae298aaec6b-kube-api-access-phqpc\") pod \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\" (UID: \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\") " Oct 31 13:01:23.583454 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.583416 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35d138be-ff3b-43d9-ae6b-bae298aaec6b-istio-token\" (UniqueName: \"kubernetes.io/projected/35d138be-ff3b-43d9-ae6b-bae298aaec6b-istio-token\") pod \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\" (UID: \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\") " Oct 31 13:01:23.583454 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.583409 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35d138be-ff3b-43d9-ae6b-bae298aaec6b-credential-socket" pod "35d138be-ff3b-43d9-ae6b-bae298aaec6b" (UID: "35d138be-ff3b-43d9-ae6b-bae298aaec6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:01:23.583567 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.583463 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35d138be-ff3b-43d9-ae6b-bae298aaec6b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35d138be-ff3b-43d9-ae6b-bae298aaec6b-istio-podinfo\") pod \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\" (UID: \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\") " Oct 31 13:01:23.583567 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.583469 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35d138be-ff3b-43d9-ae6b-bae298aaec6b-workload-socket" pod "35d138be-ff3b-43d9-ae6b-bae298aaec6b" (UID: "35d138be-ff3b-43d9-ae6b-bae298aaec6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:01:23.584377 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.583813 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35d138be-ff3b-43d9-ae6b-bae298aaec6b-istiod-ca-cert" pod "35d138be-ff3b-43d9-ae6b-bae298aaec6b" (UID: "35d138be-ff3b-43d9-ae6b-bae298aaec6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:01:23.584377 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.583862 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35d138be-ff3b-43d9-ae6b-bae298aaec6b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35d138be-ff3b-43d9-ae6b-bae298aaec6b-istio-data\") pod \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\" (UID: \"35d138be-ff3b-43d9-ae6b-bae298aaec6b\") " Oct 31 13:01:23.584377 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.584248 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35d138be-ff3b-43d9-ae6b-bae298aaec6b-istio-data" pod "35d138be-ff3b-43d9-ae6b-bae298aaec6b" (UID: "35d138be-ff3b-43d9-ae6b-bae298aaec6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:01:23.584816 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.584793 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35d138be-ff3b-43d9-ae6b-bae298aaec6b-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:23.585014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.584823 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35d138be-ff3b-43d9-ae6b-bae298aaec6b-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:23.585014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.584840 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d138be-ff3b-43d9-ae6b-bae298aaec6b-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:23.585014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.584854 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d138be-ff3b-43d9-ae6b-bae298aaec6b-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:23.585014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.584868 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35d138be-ff3b-43d9-ae6b-bae298aaec6b-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:23.585592 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.585570 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35d138be-ff3b-43d9-ae6b-bae298aaec6b-istio-envoy" pod "35d138be-ff3b-43d9-ae6b-bae298aaec6b" (UID: "35d138be-ff3b-43d9-ae6b-bae298aaec6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:01:23.585849 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.585827 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35d138be-ff3b-43d9-ae6b-bae298aaec6b-kube-api-access-phqpc" pod "35d138be-ff3b-43d9-ae6b-bae298aaec6b" (UID: "35d138be-ff3b-43d9-ae6b-bae298aaec6b"). InnerVolumeSpecName "kube-api-access-phqpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:01:23.586422 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.586401 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35d138be-ff3b-43d9-ae6b-bae298aaec6b-istio-token" pod "35d138be-ff3b-43d9-ae6b-bae298aaec6b" (UID: "35d138be-ff3b-43d9-ae6b-bae298aaec6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:01:23.587854 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.587831 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35d138be-ff3b-43d9-ae6b-bae298aaec6b-istio-podinfo" pod "35d138be-ff3b-43d9-ae6b-bae298aaec6b" (UID: "35d138be-ff3b-43d9-ae6b-bae298aaec6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:01:23.649075 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.649040 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:23.650161 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.650136 6841 generic.go:356] "Generic (PLEG): container finished" podID="35d138be-ff3b-43d9-ae6b-bae298aaec6b" containerID="535a8c90bfaf4cf2f66bfc858f198361bdf1911ba87c30a088d5522f570df7f3" exitCode=0 Oct 31 13:01:23.650295 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.650217 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" event={"ID":"35d138be-ff3b-43d9-ae6b-bae298aaec6b","Type":"ContainerDied","Data":"535a8c90bfaf4cf2f66bfc858f198361bdf1911ba87c30a088d5522f570df7f3"} Oct 31 13:01:23.650295 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.650265 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" event={"ID":"35d138be-ff3b-43d9-ae6b-bae298aaec6b","Type":"ContainerDied","Data":"9b2475212121bce2a52d38e2a36eff760108531a8fc02d15790511bfe60c0d71"} Oct 31 13:01:23.650295 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.650293 6841 scope.go:122] "RemoveContainer" containerID="535a8c90bfaf4cf2f66bfc858f198361bdf1911ba87c30a088d5522f570df7f3" Oct 31 13:01:23.650457 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.650428 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg" Oct 31 13:01:23.654167 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.654137 6841 generic.go:356] "Generic (PLEG): container finished" podID="6e512c4f-871f-49c6-8114-13073e883e1a" containerID="7f91f31b7a034a1727465e671ddf923eeeeef2bb6031b640b7285dfe2e6195c2" exitCode=0 Oct 31 13:01:23.654419 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.654397 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" event={"ID":"6e512c4f-871f-49c6-8114-13073e883e1a","Type":"ContainerDied","Data":"7f91f31b7a034a1727465e671ddf923eeeeef2bb6031b640b7285dfe2e6195c2"} Oct 31 13:01:23.654492 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.654451 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" event={"ID":"6e512c4f-871f-49c6-8114-13073e883e1a","Type":"ContainerDied","Data":"c485ed3572bf7d6ac3cc06a01dfc21b9c7b124014e1052753838705e7091843c"} Oct 31 13:01:23.654551 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.654521 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9" Oct 31 13:01:23.683573 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.683550 6841 scope.go:122] "RemoveContainer" containerID="b37e75d6f82bc614102b7d3ac3f705c19eb68f5a289554cd5f76254127cfff9c" Oct 31 13:01:23.686086 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.686057 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e512c4f-871f-49c6-8114-13073e883e1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e512c4f-871f-49c6-8114-13073e883e1a-workload-socket\") pod \"6e512c4f-871f-49c6-8114-13073e883e1a\" (UID: \"6e512c4f-871f-49c6-8114-13073e883e1a\") " Oct 31 13:01:23.686210 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.686176 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e512c4f-871f-49c6-8114-13073e883e1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e512c4f-871f-49c6-8114-13073e883e1a-istio-podinfo\") pod \"6e512c4f-871f-49c6-8114-13073e883e1a\" (UID: \"6e512c4f-871f-49c6-8114-13073e883e1a\") " Oct 31 13:01:23.686325 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.686303 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e512c4f-871f-49c6-8114-13073e883e1a-workload-socket" pod "6e512c4f-871f-49c6-8114-13073e883e1a" (UID: "6e512c4f-871f-49c6-8114-13073e883e1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:01:23.686382 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.686361 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e512c4f-871f-49c6-8114-13073e883e1a-istio-token\" (UniqueName: \"kubernetes.io/projected/6e512c4f-871f-49c6-8114-13073e883e1a-istio-token\") pod \"6e512c4f-871f-49c6-8114-13073e883e1a\" (UID: \"6e512c4f-871f-49c6-8114-13073e883e1a\") " Oct 31 13:01:23.686497 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.686475 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e512c4f-871f-49c6-8114-13073e883e1a-kube-api-access-vz5b4\" (UniqueName: \"kubernetes.io/projected/6e512c4f-871f-49c6-8114-13073e883e1a-kube-api-access-vz5b4\") pod \"6e512c4f-871f-49c6-8114-13073e883e1a\" (UID: \"6e512c4f-871f-49c6-8114-13073e883e1a\") " Oct 31 13:01:23.686599 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.686515 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e512c4f-871f-49c6-8114-13073e883e1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e512c4f-871f-49c6-8114-13073e883e1a-workload-certs\") pod \"6e512c4f-871f-49c6-8114-13073e883e1a\" (UID: \"6e512c4f-871f-49c6-8114-13073e883e1a\") " Oct 31 13:01:23.686599 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.686543 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e512c4f-871f-49c6-8114-13073e883e1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e512c4f-871f-49c6-8114-13073e883e1a-istio-data\") pod \"6e512c4f-871f-49c6-8114-13073e883e1a\" (UID: \"6e512c4f-871f-49c6-8114-13073e883e1a\") " Oct 31 13:01:23.686710 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.686629 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e512c4f-871f-49c6-8114-13073e883e1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e512c4f-871f-49c6-8114-13073e883e1a-istiod-ca-cert\") pod \"6e512c4f-871f-49c6-8114-13073e883e1a\" (UID: \"6e512c4f-871f-49c6-8114-13073e883e1a\") " Oct 31 13:01:23.686765 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.686721 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e512c4f-871f-49c6-8114-13073e883e1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e512c4f-871f-49c6-8114-13073e883e1a-istio-envoy\") pod \"6e512c4f-871f-49c6-8114-13073e883e1a\" (UID: \"6e512c4f-871f-49c6-8114-13073e883e1a\") " Oct 31 13:01:23.686765 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.686758 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e512c4f-871f-49c6-8114-13073e883e1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e512c4f-871f-49c6-8114-13073e883e1a-credential-socket\") pod \"6e512c4f-871f-49c6-8114-13073e883e1a\" (UID: \"6e512c4f-871f-49c6-8114-13073e883e1a\") " Oct 31 13:01:23.687362 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.687217 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e512c4f-871f-49c6-8114-13073e883e1a-istiod-ca-cert" pod "6e512c4f-871f-49c6-8114-13073e883e1a" (UID: "6e512c4f-871f-49c6-8114-13073e883e1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:01:23.687576 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.687523 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e512c4f-871f-49c6-8114-13073e883e1a-istio-data" pod "6e512c4f-871f-49c6-8114-13073e883e1a" (UID: "6e512c4f-871f-49c6-8114-13073e883e1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:01:23.687763 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.687740 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e512c4f-871f-49c6-8114-13073e883e1a-workload-certs" pod "6e512c4f-871f-49c6-8114-13073e883e1a" (UID: "6e512c4f-871f-49c6-8114-13073e883e1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:01:23.688098 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.688078 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phqpc\" (UniqueName: \"kubernetes.io/projected/35d138be-ff3b-43d9-ae6b-bae298aaec6b-kube-api-access-phqpc\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:23.688215 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.688202 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35d138be-ff3b-43d9-ae6b-bae298aaec6b-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:23.688317 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.688304 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35d138be-ff3b-43d9-ae6b-bae298aaec6b-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:23.688413 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.688401 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e512c4f-871f-49c6-8114-13073e883e1a-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:23.688496 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.688483 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35d138be-ff3b-43d9-ae6b-bae298aaec6b-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:23.688615 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.688601 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e512c4f-871f-49c6-8114-13073e883e1a-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:23.688721 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.688708 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e512c4f-871f-49c6-8114-13073e883e1a-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:23.689054 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.689025 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e512c4f-871f-49c6-8114-13073e883e1a-credential-socket" pod "6e512c4f-871f-49c6-8114-13073e883e1a" (UID: "6e512c4f-871f-49c6-8114-13073e883e1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:01:23.689442 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.689422 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e512c4f-871f-49c6-8114-13073e883e1a-istio-token" pod "6e512c4f-871f-49c6-8114-13073e883e1a" (UID: "6e512c4f-871f-49c6-8114-13073e883e1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:01:23.690639 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.690617 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e512c4f-871f-49c6-8114-13073e883e1a-kube-api-access-vz5b4" pod "6e512c4f-871f-49c6-8114-13073e883e1a" (UID: "6e512c4f-871f-49c6-8114-13073e883e1a"). InnerVolumeSpecName "kube-api-access-vz5b4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:01:23.694133 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.690961 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e512c4f-871f-49c6-8114-13073e883e1a-istio-envoy" pod "6e512c4f-871f-49c6-8114-13073e883e1a" (UID: "6e512c4f-871f-49c6-8114-13073e883e1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:01:23.694133 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.691863 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e512c4f-871f-49c6-8114-13073e883e1a-istio-podinfo" pod "6e512c4f-871f-49c6-8114-13073e883e1a" (UID: "6e512c4f-871f-49c6-8114-13073e883e1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:01:23.694133 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.694041 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg"] Oct 31 13:01:23.696634 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.696493 6841 scope.go:122] "RemoveContainer" containerID="535a8c90bfaf4cf2f66bfc858f198361bdf1911ba87c30a088d5522f570df7f3" Oct 31 13:01:23.697089 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:01:23.697054 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"535a8c90bfaf4cf2f66bfc858f198361bdf1911ba87c30a088d5522f570df7f3\": container with ID starting with 535a8c90bfaf4cf2f66bfc858f198361bdf1911ba87c30a088d5522f570df7f3 not found: ID does not exist" containerID="535a8c90bfaf4cf2f66bfc858f198361bdf1911ba87c30a088d5522f570df7f3" Oct 31 13:01:23.697265 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.697219 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"535a8c90bfaf4cf2f66bfc858f198361bdf1911ba87c30a088d5522f570df7f3"} err="failed to get container status \"535a8c90bfaf4cf2f66bfc858f198361bdf1911ba87c30a088d5522f570df7f3\": rpc error: code = NotFound desc = could not find container \"535a8c90bfaf4cf2f66bfc858f198361bdf1911ba87c30a088d5522f570df7f3\": container with ID starting with 535a8c90bfaf4cf2f66bfc858f198361bdf1911ba87c30a088d5522f570df7f3 not found: ID does not exist" Oct 31 13:01:23.697265 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.697256 6841 scope.go:122] "RemoveContainer" containerID="b37e75d6f82bc614102b7d3ac3f705c19eb68f5a289554cd5f76254127cfff9c" Oct 31 13:01:23.698298 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:01:23.698268 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b37e75d6f82bc614102b7d3ac3f705c19eb68f5a289554cd5f76254127cfff9c\": container with ID starting with b37e75d6f82bc614102b7d3ac3f705c19eb68f5a289554cd5f76254127cfff9c not found: ID does not exist" containerID="b37e75d6f82bc614102b7d3ac3f705c19eb68f5a289554cd5f76254127cfff9c" Oct 31 13:01:23.698406 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.698308 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b37e75d6f82bc614102b7d3ac3f705c19eb68f5a289554cd5f76254127cfff9c"} err="failed to get container status \"b37e75d6f82bc614102b7d3ac3f705c19eb68f5a289554cd5f76254127cfff9c\": rpc error: code = NotFound desc = could not find container \"b37e75d6f82bc614102b7d3ac3f705c19eb68f5a289554cd5f76254127cfff9c\": container with ID starting with b37e75d6f82bc614102b7d3ac3f705c19eb68f5a289554cd5f76254127cfff9c not found: ID does not exist" Oct 31 13:01:23.698406 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.698329 6841 scope.go:122] "RemoveContainer" containerID="7f91f31b7a034a1727465e671ddf923eeeeef2bb6031b640b7285dfe2e6195c2" Oct 31 13:01:23.702498 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.702479 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-7hrlg"] Oct 31 13:01:23.711199 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.711167 6841 scope.go:122] "RemoveContainer" containerID="7f91f31b7a034a1727465e671ddf923eeeeef2bb6031b640b7285dfe2e6195c2" Oct 31 13:01:23.711553 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:01:23.711528 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f91f31b7a034a1727465e671ddf923eeeeef2bb6031b640b7285dfe2e6195c2\": container with ID starting with 7f91f31b7a034a1727465e671ddf923eeeeef2bb6031b640b7285dfe2e6195c2 not found: ID does not exist" containerID="7f91f31b7a034a1727465e671ddf923eeeeef2bb6031b640b7285dfe2e6195c2" Oct 31 13:01:23.711654 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.711560 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f91f31b7a034a1727465e671ddf923eeeeef2bb6031b640b7285dfe2e6195c2"} err="failed to get container status \"7f91f31b7a034a1727465e671ddf923eeeeef2bb6031b640b7285dfe2e6195c2\": rpc error: code = NotFound desc = could not find container \"7f91f31b7a034a1727465e671ddf923eeeeef2bb6031b640b7285dfe2e6195c2\": container with ID starting with 7f91f31b7a034a1727465e671ddf923eeeeef2bb6031b640b7285dfe2e6195c2 not found: ID does not exist" Oct 31 13:01:23.790005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.789930 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz5b4\" (UniqueName: \"kubernetes.io/projected/6e512c4f-871f-49c6-8114-13073e883e1a-kube-api-access-vz5b4\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:23.790005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.789955 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e512c4f-871f-49c6-8114-13073e883e1a-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:23.790005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.789965 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e512c4f-871f-49c6-8114-13073e883e1a-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:23.790005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.789974 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e512c4f-871f-49c6-8114-13073e883e1a-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:23.790005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.790000 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e512c4f-871f-49c6-8114-13073e883e1a-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:23.790252 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:23.790078 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e512c4f-871f-49c6-8114-13073e883e1a-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:24.023338 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.023297 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9"] Oct 31 13:01:24.026897 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.026872 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt"] Oct 31 13:01:24.027603 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.027582 6841 cpu_manager.go:414] "containerMap: removing container" podUID="35d138be-ff3b-43d9-ae6b-bae298aaec6b" containerName="istio-proxy" Oct 31 13:01:24.027603 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.027602 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35d138be-ff3b-43d9-ae6b-bae298aaec6b" containerName="istio-proxy" Oct 31 13:01:24.027768 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.027648 6841 cpu_manager.go:414] "containerMap: removing container" podUID="6e512c4f-871f-49c6-8114-13073e883e1a" containerName="istio-proxy" Oct 31 13:01:24.027768 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.027654 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e512c4f-871f-49c6-8114-13073e883e1a" containerName="istio-proxy" Oct 31 13:01:24.027835 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.027776 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e512c4f-871f-49c6-8114-13073e883e1a" containerName="istio-proxy" Oct 31 13:01:24.027835 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.027788 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="35d138be-ff3b-43d9-ae6b-bae298aaec6b" containerName="istio-proxy" Oct 31 13:01:24.027835 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.027797 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="35d138be-ff3b-43d9-ae6b-bae298aaec6b" containerName="istio-proxy" Oct 31 13:01:24.032974 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.032952 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:01:24.033323 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.033302 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:24.056554 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.056455 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-6nft9"] Oct 31 13:01:24.065861 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.065820 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-01e8d2-openshift-default-dockercfg-nvcds\"" Oct 31 13:01:24.066497 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.066473 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt"] Oct 31 13:01:24.232567 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.231684 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd421e14-02ab-4072-918a-e3c6077263c8-istio-podinfo\") pod \"e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt\" (UID: \"cd421e14-02ab-4072-918a-e3c6077263c8\") " pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:01:24.232567 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.231773 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd421e14-02ab-4072-918a-e3c6077263c8-credential-socket\") pod \"e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt\" (UID: \"cd421e14-02ab-4072-918a-e3c6077263c8\") " pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:01:24.232567 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.231861 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd421e14-02ab-4072-918a-e3c6077263c8-istiod-ca-cert\") pod \"e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt\" (UID: \"cd421e14-02ab-4072-918a-e3c6077263c8\") " pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:01:24.232567 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.231886 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd421e14-02ab-4072-918a-e3c6077263c8-istio-envoy\") pod \"e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt\" (UID: \"cd421e14-02ab-4072-918a-e3c6077263c8\") " pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:01:24.232567 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.231976 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd421e14-02ab-4072-918a-e3c6077263c8-workload-socket\") pod \"e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt\" (UID: \"cd421e14-02ab-4072-918a-e3c6077263c8\") " pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:01:24.232567 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.232022 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tllhz\" (UniqueName: \"kubernetes.io/projected/cd421e14-02ab-4072-918a-e3c6077263c8-kube-api-access-tllhz\") pod \"e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt\" (UID: \"cd421e14-02ab-4072-918a-e3c6077263c8\") " pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:01:24.232567 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.232140 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd421e14-02ab-4072-918a-e3c6077263c8-istio-data\") pod \"e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt\" (UID: \"cd421e14-02ab-4072-918a-e3c6077263c8\") " pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:01:24.232567 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.232171 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd421e14-02ab-4072-918a-e3c6077263c8-workload-certs\") pod \"e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt\" (UID: \"cd421e14-02ab-4072-918a-e3c6077263c8\") " pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:01:24.232567 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.232247 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd421e14-02ab-4072-918a-e3c6077263c8-istio-token\") pod \"e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt\" (UID: \"cd421e14-02ab-4072-918a-e3c6077263c8\") " pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:01:24.332614 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.332476 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:24.336096 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.333685 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd421e14-02ab-4072-918a-e3c6077263c8-credential-socket\") pod \"e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt\" (UID: \"cd421e14-02ab-4072-918a-e3c6077263c8\") " pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:01:24.336096 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.333776 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd421e14-02ab-4072-918a-e3c6077263c8-istiod-ca-cert\") pod \"e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt\" (UID: \"cd421e14-02ab-4072-918a-e3c6077263c8\") " pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:01:24.336096 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.333804 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd421e14-02ab-4072-918a-e3c6077263c8-istio-envoy\") pod \"e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt\" (UID: \"cd421e14-02ab-4072-918a-e3c6077263c8\") " pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:01:24.336096 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.333885 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd421e14-02ab-4072-918a-e3c6077263c8-workload-socket\") pod \"e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt\" (UID: \"cd421e14-02ab-4072-918a-e3c6077263c8\") " pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:01:24.336096 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.333912 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tllhz\" (UniqueName: \"kubernetes.io/projected/cd421e14-02ab-4072-918a-e3c6077263c8-kube-api-access-tllhz\") pod \"e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt\" (UID: \"cd421e14-02ab-4072-918a-e3c6077263c8\") " pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:01:24.336096 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.335075 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd421e14-02ab-4072-918a-e3c6077263c8-istiod-ca-cert\") pod \"e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt\" (UID: \"cd421e14-02ab-4072-918a-e3c6077263c8\") " pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:01:24.336096 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.335365 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd421e14-02ab-4072-918a-e3c6077263c8-credential-socket\") pod \"e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt\" (UID: \"cd421e14-02ab-4072-918a-e3c6077263c8\") " pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:01:24.336096 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.335996 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd421e14-02ab-4072-918a-e3c6077263c8-workload-socket\") pod \"e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt\" (UID: \"cd421e14-02ab-4072-918a-e3c6077263c8\") " pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:01:24.338976 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.337159 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd421e14-02ab-4072-918a-e3c6077263c8-istio-data\") pod \"e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt\" (UID: \"cd421e14-02ab-4072-918a-e3c6077263c8\") " pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:01:24.338976 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.337222 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd421e14-02ab-4072-918a-e3c6077263c8-workload-certs\") pod \"e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt\" (UID: \"cd421e14-02ab-4072-918a-e3c6077263c8\") " pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:01:24.338976 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.337273 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd421e14-02ab-4072-918a-e3c6077263c8-istio-token\") pod \"e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt\" (UID: \"cd421e14-02ab-4072-918a-e3c6077263c8\") " pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:01:24.338976 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.337383 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd421e14-02ab-4072-918a-e3c6077263c8-istio-podinfo\") pod \"e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt\" (UID: \"cd421e14-02ab-4072-918a-e3c6077263c8\") " pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:01:24.338976 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.338681 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd421e14-02ab-4072-918a-e3c6077263c8-workload-certs\") pod \"e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt\" (UID: \"cd421e14-02ab-4072-918a-e3c6077263c8\") " pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:01:24.338976 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.338912 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd421e14-02ab-4072-918a-e3c6077263c8-istio-data\") pod \"e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt\" (UID: \"cd421e14-02ab-4072-918a-e3c6077263c8\") " pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:01:24.356842 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.356569 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd421e14-02ab-4072-918a-e3c6077263c8-istio-podinfo\") pod \"e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt\" (UID: \"cd421e14-02ab-4072-918a-e3c6077263c8\") " pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:01:24.374502 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.374240 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd421e14-02ab-4072-918a-e3c6077263c8-istio-envoy\") pod \"e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt\" (UID: \"cd421e14-02ab-4072-918a-e3c6077263c8\") " pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:01:24.378953 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.378884 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd421e14-02ab-4072-918a-e3c6077263c8-istio-token\") pod \"e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt\" (UID: \"cd421e14-02ab-4072-918a-e3c6077263c8\") " pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:01:24.380822 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.380798 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tllhz\" (UniqueName: \"kubernetes.io/projected/cd421e14-02ab-4072-918a-e3c6077263c8-kube-api-access-tllhz\") pod \"e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt\" (UID: \"cd421e14-02ab-4072-918a-e3c6077263c8\") " pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:01:24.682058 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.655157 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:01:24.794016 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.789060 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:24.888364 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.888327 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-56a83a-7865d4d4f7-ggdfx"] Oct 31 13:01:24.973777 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:24.973745 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-56a83a-7d476bc98c-mxvqr"] Oct 31 13:01:25.188448 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:25.188413 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:25.234434 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:25.234360 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt"] Oct 31 13:01:25.242214 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:01:25.242187 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd421e14_02ab_4072_918a_e3c6077263c8.slice/crio-03fd4b1ffc0798f75eb763c21600430ba9d2d1873f994f2d35986b86c00f8e8e WatchSource:0}: Error finding container 03fd4b1ffc0798f75eb763c21600430ba9d2d1873f994f2d35986b86c00f8e8e: Status 404 returned error can't find the container with id 03fd4b1ffc0798f75eb763c21600430ba9d2d1873f994f2d35986b86c00f8e8e Oct 31 13:01:25.245590 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:25.245551 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:01:25.245714 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:25.245645 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:01:25.245714 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:25.245698 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:01:25.560063 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:25.559214 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:25.706049 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:25.703671 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35d138be-ff3b-43d9-ae6b-bae298aaec6b" path="/var/lib/kubelet/pods/35d138be-ff3b-43d9-ae6b-bae298aaec6b/volumes" Oct 31 13:01:25.706049 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:25.704454 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e512c4f-871f-49c6-8114-13073e883e1a" path="/var/lib/kubelet/pods/6e512c4f-871f-49c6-8114-13073e883e1a/volumes" Oct 31 13:01:25.706049 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:25.704935 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln" Oct 31 13:01:25.823940 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:25.823905 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" event={"ID":"cd421e14-02ab-4072-918a-e3c6077263c8","Type":"ContainerStarted","Data":"5ee31ac384bff02e3313e7c063614a58abdc9a5ab2e8039edaa7438800835cd8"} Oct 31 13:01:25.824162 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:25.823968 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" event={"ID":"cd421e14-02ab-4072-918a-e3c6077263c8","Type":"ContainerStarted","Data":"03fd4b1ffc0798f75eb763c21600430ba9d2d1873f994f2d35986b86c00f8e8e"} Oct 31 13:01:25.873632 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:25.873574 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" podStartSLOduration=2.8735594730000003 podStartE2EDuration="2.873559473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:01:25.872125603 +0000 UTC m=+1732.355248506" watchObservedRunningTime="2026-10-31 13:01:25.873559473 +0000 UTC m=+1732.356682373" Oct 31 13:01:26.171411 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:26.171312 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:26.656890 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:26.656856 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:01:26.662889 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:26.662863 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:01:26.835122 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:26.835089 6841 generic.go:356] "Generic (PLEG): container finished" podID="1c502702-da0b-45d4-ba77-5ae972c1bd16" containerID="062d41d9f05c95d634be86d75b50e597350d8238c78dac954f601aaccfaf0996" exitCode=0 Oct 31 13:01:26.835317 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:26.835160 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" event={"ID":"1c502702-da0b-45d4-ba77-5ae972c1bd16","Type":"ContainerDied","Data":"062d41d9f05c95d634be86d75b50e597350d8238c78dac954f601aaccfaf0996"} Oct 31 13:01:26.836024 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:26.835971 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:01:26.836786 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:26.836763 6841 scope.go:122] "RemoveContainer" containerID="062d41d9f05c95d634be86d75b50e597350d8238c78dac954f601aaccfaf0996" Oct 31 13:01:26.837463 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:26.837333 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:01:26.838913 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:26.838880 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:01:26.839025 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:26.839004 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:01:26.839096 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:26.839071 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:01:26.972006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:26.966574 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:27.277005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:27.265759 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-65f69c944c-g26k2"] Oct 31 13:01:27.331014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:27.328665 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:27.441014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:27.351053 6841 cpu_manager.go:414] "containerMap: removing container" podUID="35d138be-ff3b-43d9-ae6b-bae298aaec6b" containerName="istio-proxy" Oct 31 13:01:27.441014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:27.351086 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35d138be-ff3b-43d9-ae6b-bae298aaec6b" containerName="istio-proxy" Oct 31 13:01:27.541318 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:27.541230 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-65f69c944c-g26k2"] Oct 31 13:01:27.541499 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:27.541357 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-65f69c944c-g26k2" Oct 31 13:01:27.545659 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:27.545631 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-adfd11\"" Oct 31 13:01:27.545798 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:27.545720 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-adfd11-dockercfg-928z5\"" Oct 31 13:01:27.568868 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:27.568828 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe022e44-83a1-4f5e-b2be-dc6438d8631b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-adfd11-65f69c944c-g26k2\" (UID: \"fe022e44-83a1-4f5e-b2be-dc6438d8631b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-65f69c944c-g26k2" Oct 31 13:01:27.569087 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:27.569008 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kftd\" (UniqueName: \"kubernetes.io/projected/fe022e44-83a1-4f5e-b2be-dc6438d8631b-kube-api-access-6kftd\") pod \"payload-pre-processing-e2e-ipp-a-adfd11-65f69c944c-g26k2\" (UID: \"fe022e44-83a1-4f5e-b2be-dc6438d8631b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-65f69c944c-g26k2" Oct 31 13:01:27.670556 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:27.670495 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe022e44-83a1-4f5e-b2be-dc6438d8631b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-adfd11-65f69c944c-g26k2\" (UID: \"fe022e44-83a1-4f5e-b2be-dc6438d8631b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-65f69c944c-g26k2" Oct 31 13:01:27.671164 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:27.670702 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kftd\" (UniqueName: \"kubernetes.io/projected/fe022e44-83a1-4f5e-b2be-dc6438d8631b-kube-api-access-6kftd\") pod \"payload-pre-processing-e2e-ipp-a-adfd11-65f69c944c-g26k2\" (UID: \"fe022e44-83a1-4f5e-b2be-dc6438d8631b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-65f69c944c-g26k2" Oct 31 13:01:27.671666 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:27.671622 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe022e44-83a1-4f5e-b2be-dc6438d8631b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-adfd11-65f69c944c-g26k2\" (UID: \"fe022e44-83a1-4f5e-b2be-dc6438d8631b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-65f69c944c-g26k2" Oct 31 13:01:27.685493 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:27.685469 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kftd\" (UniqueName: \"kubernetes.io/projected/fe022e44-83a1-4f5e-b2be-dc6438d8631b-kube-api-access-6kftd\") pod \"payload-pre-processing-e2e-ipp-a-adfd11-65f69c944c-g26k2\" (UID: \"fe022e44-83a1-4f5e-b2be-dc6438d8631b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-65f69c944c-g26k2" Oct 31 13:01:27.860362 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:27.860269 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-65f69c944c-g26k2" Oct 31 13:01:27.869283 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:27.869243 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" event={"ID":"1c502702-da0b-45d4-ba77-5ae972c1bd16","Type":"ContainerStarted","Data":"6dfa34e2c9a6a902add417d53971c6b934d50782523538c27d21c25cfcdf5add"} Oct 31 13:01:27.940006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:27.939057 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:01:27.964148 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:27.958895 6841 patch_prober.go:29] interesting pod/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 31 13:01:27.964148 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:27.958961 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" podUID="1c502702-da0b-45d4-ba77-5ae972c1bd16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 31 13:01:27.992011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:27.987786 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:28.325520 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:28.325489 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-65f69c944c-g26k2"] Oct 31 13:01:28.326284 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:01:28.326257 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe022e44_83a1_4f5e_b2be_dc6438d8631b.slice/crio-ed2f9b642fbdf56d7177f0d2cd16f39d0c000a42c9899fce70b9d3f3f9a7e902 WatchSource:0}: Error finding container ed2f9b642fbdf56d7177f0d2cd16f39d0c000a42c9899fce70b9d3f3f9a7e902: Status 404 returned error can't find the container with id ed2f9b642fbdf56d7177f0d2cd16f39d0c000a42c9899fce70b9d3f3f9a7e902 Oct 31 13:01:28.596182 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:28.596082 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:28.861056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:28.860517 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:28.880235 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:28.880200 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-65f69c944c-g26k2" event={"ID":"fe022e44-83a1-4f5e-b2be-dc6438d8631b","Type":"ContainerStarted","Data":"4e4263adcb3a30a5423ee74a8fa565d6d4bfb835f48e0a5f10c6cd19c076ac4d"} Oct 31 13:01:28.880458 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:28.880251 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-65f69c944c-g26k2" event={"ID":"fe022e44-83a1-4f5e-b2be-dc6438d8631b","Type":"ContainerStarted","Data":"ed2f9b642fbdf56d7177f0d2cd16f39d0c000a42c9899fce70b9d3f3f9a7e902"} Oct 31 13:01:28.880458 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:28.880285 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-65f69c944c-g26k2" Oct 31 13:01:28.910783 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:28.910734 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-65f69c944c-g26k2" podStartSLOduration=1.9107216550000001 podStartE2EDuration="1.910721655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:01:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:01:28.905771981 +0000 UTC m=+1735.388894883" watchObservedRunningTime="2026-10-31 13:01:28.910721655 +0000 UTC m=+1735.393844555" Oct 31 13:01:28.922077 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:28.922052 6841 patch_prober.go:29] interesting pod/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 31 13:01:28.922210 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:28.922098 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" podUID="1c502702-da0b-45d4-ba77-5ae972c1bd16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 31 13:01:29.189415 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:29.189315 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:29.497131 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:29.497097 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:29.889270 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:29.889185 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-56a83a-7865d4d4f7-ggdfx" podUID="6324318a-fb95-45af-a04d-8c73774e8c84" containerName="payload-processing" containerID="cri-o://619cfe6ee46c8be893a6f30ce6bbdd1d9f3017f2a35b2116465cb9166c56c4e0" gracePeriod=25 Oct 31 13:01:29.922346 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:29.922318 6841 patch_prober.go:29] interesting pod/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 31 13:01:29.922466 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:29.922373 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" podUID="1c502702-da0b-45d4-ba77-5ae972c1bd16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 31 13:01:29.970006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:29.962241 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:29.975753 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:29.975499 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56a83a-7d476bc98c-mxvqr" podUID="bc662639-7616-4ebe-88cc-ef4026b066c8" containerName="payload-pre-processing" containerID="cri-o://f8149ee2c7f053495790d73a1a36258eb6d3c42b1e3db6925e1a454f68105d28" gracePeriod=25 Oct 31 13:01:30.229797 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:30.229722 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:30.628631 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:30.628613 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56a83a-7d476bc98c-mxvqr" Oct 31 13:01:30.633045 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:30.633027 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc662639-7616-4ebe-88cc-ef4026b066c8-kube-api-access-shvb6\" (UniqueName: \"kubernetes.io/projected/bc662639-7616-4ebe-88cc-ef4026b066c8-kube-api-access-shvb6\") pod \"bc662639-7616-4ebe-88cc-ef4026b066c8\" (UID: \"bc662639-7616-4ebe-88cc-ef4026b066c8\") " Oct 31 13:01:30.633118 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:30.633086 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc662639-7616-4ebe-88cc-ef4026b066c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc662639-7616-4ebe-88cc-ef4026b066c8-plugins-config-volume\") pod \"bc662639-7616-4ebe-88cc-ef4026b066c8\" (UID: \"bc662639-7616-4ebe-88cc-ef4026b066c8\") " Oct 31 13:01:30.633461 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:30.633403 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc662639-7616-4ebe-88cc-ef4026b066c8-plugins-config-volume" pod "bc662639-7616-4ebe-88cc-ef4026b066c8" (UID: "bc662639-7616-4ebe-88cc-ef4026b066c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:01:30.633791 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:30.633774 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc662639-7616-4ebe-88cc-ef4026b066c8-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:30.635091 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:30.635068 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc662639-7616-4ebe-88cc-ef4026b066c8-kube-api-access-shvb6" pod "bc662639-7616-4ebe-88cc-ef4026b066c8" (UID: "bc662639-7616-4ebe-88cc-ef4026b066c8"). InnerVolumeSpecName "kube-api-access-shvb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:01:30.683773 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:30.683737 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-56a83a-7865d4d4f7-ggdfx" Oct 31 13:01:30.734312 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:30.734241 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6324318a-fb95-45af-a04d-8c73774e8c84-kube-api-access-kjlxw\" (UniqueName: \"kubernetes.io/projected/6324318a-fb95-45af-a04d-8c73774e8c84-kube-api-access-kjlxw\") pod \"6324318a-fb95-45af-a04d-8c73774e8c84\" (UID: \"6324318a-fb95-45af-a04d-8c73774e8c84\") " Oct 31 13:01:30.734459 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:30.734442 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6324318a-fb95-45af-a04d-8c73774e8c84-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6324318a-fb95-45af-a04d-8c73774e8c84-plugins-config-volume\") pod \"6324318a-fb95-45af-a04d-8c73774e8c84\" (UID: \"6324318a-fb95-45af-a04d-8c73774e8c84\") " Oct 31 13:01:30.734793 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:30.734769 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6324318a-fb95-45af-a04d-8c73774e8c84-plugins-config-volume" pod "6324318a-fb95-45af-a04d-8c73774e8c84" (UID: "6324318a-fb95-45af-a04d-8c73774e8c84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:01:30.735003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:30.734972 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shvb6\" (UniqueName: \"kubernetes.io/projected/bc662639-7616-4ebe-88cc-ef4026b066c8-kube-api-access-shvb6\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:30.735094 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:30.735008 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6324318a-fb95-45af-a04d-8c73774e8c84-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:30.736738 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:30.736710 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6324318a-fb95-45af-a04d-8c73774e8c84-kube-api-access-kjlxw" pod "6324318a-fb95-45af-a04d-8c73774e8c84" (UID: "6324318a-fb95-45af-a04d-8c73774e8c84"). InnerVolumeSpecName "kube-api-access-kjlxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:01:30.836539 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:30.836507 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjlxw\" (UniqueName: \"kubernetes.io/projected/6324318a-fb95-45af-a04d-8c73774e8c84-kube-api-access-kjlxw\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:30.983008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:30.974478 6841 generic.go:356] "Generic (PLEG): container finished" podID="bc662639-7616-4ebe-88cc-ef4026b066c8" containerID="f8149ee2c7f053495790d73a1a36258eb6d3c42b1e3db6925e1a454f68105d28" exitCode=0 Oct 31 13:01:30.983008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:30.974700 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56a83a-7d476bc98c-mxvqr" event={"ID":"bc662639-7616-4ebe-88cc-ef4026b066c8","Type":"ContainerDied","Data":"f8149ee2c7f053495790d73a1a36258eb6d3c42b1e3db6925e1a454f68105d28"} Oct 31 13:01:30.983008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:30.974833 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56a83a-7d476bc98c-mxvqr" event={"ID":"bc662639-7616-4ebe-88cc-ef4026b066c8","Type":"ContainerDied","Data":"e2abfebf65d21c8d12a9cc62e0a992761d79b27186d0e1eb3027411cf053ca27"} Oct 31 13:01:30.983008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:30.974864 6841 scope.go:122] "RemoveContainer" containerID="f8149ee2c7f053495790d73a1a36258eb6d3c42b1e3db6925e1a454f68105d28" Oct 31 13:01:30.986217 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:30.986076 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56a83a-7d476bc98c-mxvqr" Oct 31 13:01:30.990018 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:30.989910 6841 patch_prober.go:29] interesting pod/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 31 13:01:30.990292 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:30.990270 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" podUID="1c502702-da0b-45d4-ba77-5ae972c1bd16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 31 13:01:31.009084 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:31.009004 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:31.011643 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:31.011219 6841 generic.go:356] "Generic (PLEG): container finished" podID="6324318a-fb95-45af-a04d-8c73774e8c84" containerID="619cfe6ee46c8be893a6f30ce6bbdd1d9f3017f2a35b2116465cb9166c56c4e0" exitCode=0 Oct 31 13:01:31.011643 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:31.011348 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-56a83a-7865d4d4f7-ggdfx" event={"ID":"6324318a-fb95-45af-a04d-8c73774e8c84","Type":"ContainerDied","Data":"619cfe6ee46c8be893a6f30ce6bbdd1d9f3017f2a35b2116465cb9166c56c4e0"} Oct 31 13:01:31.011643 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:31.011520 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-56a83a-7865d4d4f7-ggdfx" event={"ID":"6324318a-fb95-45af-a04d-8c73774e8c84","Type":"ContainerDied","Data":"968d518175e1e017894ac90aafa469a80e83dae4ba83d482064c0c0c03d158d9"} Oct 31 13:01:31.011643 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:31.011364 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-56a83a-7865d4d4f7-ggdfx" Oct 31 13:01:31.026776 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:31.025765 6841 scope.go:122] "RemoveContainer" containerID="f8149ee2c7f053495790d73a1a36258eb6d3c42b1e3db6925e1a454f68105d28" Oct 31 13:01:31.027657 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:01:31.027373 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8149ee2c7f053495790d73a1a36258eb6d3c42b1e3db6925e1a454f68105d28\": container with ID starting with f8149ee2c7f053495790d73a1a36258eb6d3c42b1e3db6925e1a454f68105d28 not found: ID does not exist" containerID="f8149ee2c7f053495790d73a1a36258eb6d3c42b1e3db6925e1a454f68105d28" Oct 31 13:01:31.027657 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:31.027421 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8149ee2c7f053495790d73a1a36258eb6d3c42b1e3db6925e1a454f68105d28"} err="failed to get container status \"f8149ee2c7f053495790d73a1a36258eb6d3c42b1e3db6925e1a454f68105d28\": rpc error: code = NotFound desc = could not find container \"f8149ee2c7f053495790d73a1a36258eb6d3c42b1e3db6925e1a454f68105d28\": container with ID starting with f8149ee2c7f053495790d73a1a36258eb6d3c42b1e3db6925e1a454f68105d28 not found: ID does not exist" Oct 31 13:01:31.027657 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:31.027445 6841 scope.go:122] "RemoveContainer" containerID="619cfe6ee46c8be893a6f30ce6bbdd1d9f3017f2a35b2116465cb9166c56c4e0" Oct 31 13:01:31.045460 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:31.045436 6841 scope.go:122] "RemoveContainer" containerID="619cfe6ee46c8be893a6f30ce6bbdd1d9f3017f2a35b2116465cb9166c56c4e0" Oct 31 13:01:31.046435 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:01:31.046406 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"619cfe6ee46c8be893a6f30ce6bbdd1d9f3017f2a35b2116465cb9166c56c4e0\": container with ID starting with 619cfe6ee46c8be893a6f30ce6bbdd1d9f3017f2a35b2116465cb9166c56c4e0 not found: ID does not exist" containerID="619cfe6ee46c8be893a6f30ce6bbdd1d9f3017f2a35b2116465cb9166c56c4e0" Oct 31 13:01:31.046592 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:31.046568 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"619cfe6ee46c8be893a6f30ce6bbdd1d9f3017f2a35b2116465cb9166c56c4e0"} err="failed to get container status \"619cfe6ee46c8be893a6f30ce6bbdd1d9f3017f2a35b2116465cb9166c56c4e0\": rpc error: code = NotFound desc = could not find container \"619cfe6ee46c8be893a6f30ce6bbdd1d9f3017f2a35b2116465cb9166c56c4e0\": container with ID starting with 619cfe6ee46c8be893a6f30ce6bbdd1d9f3017f2a35b2116465cb9166c56c4e0 not found: ID does not exist" Oct 31 13:01:31.073330 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:31.073297 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-56a83a-7d476bc98c-mxvqr"] Oct 31 13:01:31.078189 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:31.078162 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-56a83a-7d476bc98c-mxvqr"] Oct 31 13:01:31.086463 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:31.086439 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-56a83a-7865d4d4f7-ggdfx"] Oct 31 13:01:31.093567 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:31.093539 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-56a83a-7865d4d4f7-ggdfx"] Oct 31 13:01:31.668647 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:31.668608 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6324318a-fb95-45af-a04d-8c73774e8c84" path="/var/lib/kubelet/pods/6324318a-fb95-45af-a04d-8c73774e8c84/volumes" Oct 31 13:01:31.669324 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:31.669305 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc662639-7616-4ebe-88cc-ef4026b066c8" path="/var/lib/kubelet/pods/bc662639-7616-4ebe-88cc-ef4026b066c8/volumes" Oct 31 13:01:31.921852 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:31.921804 6841 patch_prober.go:29] interesting pod/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 31 13:01:31.922031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:31.921874 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" podUID="1c502702-da0b-45d4-ba77-5ae972c1bd16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 31 13:01:31.955005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:31.950774 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:01:31.990509 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:01:31.990476 6841 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/NetworkManager-dispatcher.service\": RecentStats: unable to find data in memory cache]" Oct 31 13:01:31.996110 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:01:31.996059 6841 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/NetworkManager-dispatcher.service\": RecentStats: unable to find data in memory cache]" Oct 31 13:01:32.064733 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:32.064672 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:32.872751 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:32.872710 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:32.922829 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:32.922780 6841 patch_prober.go:29] interesting pod/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 31 13:01:32.923056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:32.922838 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" podUID="1c502702-da0b-45d4-ba77-5ae972c1bd16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 31 13:01:33.957085 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:33.957052 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:33.961032 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:33.961008 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:01:33.962022 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:33.961999 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:01:34.957048 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:34.956042 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:35.057642 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:35.057610 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-76774864d5-q9b96"] Oct 31 13:01:35.250415 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:35.250322 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:35.491005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:35.487013 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:35.724002 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:35.718853 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:36.112634 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:36.110239 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:36.631003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:36.626212 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:36.986285 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:36.986252 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:37.812049 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:37.808846 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:38.645008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:38.641680 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:39.542051 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:39.542013 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:39.890236 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:39.890155 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-65f69c944c-g26k2" Oct 31 13:01:40.060166 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:40.060072 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-76774864d5-q9b96" podUID="81517822-a07f-45ea-a5ca-119633d02dc3" containerName="payload-processing" containerID="cri-o://29e3e2f488e29cff7eb133b99bb5978754a7a7c48d203d6396d4b46f899c441f" gracePeriod=25 Oct 31 13:01:40.196963 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:40.196933 6841 generic.go:356] "Generic (PLEG): container finished" podID="81517822-a07f-45ea-a5ca-119633d02dc3" containerID="29e3e2f488e29cff7eb133b99bb5978754a7a7c48d203d6396d4b46f899c441f" exitCode=0 Oct 31 13:01:40.197166 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:40.197017 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-76774864d5-q9b96" event={"ID":"81517822-a07f-45ea-a5ca-119633d02dc3","Type":"ContainerDied","Data":"29e3e2f488e29cff7eb133b99bb5978754a7a7c48d203d6396d4b46f899c441f"} Oct 31 13:01:40.378743 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:40.378714 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-76774864d5-q9b96" Oct 31 13:01:40.479639 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:40.479608 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81517822-a07f-45ea-a5ca-119633d02dc3-kube-api-access-9vxcp\" (UniqueName: \"kubernetes.io/projected/81517822-a07f-45ea-a5ca-119633d02dc3-kube-api-access-9vxcp\") pod \"81517822-a07f-45ea-a5ca-119633d02dc3\" (UID: \"81517822-a07f-45ea-a5ca-119633d02dc3\") " Oct 31 13:01:40.479944 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:40.479922 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81517822-a07f-45ea-a5ca-119633d02dc3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81517822-a07f-45ea-a5ca-119633d02dc3-plugins-config-volume\") pod \"81517822-a07f-45ea-a5ca-119633d02dc3\" (UID: \"81517822-a07f-45ea-a5ca-119633d02dc3\") " Oct 31 13:01:40.480485 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:40.480375 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81517822-a07f-45ea-a5ca-119633d02dc3-plugins-config-volume" pod "81517822-a07f-45ea-a5ca-119633d02dc3" (UID: "81517822-a07f-45ea-a5ca-119633d02dc3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:01:40.481093 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:40.481068 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81517822-a07f-45ea-a5ca-119633d02dc3-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:40.483031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:40.482597 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81517822-a07f-45ea-a5ca-119633d02dc3-kube-api-access-9vxcp" pod "81517822-a07f-45ea-a5ca-119633d02dc3" (UID: "81517822-a07f-45ea-a5ca-119633d02dc3"). InnerVolumeSpecName "kube-api-access-9vxcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:01:40.577525 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:40.577490 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:40.582765 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:40.582739 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vxcp\" (UniqueName: \"kubernetes.io/projected/81517822-a07f-45ea-a5ca-119633d02dc3-kube-api-access-9vxcp\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:40.877148 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:40.877035 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:41.212529 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:41.212497 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-76774864d5-q9b96" event={"ID":"81517822-a07f-45ea-a5ca-119633d02dc3","Type":"ContainerDied","Data":"0fbda5faf4c3b0bd839f190b35d1a2621acda1a8c1c6b7616b0c86372db061f1"} Oct 31 13:01:41.212821 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:41.212807 6841 scope.go:122] "RemoveContainer" containerID="29e3e2f488e29cff7eb133b99bb5978754a7a7c48d203d6396d4b46f899c441f" Oct 31 13:01:41.213038 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:41.212688 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-76774864d5-q9b96" Oct 31 13:01:41.280352 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:41.280314 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-76774864d5-q9b96"] Oct 31 13:01:41.287136 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:41.287108 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-76774864d5-q9b96"] Oct 31 13:01:41.367553 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:41.367518 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:41.593248 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:41.592511 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:41.700116 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:41.700082 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81517822-a07f-45ea-a5ca-119633d02dc3" path="/var/lib/kubelet/pods/81517822-a07f-45ea-a5ca-119633d02dc3/volumes" Oct 31 13:01:41.831333 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:41.831286 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:41.933084 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:41.924645 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr"] Oct 31 13:01:41.933084 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:41.925797 6841 cpu_manager.go:414] "containerMap: removing container" podUID="6324318a-fb95-45af-a04d-8c73774e8c84" containerName="payload-processing" Oct 31 13:01:41.933084 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:41.925819 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6324318a-fb95-45af-a04d-8c73774e8c84" containerName="payload-processing" Oct 31 13:01:41.933084 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:41.925844 6841 cpu_manager.go:414] "containerMap: removing container" podUID="bc662639-7616-4ebe-88cc-ef4026b066c8" containerName="payload-pre-processing" Oct 31 13:01:41.933084 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:41.925852 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc662639-7616-4ebe-88cc-ef4026b066c8" containerName="payload-pre-processing" Oct 31 13:01:41.933084 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:41.925909 6841 cpu_manager.go:414] "containerMap: removing container" podUID="81517822-a07f-45ea-a5ca-119633d02dc3" containerName="payload-processing" Oct 31 13:01:41.933084 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:41.925918 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81517822-a07f-45ea-a5ca-119633d02dc3" containerName="payload-processing" Oct 31 13:01:41.933084 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:41.926108 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="81517822-a07f-45ea-a5ca-119633d02dc3" containerName="payload-processing" Oct 31 13:01:41.933084 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:41.926133 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="6324318a-fb95-45af-a04d-8c73774e8c84" containerName="payload-processing" Oct 31 13:01:41.933084 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:41.926146 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc662639-7616-4ebe-88cc-ef4026b066c8" containerName="payload-pre-processing" Oct 31 13:01:41.976007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:41.971526 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr"] Oct 31 13:01:41.976007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:41.971638 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr" Oct 31 13:01:41.984020 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:41.982527 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-01e8d2-serving-cert\"" Oct 31 13:01:42.026110 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:42.026079 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eadb1e5-3fb0-49a8-964b-06d862381d04-metrics-certs\") pod \"maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr\" (UID: \"6eadb1e5-3fb0-49a8-964b-06d862381d04\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr" Oct 31 13:01:42.026284 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:42.026259 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfrnv\" (UniqueName: \"kubernetes.io/projected/6eadb1e5-3fb0-49a8-964b-06d862381d04-kube-api-access-lfrnv\") pod \"maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr\" (UID: \"6eadb1e5-3fb0-49a8-964b-06d862381d04\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr" Oct 31 13:01:42.026467 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:42.026444 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eadb1e5-3fb0-49a8-964b-06d862381d04-maas-api-tls\") pod \"maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr\" (UID: \"6eadb1e5-3fb0-49a8-964b-06d862381d04\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr" Oct 31 13:01:42.026593 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:42.026573 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eadb1e5-3fb0-49a8-964b-06d862381d04-openshift-service-ca\") pod \"maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr\" (UID: \"6eadb1e5-3fb0-49a8-964b-06d862381d04\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr" Oct 31 13:01:42.127808 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:42.127774 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfrnv\" (UniqueName: \"kubernetes.io/projected/6eadb1e5-3fb0-49a8-964b-06d862381d04-kube-api-access-lfrnv\") pod \"maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr\" (UID: \"6eadb1e5-3fb0-49a8-964b-06d862381d04\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr" Oct 31 13:01:42.128013 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:42.127882 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eadb1e5-3fb0-49a8-964b-06d862381d04-maas-api-tls\") pod \"maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr\" (UID: \"6eadb1e5-3fb0-49a8-964b-06d862381d04\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr" Oct 31 13:01:42.128013 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:42.127940 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eadb1e5-3fb0-49a8-964b-06d862381d04-openshift-service-ca\") pod \"maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr\" (UID: \"6eadb1e5-3fb0-49a8-964b-06d862381d04\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr" Oct 31 13:01:42.128013 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:42.128004 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eadb1e5-3fb0-49a8-964b-06d862381d04-metrics-certs\") pod \"maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr\" (UID: \"6eadb1e5-3fb0-49a8-964b-06d862381d04\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr" Oct 31 13:01:42.128742 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:42.128715 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eadb1e5-3fb0-49a8-964b-06d862381d04-openshift-service-ca\") pod \"maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr\" (UID: \"6eadb1e5-3fb0-49a8-964b-06d862381d04\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr" Oct 31 13:01:42.130436 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:42.130415 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eadb1e5-3fb0-49a8-964b-06d862381d04-maas-api-tls\") pod \"maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr\" (UID: \"6eadb1e5-3fb0-49a8-964b-06d862381d04\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr" Oct 31 13:01:42.130503 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:42.130449 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eadb1e5-3fb0-49a8-964b-06d862381d04-metrics-certs\") pod \"maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr\" (UID: \"6eadb1e5-3fb0-49a8-964b-06d862381d04\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr" Oct 31 13:01:42.187763 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:42.187689 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfrnv\" (UniqueName: \"kubernetes.io/projected/6eadb1e5-3fb0-49a8-964b-06d862381d04-kube-api-access-lfrnv\") pod \"maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr\" (UID: \"6eadb1e5-3fb0-49a8-964b-06d862381d04\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr" Oct 31 13:01:42.228404 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:42.228368 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:42.302623 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:42.302572 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr" Oct 31 13:01:42.600773 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:42.600468 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:42.819514 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:42.819486 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr"] Oct 31 13:01:42.819859 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:01:42.819834 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eadb1e5_3fb0_49a8_964b_06d862381d04.slice/crio-5699c2339cadb6e47f074c7c14de753cd5f3b6db1f01f87187cef2e61f27eec2 WatchSource:0}: Error finding container 5699c2339cadb6e47f074c7c14de753cd5f3b6db1f01f87187cef2e61f27eec2: Status 404 returned error can't find the container with id 5699c2339cadb6e47f074c7c14de753cd5f3b6db1f01f87187cef2e61f27eec2 Oct 31 13:01:42.834942 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:42.834897 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:43.106008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:43.105314 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:43.434615 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:43.434465 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:43.450055 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:43.449649 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr" event={"ID":"6eadb1e5-3fb0-49a8-964b-06d862381d04","Type":"ContainerStarted","Data":"5699c2339cadb6e47f074c7c14de753cd5f3b6db1f01f87187cef2e61f27eec2"} Oct 31 13:01:44.268113 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:44.267926 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:44.458062 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:44.458019 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr" event={"ID":"6eadb1e5-3fb0-49a8-964b-06d862381d04","Type":"ContainerStarted","Data":"1fe0703aa482df70feee2facb1721e8ebfcc0047dc8cfe22b19a27f0e8ca2c2c"} Oct 31 13:01:44.458274 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:44.458256 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr" Oct 31 13:01:44.483867 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:44.483812 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr" podStartSLOduration=2.821980435 podStartE2EDuration="3.483795092s" totalImagesPullingTime="661.814657ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:01:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:01:44.481899273 +0000 UTC m=+1750.965022183" watchObservedRunningTime="2026-10-31 13:01:44.483795092 +0000 UTC m=+1750.966917994" Oct 31 13:01:44.962023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:44.953938 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:45.192588 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:45.192507 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:46.630749 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:46.630711 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:46.900608 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:46.900527 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:47.251118 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:47.251079 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:47.773814 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:47.773775 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:48.087338 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:48.087192 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:48.346774 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:48.346681 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:48.733615 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:48.733576 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:48.813303 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:48.813267 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-794f7bb6dc-4dcjj"] Oct 31 13:01:48.823341 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:48.823313 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-794f7bb6dc-4dcjj" Oct 31 13:01:48.825727 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:48.825701 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-794f7bb6dc-4dcjj"] Oct 31 13:01:48.863308 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:48.863286 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3c8c51\"" Oct 31 13:01:49.042070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:48.911013 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95r45\" (UniqueName: \"kubernetes.io/projected/f30ce016-5c64-4a2c-9b4e-a942e53c1e59-kube-api-access-95r45\") pod \"payload-pre-processing-e2e-ipp-b-3c8c51-794f7bb6dc-4dcjj\" (UID: \"f30ce016-5c64-4a2c-9b4e-a942e53c1e59\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-794f7bb6dc-4dcjj" Oct 31 13:01:49.042070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:48.911152 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f30ce016-5c64-4a2c-9b4e-a942e53c1e59-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3c8c51-794f7bb6dc-4dcjj\" (UID: \"f30ce016-5c64-4a2c-9b4e-a942e53c1e59\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-794f7bb6dc-4dcjj" Oct 31 13:01:49.042070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:48.980408 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3c8c51-dockercfg-cdplk\"" Oct 31 13:01:49.042070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:49.025552 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f30ce016-5c64-4a2c-9b4e-a942e53c1e59-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3c8c51-794f7bb6dc-4dcjj\" (UID: \"f30ce016-5c64-4a2c-9b4e-a942e53c1e59\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-794f7bb6dc-4dcjj" Oct 31 13:01:49.042070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:49.025841 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95r45\" (UniqueName: \"kubernetes.io/projected/f30ce016-5c64-4a2c-9b4e-a942e53c1e59-kube-api-access-95r45\") pod \"payload-pre-processing-e2e-ipp-b-3c8c51-794f7bb6dc-4dcjj\" (UID: \"f30ce016-5c64-4a2c-9b4e-a942e53c1e59\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-794f7bb6dc-4dcjj" Oct 31 13:01:49.042070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:49.027139 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f30ce016-5c64-4a2c-9b4e-a942e53c1e59-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3c8c51-794f7bb6dc-4dcjj\" (UID: \"f30ce016-5c64-4a2c-9b4e-a942e53c1e59\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-794f7bb6dc-4dcjj" Oct 31 13:01:49.079423 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:49.060815 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:49.217584 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:49.129776 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95r45\" (UniqueName: \"kubernetes.io/projected/f30ce016-5c64-4a2c-9b4e-a942e53c1e59-kube-api-access-95r45\") pod \"payload-pre-processing-e2e-ipp-b-3c8c51-794f7bb6dc-4dcjj\" (UID: \"f30ce016-5c64-4a2c-9b4e-a942e53c1e59\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-794f7bb6dc-4dcjj" Oct 31 13:01:49.217584 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:49.143947 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-794f7bb6dc-4dcjj" Oct 31 13:01:49.217584 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:49.150080 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-64bd66bc8b-9kvnt"] Oct 31 13:01:49.217584 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:49.158557 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-64bd66bc8b-9kvnt" Oct 31 13:01:49.226664 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:49.226622 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-64bd66bc8b-9kvnt"] Oct 31 13:01:49.232212 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:49.229719 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96cc2f18-6920-405d-9186-a70543773377-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3c8c51-64bd66bc8b-9kvnt\" (UID: \"96cc2f18-6920-405d-9186-a70543773377\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-64bd66bc8b-9kvnt" Oct 31 13:01:49.232212 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:49.229847 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpl2g\" (UniqueName: \"kubernetes.io/projected/96cc2f18-6920-405d-9186-a70543773377-kube-api-access-zpl2g\") pod \"payload-processing-e2e-ipp-b-3c8c51-64bd66bc8b-9kvnt\" (UID: \"96cc2f18-6920-405d-9186-a70543773377\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-64bd66bc8b-9kvnt" Oct 31 13:01:49.331226 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:49.331189 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpl2g\" (UniqueName: \"kubernetes.io/projected/96cc2f18-6920-405d-9186-a70543773377-kube-api-access-zpl2g\") pod \"payload-processing-e2e-ipp-b-3c8c51-64bd66bc8b-9kvnt\" (UID: \"96cc2f18-6920-405d-9186-a70543773377\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-64bd66bc8b-9kvnt" Oct 31 13:01:49.331443 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:49.331423 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96cc2f18-6920-405d-9186-a70543773377-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3c8c51-64bd66bc8b-9kvnt\" (UID: \"96cc2f18-6920-405d-9186-a70543773377\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-64bd66bc8b-9kvnt" Oct 31 13:01:49.332245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:49.332204 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96cc2f18-6920-405d-9186-a70543773377-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3c8c51-64bd66bc8b-9kvnt\" (UID: \"96cc2f18-6920-405d-9186-a70543773377\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-64bd66bc8b-9kvnt" Oct 31 13:01:49.346219 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:49.346197 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpl2g\" (UniqueName: \"kubernetes.io/projected/96cc2f18-6920-405d-9186-a70543773377-kube-api-access-zpl2g\") pod \"payload-processing-e2e-ipp-b-3c8c51-64bd66bc8b-9kvnt\" (UID: \"96cc2f18-6920-405d-9186-a70543773377\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-64bd66bc8b-9kvnt" Oct 31 13:01:49.457366 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:49.454600 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:49.503689 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:49.503660 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-64bd66bc8b-9kvnt" Oct 31 13:01:49.515682 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:49.515651 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-794f7bb6dc-4dcjj"] Oct 31 13:01:49.517191 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:01:49.517166 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf30ce016_5c64_4a2c_9b4e_a942e53c1e59.slice/crio-56e8b573f7b487558905601b3495328b14f2fa8d7ffb240349238288c54c4948 WatchSource:0}: Error finding container 56e8b573f7b487558905601b3495328b14f2fa8d7ffb240349238288c54c4948: Status 404 returned error can't find the container with id 56e8b573f7b487558905601b3495328b14f2fa8d7ffb240349238288c54c4948 Oct 31 13:01:49.543968 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:49.543927 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-794f7bb6dc-4dcjj" event={"ID":"f30ce016-5c64-4a2c-9b4e-a942e53c1e59","Type":"ContainerStarted","Data":"56e8b573f7b487558905601b3495328b14f2fa8d7ffb240349238288c54c4948"} Oct 31 13:01:49.545799 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:49.545775 6841 generic.go:356] "Generic (PLEG): container finished" podID="ba9a674c-9aec-46ad-9600-56952a8128c0" containerID="c94e46db8065457675b759a07fa1b7b20fc687e393dd3c812bd1dab9da8264bd" exitCode=0 Oct 31 13:01:49.545896 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:49.545818 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" event={"ID":"ba9a674c-9aec-46ad-9600-56952a8128c0","Type":"ContainerDied","Data":"c94e46db8065457675b759a07fa1b7b20fc687e393dd3c812bd1dab9da8264bd"} Oct 31 13:01:49.546409 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:49.546387 6841 scope.go:122] "RemoveContainer" containerID="c94e46db8065457675b759a07fa1b7b20fc687e393dd3c812bd1dab9da8264bd" Oct 31 13:01:49.548168 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:49.548139 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:01:49.548269 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:49.548225 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:01:49.548334 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:49.548275 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:01:49.794269 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:01:49.794237 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96cc2f18_6920_405d_9186_a70543773377.slice/crio-b1ed172254a28b1d3b4dce27c78439c8c27949b884313d3be6918cd460fac0a6 WatchSource:0}: Error finding container b1ed172254a28b1d3b4dce27c78439c8c27949b884313d3be6918cd460fac0a6: Status 404 returned error can't find the container with id b1ed172254a28b1d3b4dce27c78439c8c27949b884313d3be6918cd460fac0a6 Oct 31 13:01:49.796247 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:49.796221 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-64bd66bc8b-9kvnt"] Oct 31 13:01:50.038652 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:01:50.038620 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:01:50.320038 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:50.304049 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:50.533248 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:50.533214 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr" Oct 31 13:01:50.573091 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:50.569734 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-64bd66bc8b-9kvnt" event={"ID":"96cc2f18-6920-405d-9186-a70543773377","Type":"ContainerStarted","Data":"2cefc7c8d9ee6722b70f6924169424f3f6e785c9398026859b1a71a9dba6b203"} Oct 31 13:01:50.573091 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:50.569798 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-64bd66bc8b-9kvnt" event={"ID":"96cc2f18-6920-405d-9186-a70543773377","Type":"ContainerStarted","Data":"b1ed172254a28b1d3b4dce27c78439c8c27949b884313d3be6918cd460fac0a6"} Oct 31 13:01:50.597003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:50.595838 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-64bd66bc8b-9kvnt" Oct 31 13:01:50.603337 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:50.603282 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" event={"ID":"ba9a674c-9aec-46ad-9600-56952a8128c0","Type":"ContainerStarted","Data":"4e581b0af62adc108f1ad054a96625d9cd7b08771a1b54cc98cec23286c1935e"} Oct 31 13:01:50.609324 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:50.608263 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-794f7bb6dc-4dcjj" event={"ID":"f30ce016-5c64-4a2c-9b4e-a942e53c1e59","Type":"ContainerStarted","Data":"e60e30e2f815b45949a75656d57420a90232341c8c5c24b2d1085d2c24b5271b"} Oct 31 13:01:50.609324 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:50.608339 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-794f7bb6dc-4dcjj" Oct 31 13:01:50.649950 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:50.641848 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-64bd66bc8b-9kvnt" podStartSLOduration=2.641829859 podStartE2EDuration="2.641829859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:01:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:01:50.639481173 +0000 UTC m=+1757.122604080" watchObservedRunningTime="2026-10-31 13:01:50.641829859 +0000 UTC m=+1757.124952761" Oct 31 13:01:50.687339 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:50.687277 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-794f7bb6dc-4dcjj" podStartSLOduration=2.687258962 podStartE2EDuration="2.687258962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:01:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:01:50.684883753 +0000 UTC m=+1757.168006679" watchObservedRunningTime="2026-10-31 13:01:50.687258962 +0000 UTC m=+1757.170381867" Oct 31 13:01:50.901802 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:50.901696 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:51.017055 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:51.017019 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-6589fb84db-9frk4"] Oct 31 13:01:51.024905 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:51.024379 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-6589fb84db-9frk4" Oct 31 13:01:51.128046 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:51.104638 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-6589fb84db-9frk4"] Oct 31 13:01:51.259348 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:51.259305 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-48dcc1-6589fb84db-9frk4\" (UID: \"fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-6589fb84db-9frk4" Oct 31 13:01:51.338253 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:51.317184 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5dws\" (UniqueName: \"kubernetes.io/projected/fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b-kube-api-access-c5dws\") pod \"payload-processing-e2e-ipp-b-48dcc1-6589fb84db-9frk4\" (UID: \"fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-6589fb84db-9frk4" Oct 31 13:01:51.338253 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:51.260053 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-48dcc1\"" Oct 31 13:01:51.338253 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:51.316956 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:51.354472 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:51.260157 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-48dcc1-dockercfg-kk89r\"" Oct 31 13:01:51.497711 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:51.475808 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:01:51.497711 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:51.496236 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-48dcc1-6589fb84db-9frk4\" (UID: \"fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-6589fb84db-9frk4" Oct 31 13:01:51.497711 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:51.496326 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5dws\" (UniqueName: \"kubernetes.io/projected/fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b-kube-api-access-c5dws\") pod \"payload-processing-e2e-ipp-b-48dcc1-6589fb84db-9frk4\" (UID: \"fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-6589fb84db-9frk4" Oct 31 13:01:51.497711 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:51.497590 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-48dcc1-6589fb84db-9frk4\" (UID: \"fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-6589fb84db-9frk4" Oct 31 13:01:51.500579 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:51.499916 6841 patch_prober.go:29] interesting pod/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 31 13:01:51.500579 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:51.499970 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" podUID="ba9a674c-9aec-46ad-9600-56952a8128c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 31 13:01:51.509829 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:51.509796 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5dws\" (UniqueName: \"kubernetes.io/projected/fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b-kube-api-access-c5dws\") pod \"payload-processing-e2e-ipp-b-48dcc1-6589fb84db-9frk4\" (UID: \"fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-6589fb84db-9frk4" Oct 31 13:01:51.652012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:51.649069 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:51.678006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:51.668777 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-6589fb84db-9frk4" Oct 31 13:01:52.008975 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:01:52.008944 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe562fe6_8f04_4d55_ab6a_4ad4aaa8fb4b.slice/crio-ed8002984da64326c4f4b2f19457d1356ca3a1a4ee75d1ba2a5be1e574a6ec7d WatchSource:0}: Error finding container ed8002984da64326c4f4b2f19457d1356ca3a1a4ee75d1ba2a5be1e574a6ec7d: Status 404 returned error can't find the container with id ed8002984da64326c4f4b2f19457d1356ca3a1a4ee75d1ba2a5be1e574a6ec7d Oct 31 13:01:52.009505 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:52.009477 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-6589fb84db-9frk4"] Oct 31 13:01:52.467006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:52.462363 6841 patch_prober.go:29] interesting pod/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 31 13:01:52.467006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:52.462427 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" podUID="ba9a674c-9aec-46ad-9600-56952a8128c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 31 13:01:52.634383 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:52.634347 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-6589fb84db-9frk4" event={"ID":"fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b","Type":"ContainerStarted","Data":"7b409ccf2816ad178319f5aab252d5d76c2409071143a78622bcc9aa40ab7a43"} Oct 31 13:01:52.634556 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:52.634410 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-6589fb84db-9frk4" event={"ID":"fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b","Type":"ContainerStarted","Data":"ed8002984da64326c4f4b2f19457d1356ca3a1a4ee75d1ba2a5be1e574a6ec7d"} Oct 31 13:01:52.634556 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:52.634438 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-6589fb84db-9frk4" Oct 31 13:01:52.676233 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:52.676182 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-6589fb84db-9frk4" podStartSLOduration=2.676166168 podStartE2EDuration="2.676166168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:01:52.659535116 +0000 UTC m=+1759.142658019" watchObservedRunningTime="2026-10-31 13:01:52.676166168 +0000 UTC m=+1759.159289069" Oct 31 13:01:52.871708 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:52.871606 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:53.215006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:53.209364 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:53.461405 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:53.461367 6841 patch_prober.go:29] interesting pod/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 31 13:01:53.461645 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:53.461617 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" podUID="ba9a674c-9aec-46ad-9600-56952a8128c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 31 13:01:54.112632 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:54.112596 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:54.365342 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:54.365200 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:54.460442 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:54.460405 6841 patch_prober.go:29] interesting pod/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 31 13:01:54.460721 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:54.460687 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" podUID="ba9a674c-9aec-46ad-9600-56952a8128c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 31 13:01:55.401071 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:55.390976 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:55.463007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:55.459395 6841 patch_prober.go:29] interesting pod/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 31 13:01:55.463007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:55.459481 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" podUID="ba9a674c-9aec-46ad-9600-56952a8128c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 31 13:01:56.406024 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:56.405417 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:56.490649 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:56.490605 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:01:56.491136 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:56.491106 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:01:56.492402 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:56.492379 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:01:57.314267 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:57.314233 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:58.239015 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:58.237518 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:58.799122 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:58.799030 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:59.038015 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:59.033527 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:01:59.832806 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:01:59.832718 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:00.198060 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:00.198028 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:00.607514 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:00.607438 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:00.818058 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:00.818026 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:01.631155 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:01.631124 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-794f7bb6dc-4dcjj" Oct 31 13:02:01.668701 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:01.668660 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:02.636908 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:02.636872 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-64bd66bc8b-9kvnt" Oct 31 13:02:02.813016 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:02.807399 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:03.700777 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:03.697370 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-6589fb84db-9frk4" Oct 31 13:02:03.831666 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:03.831632 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:04.531695 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:04.531655 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:05.663936 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:05.663899 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:05.876707 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:05.876664 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:06.711381 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:06.711349 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:06.965071 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:06.957652 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:07.210007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:07.203281 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:07.755953 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:07.755881 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:07.992971 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:07.992933 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:08.354908 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:08.354870 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:08.806165 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:08.806097 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:09.318005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:09.309598 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:09.549327 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:09.549292 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:10.207879 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:10.207846 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:11.297086 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:11.297046 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:12.184343 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:12.184293 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:12.631387 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:12.629761 6841 patch_prober.go:29] interesting pod/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": read tcp 10.134.0.2:33372->10.134.0.113:15021: read: connection reset by peer" start-of-body= Oct 31 13:02:12.631387 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:12.629828 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" podUID="cd421e14-02ab-4072-918a-e3c6077263c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": read tcp 10.134.0.2:33372->10.134.0.113:15021: read: connection reset by peer" Oct 31 13:02:13.054887 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:13.054822 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:13.062346 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:13.062321 6841 generic.go:356] "Generic (PLEG): container finished" podID="cd421e14-02ab-4072-918a-e3c6077263c8" containerID="5ee31ac384bff02e3313e7c063614a58abdc9a5ab2e8039edaa7438800835cd8" exitCode=0 Oct 31 13:02:13.062483 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:13.062370 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" event={"ID":"cd421e14-02ab-4072-918a-e3c6077263c8","Type":"ContainerDied","Data":"5ee31ac384bff02e3313e7c063614a58abdc9a5ab2e8039edaa7438800835cd8"} Oct 31 13:02:13.062802 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:13.062786 6841 scope.go:122] "RemoveContainer" containerID="5ee31ac384bff02e3313e7c063614a58abdc9a5ab2e8039edaa7438800835cd8" Oct 31 13:02:13.064497 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:13.064466 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:02:13.064631 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:13.064539 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:02:13.064631 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:13.064592 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:02:13.521006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:13.518008 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:13.743050 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:13.743015 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:13.965417 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:13.965385 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:14.089529 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:14.089482 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" event={"ID":"cd421e14-02ab-4072-918a-e3c6077263c8","Type":"ContainerStarted","Data":"44c37590ceca231b5b5ba8c3e595cee0f52dee2b714a0671f8cad56d504b48dd"} Oct 31 13:02:14.266156 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:14.257641 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:14.281677 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:14.281101 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr"] Oct 31 13:02:14.281677 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:14.281373 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" podUID="1c502702-da0b-45d4-ba77-5ae972c1bd16" containerName="istio-proxy" containerID="cri-o://6dfa34e2c9a6a902add417d53971c6b934d50782523538c27d21c25cfcdf5add" gracePeriod=30 Oct 31 13:02:14.513206 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:14.512347 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6"] Oct 31 13:02:14.513206 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:14.512585 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" podUID="3e30e333-c7fd-4aaa-9541-3a1ef29189a4" containerName="istio-proxy" containerID="cri-o://1158b55b93e76e20caa4fc7949166c62ef6013a1f74677505f9ceec92b76811e" gracePeriod=30 Oct 31 13:02:14.545286 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:14.545233 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn"] Oct 31 13:02:14.546117 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:14.545817 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" podUID="3897f656-b3a9-4d66-b894-3663c88c253d" containerName="istio-proxy" containerID="cri-o://c7c000fcad49d07c31892d965142e5219ad5168a4261310fbad6a414f7e3ed80" gracePeriod=30 Oct 31 13:02:14.723088 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:14.717220 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:02:14.723088 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:14.717340 6841 patch_prober.go:29] interesting pod/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 31 13:02:14.723088 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:14.717393 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" podUID="cd421e14-02ab-4072-918a-e3c6077263c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 31 13:02:14.738544 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:14.738510 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:15.247890 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:15.247855 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:15.566065 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:15.563548 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:15.666605 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:15.666565 6841 patch_prober.go:29] interesting pod/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 31 13:02:15.666802 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:15.666634 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" podUID="cd421e14-02ab-4072-918a-e3c6077263c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 31 13:02:16.268788 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:16.268743 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:16.475949 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:16.475913 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:16.658509 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:16.658414 6841 patch_prober.go:29] interesting pod/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 31 13:02:16.658778 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:16.658746 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" podUID="cd421e14-02ab-4072-918a-e3c6077263c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 31 13:02:17.032213 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:17.032162 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:17.332128 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:17.325636 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:17.574042 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:17.568130 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:17.660059 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:17.659518 6841 patch_prober.go:29] interesting pod/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 31 13:02:17.660059 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:17.659578 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" podUID="cd421e14-02ab-4072-918a-e3c6077263c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 31 13:02:18.660007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:18.659739 6841 patch_prober.go:29] interesting pod/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 31 13:02:18.660007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:18.659820 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" podUID="cd421e14-02ab-4072-918a-e3c6077263c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 31 13:02:18.756009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:18.753613 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:19.006360 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.006318 6841 patch_prober.go:29] interesting pod/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:02:19.006547 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.006453 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" podUID="1c502702-da0b-45d4-ba77-5ae972c1bd16" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:02:19.639811 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.639504 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:19.679780 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.672682 6841 patch_prober.go:29] interesting pod/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 31 13:02:19.679780 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.672742 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" podUID="cd421e14-02ab-4072-918a-e3c6077263c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 31 13:02:19.828085 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.828054 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:02:19.892279 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.889064 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c502702-da0b-45d4-ba77-5ae972c1bd16-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c502702-da0b-45d4-ba77-5ae972c1bd16-istio-envoy\") pod \"1c502702-da0b-45d4-ba77-5ae972c1bd16\" (UID: \"1c502702-da0b-45d4-ba77-5ae972c1bd16\") " Oct 31 13:02:19.892279 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.889188 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c502702-da0b-45d4-ba77-5ae972c1bd16-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c502702-da0b-45d4-ba77-5ae972c1bd16-credential-socket\") pod \"1c502702-da0b-45d4-ba77-5ae972c1bd16\" (UID: \"1c502702-da0b-45d4-ba77-5ae972c1bd16\") " Oct 31 13:02:19.892279 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.889246 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c502702-da0b-45d4-ba77-5ae972c1bd16-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c502702-da0b-45d4-ba77-5ae972c1bd16-istio-podinfo\") pod \"1c502702-da0b-45d4-ba77-5ae972c1bd16\" (UID: \"1c502702-da0b-45d4-ba77-5ae972c1bd16\") " Oct 31 13:02:19.892279 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.889302 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c502702-da0b-45d4-ba77-5ae972c1bd16-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c502702-da0b-45d4-ba77-5ae972c1bd16-istiod-ca-cert\") pod \"1c502702-da0b-45d4-ba77-5ae972c1bd16\" (UID: \"1c502702-da0b-45d4-ba77-5ae972c1bd16\") " Oct 31 13:02:19.892279 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.889359 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c502702-da0b-45d4-ba77-5ae972c1bd16-kube-api-access-4mzx6\" (UniqueName: \"kubernetes.io/projected/1c502702-da0b-45d4-ba77-5ae972c1bd16-kube-api-access-4mzx6\") pod \"1c502702-da0b-45d4-ba77-5ae972c1bd16\" (UID: \"1c502702-da0b-45d4-ba77-5ae972c1bd16\") " Oct 31 13:02:19.892279 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.889400 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c502702-da0b-45d4-ba77-5ae972c1bd16-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c502702-da0b-45d4-ba77-5ae972c1bd16-istio-data\") pod \"1c502702-da0b-45d4-ba77-5ae972c1bd16\" (UID: \"1c502702-da0b-45d4-ba77-5ae972c1bd16\") " Oct 31 13:02:19.892279 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.889454 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c502702-da0b-45d4-ba77-5ae972c1bd16-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c502702-da0b-45d4-ba77-5ae972c1bd16-workload-socket\") pod \"1c502702-da0b-45d4-ba77-5ae972c1bd16\" (UID: \"1c502702-da0b-45d4-ba77-5ae972c1bd16\") " Oct 31 13:02:19.892279 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.889493 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c502702-da0b-45d4-ba77-5ae972c1bd16-istio-token\" (UniqueName: \"kubernetes.io/projected/1c502702-da0b-45d4-ba77-5ae972c1bd16-istio-token\") pod \"1c502702-da0b-45d4-ba77-5ae972c1bd16\" (UID: \"1c502702-da0b-45d4-ba77-5ae972c1bd16\") " Oct 31 13:02:19.892279 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.889524 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c502702-da0b-45d4-ba77-5ae972c1bd16-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c502702-da0b-45d4-ba77-5ae972c1bd16-workload-certs\") pod \"1c502702-da0b-45d4-ba77-5ae972c1bd16\" (UID: \"1c502702-da0b-45d4-ba77-5ae972c1bd16\") " Oct 31 13:02:19.892279 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.890456 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c502702-da0b-45d4-ba77-5ae972c1bd16-workload-certs" pod "1c502702-da0b-45d4-ba77-5ae972c1bd16" (UID: "1c502702-da0b-45d4-ba77-5ae972c1bd16"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:19.892279 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.891026 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c502702-da0b-45d4-ba77-5ae972c1bd16-workload-socket" pod "1c502702-da0b-45d4-ba77-5ae972c1bd16" (UID: "1c502702-da0b-45d4-ba77-5ae972c1bd16"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:19.892279 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.891200 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c502702-da0b-45d4-ba77-5ae972c1bd16-istio-data" pod "1c502702-da0b-45d4-ba77-5ae972c1bd16" (UID: "1c502702-da0b-45d4-ba77-5ae972c1bd16"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:19.892279 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.891721 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c502702-da0b-45d4-ba77-5ae972c1bd16-credential-socket" pod "1c502702-da0b-45d4-ba77-5ae972c1bd16" (UID: "1c502702-da0b-45d4-ba77-5ae972c1bd16"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:19.892279 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.891806 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c502702-da0b-45d4-ba77-5ae972c1bd16-istiod-ca-cert" pod "1c502702-da0b-45d4-ba77-5ae972c1bd16" (UID: "1c502702-da0b-45d4-ba77-5ae972c1bd16"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:02:19.926918 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.924056 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c502702-da0b-45d4-ba77-5ae972c1bd16-istio-token" pod "1c502702-da0b-45d4-ba77-5ae972c1bd16" (UID: "1c502702-da0b-45d4-ba77-5ae972c1bd16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:19.926918 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.926604 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c502702-da0b-45d4-ba77-5ae972c1bd16-istio-envoy" pod "1c502702-da0b-45d4-ba77-5ae972c1bd16" (UID: "1c502702-da0b-45d4-ba77-5ae972c1bd16"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:19.928130 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.928087 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c502702-da0b-45d4-ba77-5ae972c1bd16-istio-podinfo" pod "1c502702-da0b-45d4-ba77-5ae972c1bd16" (UID: "1c502702-da0b-45d4-ba77-5ae972c1bd16"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:02:19.929798 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.929740 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c502702-da0b-45d4-ba77-5ae972c1bd16-kube-api-access-4mzx6" pod "1c502702-da0b-45d4-ba77-5ae972c1bd16" (UID: "1c502702-da0b-45d4-ba77-5ae972c1bd16"). InnerVolumeSpecName "kube-api-access-4mzx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:19.979828 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.979784 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" Oct 31 13:02:19.991933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.990769 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3897f656-b3a9-4d66-b894-3663c88c253d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3897f656-b3a9-4d66-b894-3663c88c253d-workload-socket\") pod \"3897f656-b3a9-4d66-b894-3663c88c253d\" (UID: \"3897f656-b3a9-4d66-b894-3663c88c253d\") " Oct 31 13:02:19.991933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.990868 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3897f656-b3a9-4d66-b894-3663c88c253d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3897f656-b3a9-4d66-b894-3663c88c253d-istio-podinfo\") pod \"3897f656-b3a9-4d66-b894-3663c88c253d\" (UID: \"3897f656-b3a9-4d66-b894-3663c88c253d\") " Oct 31 13:02:19.991933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.990893 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3897f656-b3a9-4d66-b894-3663c88c253d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3897f656-b3a9-4d66-b894-3663c88c253d-istio-data\") pod \"3897f656-b3a9-4d66-b894-3663c88c253d\" (UID: \"3897f656-b3a9-4d66-b894-3663c88c253d\") " Oct 31 13:02:19.991933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.990954 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3897f656-b3a9-4d66-b894-3663c88c253d-kube-api-access-8t7f5\" (UniqueName: \"kubernetes.io/projected/3897f656-b3a9-4d66-b894-3663c88c253d-kube-api-access-8t7f5\") pod \"3897f656-b3a9-4d66-b894-3663c88c253d\" (UID: \"3897f656-b3a9-4d66-b894-3663c88c253d\") " Oct 31 13:02:19.991933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.990999 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3897f656-b3a9-4d66-b894-3663c88c253d-istio-token\" (UniqueName: \"kubernetes.io/projected/3897f656-b3a9-4d66-b894-3663c88c253d-istio-token\") pod \"3897f656-b3a9-4d66-b894-3663c88c253d\" (UID: \"3897f656-b3a9-4d66-b894-3663c88c253d\") " Oct 31 13:02:19.991933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.991065 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3897f656-b3a9-4d66-b894-3663c88c253d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3897f656-b3a9-4d66-b894-3663c88c253d-istiod-ca-cert\") pod \"3897f656-b3a9-4d66-b894-3663c88c253d\" (UID: \"3897f656-b3a9-4d66-b894-3663c88c253d\") " Oct 31 13:02:19.991933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.991142 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3897f656-b3a9-4d66-b894-3663c88c253d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3897f656-b3a9-4d66-b894-3663c88c253d-credential-socket\") pod \"3897f656-b3a9-4d66-b894-3663c88c253d\" (UID: \"3897f656-b3a9-4d66-b894-3663c88c253d\") " Oct 31 13:02:19.991933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.991195 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3897f656-b3a9-4d66-b894-3663c88c253d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3897f656-b3a9-4d66-b894-3663c88c253d-istio-envoy\") pod \"3897f656-b3a9-4d66-b894-3663c88c253d\" (UID: \"3897f656-b3a9-4d66-b894-3663c88c253d\") " Oct 31 13:02:19.991933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.991220 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3897f656-b3a9-4d66-b894-3663c88c253d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3897f656-b3a9-4d66-b894-3663c88c253d-workload-certs\") pod \"3897f656-b3a9-4d66-b894-3663c88c253d\" (UID: \"3897f656-b3a9-4d66-b894-3663c88c253d\") " Oct 31 13:02:19.991933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.991698 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c502702-da0b-45d4-ba77-5ae972c1bd16-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:19.991933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.991713 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mzx6\" (UniqueName: \"kubernetes.io/projected/1c502702-da0b-45d4-ba77-5ae972c1bd16-kube-api-access-4mzx6\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:19.991933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.991723 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c502702-da0b-45d4-ba77-5ae972c1bd16-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:19.991933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.991732 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c502702-da0b-45d4-ba77-5ae972c1bd16-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:19.991933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.991741 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c502702-da0b-45d4-ba77-5ae972c1bd16-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:19.991933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.991750 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c502702-da0b-45d4-ba77-5ae972c1bd16-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:19.991933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.991759 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c502702-da0b-45d4-ba77-5ae972c1bd16-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:19.991933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.991768 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c502702-da0b-45d4-ba77-5ae972c1bd16-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:19.991933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.991776 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c502702-da0b-45d4-ba77-5ae972c1bd16-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:19.993679 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.993053 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3897f656-b3a9-4d66-b894-3663c88c253d-workload-certs" pod "3897f656-b3a9-4d66-b894-3663c88c253d" (UID: "3897f656-b3a9-4d66-b894-3663c88c253d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:19.993679 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.993610 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3897f656-b3a9-4d66-b894-3663c88c253d-istio-data" pod "3897f656-b3a9-4d66-b894-3663c88c253d" (UID: "3897f656-b3a9-4d66-b894-3663c88c253d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:19.993886 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.993717 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3897f656-b3a9-4d66-b894-3663c88c253d-credential-socket" pod "3897f656-b3a9-4d66-b894-3663c88c253d" (UID: "3897f656-b3a9-4d66-b894-3663c88c253d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:19.993886 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.993807 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3897f656-b3a9-4d66-b894-3663c88c253d-workload-socket" pod "3897f656-b3a9-4d66-b894-3663c88c253d" (UID: "3897f656-b3a9-4d66-b894-3663c88c253d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:19.993959 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.993940 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3897f656-b3a9-4d66-b894-3663c88c253d-istiod-ca-cert" pod "3897f656-b3a9-4d66-b894-3663c88c253d" (UID: "3897f656-b3a9-4d66-b894-3663c88c253d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:02:19.996062 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.996027 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3897f656-b3a9-4d66-b894-3663c88c253d-istio-podinfo" pod "3897f656-b3a9-4d66-b894-3663c88c253d" (UID: "3897f656-b3a9-4d66-b894-3663c88c253d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:02:19.996850 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.996826 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3897f656-b3a9-4d66-b894-3663c88c253d-istio-envoy" pod "3897f656-b3a9-4d66-b894-3663c88c253d" (UID: "3897f656-b3a9-4d66-b894-3663c88c253d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:19.997076 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.997051 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3897f656-b3a9-4d66-b894-3663c88c253d-istio-token" pod "3897f656-b3a9-4d66-b894-3663c88c253d" (UID: "3897f656-b3a9-4d66-b894-3663c88c253d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:19.999055 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:19.999015 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3897f656-b3a9-4d66-b894-3663c88c253d-kube-api-access-8t7f5" pod "3897f656-b3a9-4d66-b894-3663c88c253d" (UID: "3897f656-b3a9-4d66-b894-3663c88c253d"). InnerVolumeSpecName "kube-api-access-8t7f5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:20.017079 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.016724 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" Oct 31 13:02:20.080556 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:02:20.080515 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:02:20.092577 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.092544 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-kube-api-access-92tzv\" (UniqueName: \"kubernetes.io/projected/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-kube-api-access-92tzv\") pod \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\" (UID: \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\") " Oct 31 13:02:20.092733 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.092637 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-istio-podinfo\") pod \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\" (UID: \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\") " Oct 31 13:02:20.092810 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.092756 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-istio-token\" (UniqueName: \"kubernetes.io/projected/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-istio-token\") pod \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\" (UID: \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\") " Oct 31 13:02:20.093250 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.092879 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-workload-certs\") pod \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\" (UID: \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\") " Oct 31 13:02:20.093447 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.093380 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-workload-certs" pod "3e30e333-c7fd-4aaa-9541-3a1ef29189a4" (UID: "3e30e333-c7fd-4aaa-9541-3a1ef29189a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:20.093447 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.093391 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-istio-data\") pod \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\" (UID: \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\") " Oct 31 13:02:20.093596 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.093472 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-istio-envoy\") pod \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\" (UID: \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\") " Oct 31 13:02:20.093596 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.093572 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-istio-data" pod "3e30e333-c7fd-4aaa-9541-3a1ef29189a4" (UID: "3e30e333-c7fd-4aaa-9541-3a1ef29189a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:20.093837 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.093506 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-workload-socket\") pod \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\" (UID: \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\") " Oct 31 13:02:20.093922 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.093821 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-workload-socket" pod "3e30e333-c7fd-4aaa-9541-3a1ef29189a4" (UID: "3e30e333-c7fd-4aaa-9541-3a1ef29189a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:20.093922 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.093913 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-credential-socket\") pod \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\" (UID: \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\") " Oct 31 13:02:20.094031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.093956 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-istiod-ca-cert\") pod \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\" (UID: \"3e30e333-c7fd-4aaa-9541-3a1ef29189a4\") " Oct 31 13:02:20.094174 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.094156 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-credential-socket" pod "3e30e333-c7fd-4aaa-9541-3a1ef29189a4" (UID: "3e30e333-c7fd-4aaa-9541-3a1ef29189a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:20.094456 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.094438 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-istiod-ca-cert" pod "3e30e333-c7fd-4aaa-9541-3a1ef29189a4" (UID: "3e30e333-c7fd-4aaa-9541-3a1ef29189a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:02:20.094689 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.094672 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3897f656-b3a9-4d66-b894-3663c88c253d-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:20.094761 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.094694 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3897f656-b3a9-4d66-b894-3663c88c253d-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:20.094761 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.094709 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:20.094761 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.094724 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3897f656-b3a9-4d66-b894-3663c88c253d-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:20.094761 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.094737 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3897f656-b3a9-4d66-b894-3663c88c253d-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:20.094761 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.094751 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:20.095038 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.094767 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:20.095038 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.094781 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3897f656-b3a9-4d66-b894-3663c88c253d-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:20.095038 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.094795 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:20.095038 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.094809 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:20.095038 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.094823 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3897f656-b3a9-4d66-b894-3663c88c253d-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:20.095038 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.094837 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3897f656-b3a9-4d66-b894-3663c88c253d-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:20.095038 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.094851 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t7f5\" (UniqueName: \"kubernetes.io/projected/3897f656-b3a9-4d66-b894-3663c88c253d-kube-api-access-8t7f5\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:20.095038 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.094867 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3897f656-b3a9-4d66-b894-3663c88c253d-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:20.095362 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.095231 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-kube-api-access-92tzv" pod "3e30e333-c7fd-4aaa-9541-3a1ef29189a4" (UID: "3e30e333-c7fd-4aaa-9541-3a1ef29189a4"). InnerVolumeSpecName "kube-api-access-92tzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:20.095362 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.095303 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-istio-podinfo" pod "3e30e333-c7fd-4aaa-9541-3a1ef29189a4" (UID: "3e30e333-c7fd-4aaa-9541-3a1ef29189a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:02:20.095733 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.095709 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-istio-envoy" pod "3e30e333-c7fd-4aaa-9541-3a1ef29189a4" (UID: "3e30e333-c7fd-4aaa-9541-3a1ef29189a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:20.095769 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.095716 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-istio-token" pod "3e30e333-c7fd-4aaa-9541-3a1ef29189a4" (UID: "3e30e333-c7fd-4aaa-9541-3a1ef29189a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:20.196240 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.196211 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:20.196240 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.196238 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:20.196439 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.196248 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:20.196439 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.196257 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92tzv\" (UniqueName: \"kubernetes.io/projected/3e30e333-c7fd-4aaa-9541-3a1ef29189a4-kube-api-access-92tzv\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:20.312607 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.312537 6841 generic.go:356] "Generic (PLEG): container finished" podID="1c502702-da0b-45d4-ba77-5ae972c1bd16" containerID="6dfa34e2c9a6a902add417d53971c6b934d50782523538c27d21c25cfcdf5add" exitCode=0 Oct 31 13:02:20.312750 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.312599 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" Oct 31 13:02:20.312750 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.312622 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" event={"ID":"1c502702-da0b-45d4-ba77-5ae972c1bd16","Type":"ContainerDied","Data":"6dfa34e2c9a6a902add417d53971c6b934d50782523538c27d21c25cfcdf5add"} Oct 31 13:02:20.312750 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.312674 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr" event={"ID":"1c502702-da0b-45d4-ba77-5ae972c1bd16","Type":"ContainerDied","Data":"45129d45580b3e372b04db17c9aa512dbe3ad544836a2af65251543bde9e1943"} Oct 31 13:02:20.312750 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.312702 6841 scope.go:122] "RemoveContainer" containerID="6dfa34e2c9a6a902add417d53971c6b934d50782523538c27d21c25cfcdf5add" Oct 31 13:02:20.316167 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.314805 6841 generic.go:356] "Generic (PLEG): container finished" podID="3897f656-b3a9-4d66-b894-3663c88c253d" containerID="c7c000fcad49d07c31892d965142e5219ad5168a4261310fbad6a414f7e3ed80" exitCode=0 Oct 31 13:02:20.316167 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.314870 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" event={"ID":"3897f656-b3a9-4d66-b894-3663c88c253d","Type":"ContainerDied","Data":"c7c000fcad49d07c31892d965142e5219ad5168a4261310fbad6a414f7e3ed80"} Oct 31 13:02:20.316167 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.314903 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" event={"ID":"3897f656-b3a9-4d66-b894-3663c88c253d","Type":"ContainerDied","Data":"0e6d9cbc2805f8ca0135a0384dad8b4131e839d303eb762b815b45436127f060"} Oct 31 13:02:20.316167 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.314965 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn" Oct 31 13:02:20.322191 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.322158 6841 generic.go:356] "Generic (PLEG): container finished" podID="3e30e333-c7fd-4aaa-9541-3a1ef29189a4" containerID="1158b55b93e76e20caa4fc7949166c62ef6013a1f74677505f9ceec92b76811e" exitCode=0 Oct 31 13:02:20.322662 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.322640 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" event={"ID":"3e30e333-c7fd-4aaa-9541-3a1ef29189a4","Type":"ContainerDied","Data":"1158b55b93e76e20caa4fc7949166c62ef6013a1f74677505f9ceec92b76811e"} Oct 31 13:02:20.322790 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.322770 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" event={"ID":"3e30e333-c7fd-4aaa-9541-3a1ef29189a4","Type":"ContainerDied","Data":"f8867c4653a4b8fd6a0f11ea5c7296f2c00416fce6d9d4626afb99742f0220c8"} Oct 31 13:02:20.323095 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.323077 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6" Oct 31 13:02:20.375045 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.375017 6841 scope.go:122] "RemoveContainer" containerID="062d41d9f05c95d634be86d75b50e597350d8238c78dac954f601aaccfaf0996" Oct 31 13:02:20.385409 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.385384 6841 scope.go:122] "RemoveContainer" containerID="6dfa34e2c9a6a902add417d53971c6b934d50782523538c27d21c25cfcdf5add" Oct 31 13:02:20.385730 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:02:20.385705 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dfa34e2c9a6a902add417d53971c6b934d50782523538c27d21c25cfcdf5add\": container with ID starting with 6dfa34e2c9a6a902add417d53971c6b934d50782523538c27d21c25cfcdf5add not found: ID does not exist" containerID="6dfa34e2c9a6a902add417d53971c6b934d50782523538c27d21c25cfcdf5add" Oct 31 13:02:20.385811 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.385745 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dfa34e2c9a6a902add417d53971c6b934d50782523538c27d21c25cfcdf5add"} err="failed to get container status \"6dfa34e2c9a6a902add417d53971c6b934d50782523538c27d21c25cfcdf5add\": rpc error: code = NotFound desc = could not find container \"6dfa34e2c9a6a902add417d53971c6b934d50782523538c27d21c25cfcdf5add\": container with ID starting with 6dfa34e2c9a6a902add417d53971c6b934d50782523538c27d21c25cfcdf5add not found: ID does not exist" Oct 31 13:02:20.385811 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.385771 6841 scope.go:122] "RemoveContainer" containerID="062d41d9f05c95d634be86d75b50e597350d8238c78dac954f601aaccfaf0996" Oct 31 13:02:20.386084 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:02:20.386066 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"062d41d9f05c95d634be86d75b50e597350d8238c78dac954f601aaccfaf0996\": container with ID starting with 062d41d9f05c95d634be86d75b50e597350d8238c78dac954f601aaccfaf0996 not found: ID does not exist" containerID="062d41d9f05c95d634be86d75b50e597350d8238c78dac954f601aaccfaf0996" Oct 31 13:02:20.386168 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.386094 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"062d41d9f05c95d634be86d75b50e597350d8238c78dac954f601aaccfaf0996"} err="failed to get container status \"062d41d9f05c95d634be86d75b50e597350d8238c78dac954f601aaccfaf0996\": rpc error: code = NotFound desc = could not find container \"062d41d9f05c95d634be86d75b50e597350d8238c78dac954f601aaccfaf0996\": container with ID starting with 062d41d9f05c95d634be86d75b50e597350d8238c78dac954f601aaccfaf0996 not found: ID does not exist" Oct 31 13:02:20.386168 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.386114 6841 scope.go:122] "RemoveContainer" containerID="c7c000fcad49d07c31892d965142e5219ad5168a4261310fbad6a414f7e3ed80" Oct 31 13:02:20.395820 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.395797 6841 scope.go:122] "RemoveContainer" containerID="c7c000fcad49d07c31892d965142e5219ad5168a4261310fbad6a414f7e3ed80" Oct 31 13:02:20.396156 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:02:20.396136 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7c000fcad49d07c31892d965142e5219ad5168a4261310fbad6a414f7e3ed80\": container with ID starting with c7c000fcad49d07c31892d965142e5219ad5168a4261310fbad6a414f7e3ed80 not found: ID does not exist" containerID="c7c000fcad49d07c31892d965142e5219ad5168a4261310fbad6a414f7e3ed80" Oct 31 13:02:20.396260 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.396169 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7c000fcad49d07c31892d965142e5219ad5168a4261310fbad6a414f7e3ed80"} err="failed to get container status \"c7c000fcad49d07c31892d965142e5219ad5168a4261310fbad6a414f7e3ed80\": rpc error: code = NotFound desc = could not find container \"c7c000fcad49d07c31892d965142e5219ad5168a4261310fbad6a414f7e3ed80\": container with ID starting with c7c000fcad49d07c31892d965142e5219ad5168a4261310fbad6a414f7e3ed80 not found: ID does not exist" Oct 31 13:02:20.396260 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.396186 6841 scope.go:122] "RemoveContainer" containerID="1158b55b93e76e20caa4fc7949166c62ef6013a1f74677505f9ceec92b76811e" Oct 31 13:02:20.401262 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.401189 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn"] Oct 31 13:02:20.405341 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.405320 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-8dcpn"] Oct 31 13:02:20.407368 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.407197 6841 scope.go:122] "RemoveContainer" containerID="1158b55b93e76e20caa4fc7949166c62ef6013a1f74677505f9ceec92b76811e" Oct 31 13:02:20.407505 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:02:20.407487 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1158b55b93e76e20caa4fc7949166c62ef6013a1f74677505f9ceec92b76811e\": container with ID starting with 1158b55b93e76e20caa4fc7949166c62ef6013a1f74677505f9ceec92b76811e not found: ID does not exist" containerID="1158b55b93e76e20caa4fc7949166c62ef6013a1f74677505f9ceec92b76811e" Oct 31 13:02:20.407553 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.407518 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1158b55b93e76e20caa4fc7949166c62ef6013a1f74677505f9ceec92b76811e"} err="failed to get container status \"1158b55b93e76e20caa4fc7949166c62ef6013a1f74677505f9ceec92b76811e\": rpc error: code = NotFound desc = could not find container \"1158b55b93e76e20caa4fc7949166c62ef6013a1f74677505f9ceec92b76811e\": container with ID starting with 1158b55b93e76e20caa4fc7949166c62ef6013a1f74677505f9ceec92b76811e not found: ID does not exist" Oct 31 13:02:20.418693 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.418670 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6"] Oct 31 13:02:20.429785 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.429763 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-dlhs6"] Oct 31 13:02:20.460198 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.460030 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr"] Oct 31 13:02:20.465006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.462806 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-dc2db4-openshift-default-749d87b8dc-4nqfr"] Oct 31 13:02:20.536140 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.536104 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:20.658010 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.657411 6841 patch_prober.go:29] interesting pod/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 31 13:02:20.658010 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:20.657475 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" podUID="cd421e14-02ab-4072-918a-e3c6077263c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 31 13:02:21.124374 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:21.124336 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:21.164946 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:21.164910 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c"] Oct 31 13:02:21.165180 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:21.165149 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c" podUID="a8afce0f-eb7a-45c5-b837-2024fd7c563a" containerName="maas-api" containerID="cri-o://188b352140aa98b636eb865382c17d8c072f0f5397e802a79cb0756e412c1bd1" gracePeriod=30 Oct 31 13:02:21.257606 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:21.257567 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-5d88b777bb-dnwdj"] Oct 31 13:02:21.258100 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:21.258044 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-5d88b777bb-dnwdj" podUID="e3fee6c8-5e84-4011-a626-7a3da7239b06" containerName="payload-processing" containerID="cri-o://489586f21464e62c9f1a33658064165dcad7af3027cd37708d8393c9ea65e31c" gracePeriod=30 Oct 31 13:02:21.336180 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:21.336151 6841 generic.go:356] "Generic (PLEG): container finished" podID="a8afce0f-eb7a-45c5-b837-2024fd7c563a" containerID="188b352140aa98b636eb865382c17d8c072f0f5397e802a79cb0756e412c1bd1" exitCode=0 Oct 31 13:02:21.336341 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:21.336228 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c" event={"ID":"a8afce0f-eb7a-45c5-b837-2024fd7c563a","Type":"ContainerDied","Data":"188b352140aa98b636eb865382c17d8c072f0f5397e802a79cb0756e412c1bd1"} Oct 31 13:02:21.416808 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:21.416717 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:21.688048 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:21.676184 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:21.724529 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:21.710214 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c" Oct 31 13:02:21.759099 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:21.758664 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c502702-da0b-45d4-ba77-5ae972c1bd16" path="/var/lib/kubelet/pods/1c502702-da0b-45d4-ba77-5ae972c1bd16/volumes" Oct 31 13:02:21.762269 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:21.759857 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3897f656-b3a9-4d66-b894-3663c88c253d" path="/var/lib/kubelet/pods/3897f656-b3a9-4d66-b894-3663c88c253d/volumes" Oct 31 13:02:21.772923 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:21.763174 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e30e333-c7fd-4aaa-9541-3a1ef29189a4" path="/var/lib/kubelet/pods/3e30e333-c7fd-4aaa-9541-3a1ef29189a4/volumes" Oct 31 13:02:21.875489 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:21.861783 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8afce0f-eb7a-45c5-b837-2024fd7c563a-kube-api-access-zfcdc\" (UniqueName: \"kubernetes.io/projected/a8afce0f-eb7a-45c5-b837-2024fd7c563a-kube-api-access-zfcdc\") pod \"a8afce0f-eb7a-45c5-b837-2024fd7c563a\" (UID: \"a8afce0f-eb7a-45c5-b837-2024fd7c563a\") " Oct 31 13:02:21.875489 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:21.861903 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8afce0f-eb7a-45c5-b837-2024fd7c563a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8afce0f-eb7a-45c5-b837-2024fd7c563a-metrics-certs\") pod \"a8afce0f-eb7a-45c5-b837-2024fd7c563a\" (UID: \"a8afce0f-eb7a-45c5-b837-2024fd7c563a\") " Oct 31 13:02:21.875489 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:21.862010 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8afce0f-eb7a-45c5-b837-2024fd7c563a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8afce0f-eb7a-45c5-b837-2024fd7c563a-maas-api-tls\") pod \"a8afce0f-eb7a-45c5-b837-2024fd7c563a\" (UID: \"a8afce0f-eb7a-45c5-b837-2024fd7c563a\") " Oct 31 13:02:21.875489 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:21.867059 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8afce0f-eb7a-45c5-b837-2024fd7c563a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8afce0f-eb7a-45c5-b837-2024fd7c563a-openshift-service-ca\") pod \"a8afce0f-eb7a-45c5-b837-2024fd7c563a\" (UID: \"a8afce0f-eb7a-45c5-b837-2024fd7c563a\") " Oct 31 13:02:21.876696 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:21.876272 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-5d88b777bb-dnwdj" Oct 31 13:02:21.893482 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:21.893284 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8afce0f-eb7a-45c5-b837-2024fd7c563a-openshift-service-ca" pod "a8afce0f-eb7a-45c5-b837-2024fd7c563a" (UID: "a8afce0f-eb7a-45c5-b837-2024fd7c563a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:02:21.895025 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:21.894959 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8afce0f-eb7a-45c5-b837-2024fd7c563a-maas-api-tls" pod "a8afce0f-eb7a-45c5-b837-2024fd7c563a" (UID: "a8afce0f-eb7a-45c5-b837-2024fd7c563a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:02:21.902908 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:21.902871 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8afce0f-eb7a-45c5-b837-2024fd7c563a-kube-api-access-zfcdc" pod "a8afce0f-eb7a-45c5-b837-2024fd7c563a" (UID: "a8afce0f-eb7a-45c5-b837-2024fd7c563a"). InnerVolumeSpecName "kube-api-access-zfcdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:21.909875 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:21.909845 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8afce0f-eb7a-45c5-b837-2024fd7c563a-metrics-certs" pod "a8afce0f-eb7a-45c5-b837-2024fd7c563a" (UID: "a8afce0f-eb7a-45c5-b837-2024fd7c563a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:02:21.947032 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:21.945844 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:21.970872 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:21.970185 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8afce0f-eb7a-45c5-b837-2024fd7c563a-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:21.971140 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:21.971119 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8afce0f-eb7a-45c5-b837-2024fd7c563a-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:21.971240 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:21.971228 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfcdc\" (UniqueName: \"kubernetes.io/projected/a8afce0f-eb7a-45c5-b837-2024fd7c563a-kube-api-access-zfcdc\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:21.971333 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:21.971323 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8afce0f-eb7a-45c5-b837-2024fd7c563a-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:22.073685 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:22.072747 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3fee6c8-5e84-4011-a626-7a3da7239b06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3fee6c8-5e84-4011-a626-7a3da7239b06-plugins-config-volume\") pod \"e3fee6c8-5e84-4011-a626-7a3da7239b06\" (UID: \"e3fee6c8-5e84-4011-a626-7a3da7239b06\") " Oct 31 13:02:22.073685 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:22.072803 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3fee6c8-5e84-4011-a626-7a3da7239b06-kube-api-access-67n9q\" (UniqueName: \"kubernetes.io/projected/e3fee6c8-5e84-4011-a626-7a3da7239b06-kube-api-access-67n9q\") pod \"e3fee6c8-5e84-4011-a626-7a3da7239b06\" (UID: \"e3fee6c8-5e84-4011-a626-7a3da7239b06\") " Oct 31 13:02:22.074332 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:22.074308 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3fee6c8-5e84-4011-a626-7a3da7239b06-plugins-config-volume" pod "e3fee6c8-5e84-4011-a626-7a3da7239b06" (UID: "e3fee6c8-5e84-4011-a626-7a3da7239b06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:02:22.076155 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:22.076132 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3fee6c8-5e84-4011-a626-7a3da7239b06-kube-api-access-67n9q" pod "e3fee6c8-5e84-4011-a626-7a3da7239b06" (UID: "e3fee6c8-5e84-4011-a626-7a3da7239b06"). InnerVolumeSpecName "kube-api-access-67n9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:22.110504 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:22.110452 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:02:22.110928 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:22.110907 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:02:22.111634 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:22.111614 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:02:22.174146 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:22.174114 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3fee6c8-5e84-4011-a626-7a3da7239b06-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:22.174146 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:22.174141 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67n9q\" (UniqueName: \"kubernetes.io/projected/e3fee6c8-5e84-4011-a626-7a3da7239b06-kube-api-access-67n9q\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:22.349936 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:22.349847 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c" Oct 31 13:02:22.349936 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:22.349859 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c" event={"ID":"a8afce0f-eb7a-45c5-b837-2024fd7c563a","Type":"ContainerDied","Data":"e2abec19384b30f4f10517110aba0812eccd17f2c496b1d6342a8aa4ae9a271e"} Oct 31 13:02:22.349936 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:22.349915 6841 scope.go:122] "RemoveContainer" containerID="188b352140aa98b636eb865382c17d8c072f0f5397e802a79cb0756e412c1bd1" Oct 31 13:02:22.351843 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:22.351816 6841 generic.go:356] "Generic (PLEG): container finished" podID="e3fee6c8-5e84-4011-a626-7a3da7239b06" containerID="489586f21464e62c9f1a33658064165dcad7af3027cd37708d8393c9ea65e31c" exitCode=0 Oct 31 13:02:22.351946 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:22.351902 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-5d88b777bb-dnwdj" Oct 31 13:02:22.352781 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:22.352755 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-5d88b777bb-dnwdj" event={"ID":"e3fee6c8-5e84-4011-a626-7a3da7239b06","Type":"ContainerDied","Data":"489586f21464e62c9f1a33658064165dcad7af3027cd37708d8393c9ea65e31c"} Oct 31 13:02:22.352871 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:22.352808 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-5d88b777bb-dnwdj" event={"ID":"e3fee6c8-5e84-4011-a626-7a3da7239b06","Type":"ContainerDied","Data":"0402498952dd9678242763e3355efecbbe94ee61d06dfb23c1641e164e555e87"} Oct 31 13:02:22.364057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:22.364040 6841 scope.go:122] "RemoveContainer" containerID="489586f21464e62c9f1a33658064165dcad7af3027cd37708d8393c9ea65e31c" Oct 31 13:02:22.377810 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:22.377790 6841 scope.go:122] "RemoveContainer" containerID="489586f21464e62c9f1a33658064165dcad7af3027cd37708d8393c9ea65e31c" Oct 31 13:02:22.378171 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:02:22.378147 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"489586f21464e62c9f1a33658064165dcad7af3027cd37708d8393c9ea65e31c\": container with ID starting with 489586f21464e62c9f1a33658064165dcad7af3027cd37708d8393c9ea65e31c not found: ID does not exist" containerID="489586f21464e62c9f1a33658064165dcad7af3027cd37708d8393c9ea65e31c" Oct 31 13:02:22.378270 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:22.378179 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"489586f21464e62c9f1a33658064165dcad7af3027cd37708d8393c9ea65e31c"} err="failed to get container status \"489586f21464e62c9f1a33658064165dcad7af3027cd37708d8393c9ea65e31c\": rpc error: code = NotFound desc = could not find container \"489586f21464e62c9f1a33658064165dcad7af3027cd37708d8393c9ea65e31c\": container with ID starting with 489586f21464e62c9f1a33658064165dcad7af3027cd37708d8393c9ea65e31c not found: ID does not exist" Oct 31 13:02:22.385718 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:22.385693 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-5d88b777bb-dnwdj"] Oct 31 13:02:22.397525 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:22.397501 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-5d88b777bb-dnwdj"] Oct 31 13:02:22.415569 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:22.415546 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c"] Oct 31 13:02:22.418915 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:22.418889 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc9ae8-6cbd96b84d-pl69c"] Oct 31 13:02:22.699923 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:22.699895 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z"] Oct 31 13:02:22.700162 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:22.700138 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z" podUID="4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585" containerName="maas-api" containerID="cri-o://52c869b4451aace43501bd7ead4968df9fcd0f8a179ab092c608bfef344bc546" gracePeriod=30 Oct 31 13:02:22.801753 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:22.801711 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ccf963-cff4fd7fd-dgj5x"] Oct 31 13:02:22.802157 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:22.802121 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-cff4fd7fd-dgj5x" podUID="e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9" containerName="payload-processing" containerID="cri-o://75ac6696050ce370e2cc573e1ff405d72e03cd5e5156be3f54f57bca703dbb51" gracePeriod=30 Oct 31 13:02:22.819104 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:22.818948 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-875df4b95-k59g5"] Oct 31 13:02:22.819377 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:22.819246 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-875df4b95-k59g5" podUID="399ac99e-25a1-44d3-b774-04991fbe4f47" containerName="payload-pre-processing" containerID="cri-o://21a16ed537f35e4190ec4314338b709b228a3c074a92bd8f82e3767ae1c03261" gracePeriod=30 Oct 31 13:02:22.837513 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:22.837483 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:23.053313 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.053022 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:23.066042 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.066014 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z" Oct 31 13:02:23.098803 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.098763 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585-metrics-certs\") pod \"4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585\" (UID: \"4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585\") " Oct 31 13:02:23.098972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.098819 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585-maas-api-tls\") pod \"4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585\" (UID: \"4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585\") " Oct 31 13:02:23.098972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.098899 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585-kube-api-access-wlbpf\" (UniqueName: \"kubernetes.io/projected/4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585-kube-api-access-wlbpf\") pod \"4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585\" (UID: \"4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585\") " Oct 31 13:02:23.099110 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.099052 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585-openshift-service-ca\") pod \"4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585\" (UID: \"4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585\") " Oct 31 13:02:23.100248 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.100222 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585-openshift-service-ca" pod "4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585" (UID: "4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:02:23.107921 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.106431 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585-maas-api-tls" pod "4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585" (UID: "4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:02:23.108421 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.108395 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585-metrics-certs" pod "4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585" (UID: "4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:02:23.108697 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.108668 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585-kube-api-access-wlbpf" pod "4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585" (UID: "4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585"). InnerVolumeSpecName "kube-api-access-wlbpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:23.199634 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.199607 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-cff4fd7fd-dgj5x" Oct 31 13:02:23.200688 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.200628 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:23.200688 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.200656 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:23.200688 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.200671 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlbpf\" (UniqueName: \"kubernetes.io/projected/4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585-kube-api-access-wlbpf\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:23.200688 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.200687 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:23.218959 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.218937 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-875df4b95-k59g5" Oct 31 13:02:23.272892 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.272830 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:23.497113 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.470742 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/399ac99e-25a1-44d3-b774-04991fbe4f47-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/399ac99e-25a1-44d3-b774-04991fbe4f47-plugins-config-volume\") pod \"399ac99e-25a1-44d3-b774-04991fbe4f47\" (UID: \"399ac99e-25a1-44d3-b774-04991fbe4f47\") " Oct 31 13:02:23.497113 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.470879 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/399ac99e-25a1-44d3-b774-04991fbe4f47-kube-api-access-vl87g\" (UniqueName: \"kubernetes.io/projected/399ac99e-25a1-44d3-b774-04991fbe4f47-kube-api-access-vl87g\") pod \"399ac99e-25a1-44d3-b774-04991fbe4f47\" (UID: \"399ac99e-25a1-44d3-b774-04991fbe4f47\") " Oct 31 13:02:23.497113 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.470957 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9-plugins-config-volume\") pod \"e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9\" (UID: \"e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9\") " Oct 31 13:02:23.497113 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.471048 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9-kube-api-access-w4jbx\" (UniqueName: \"kubernetes.io/projected/e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9-kube-api-access-w4jbx\") pod \"e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9\" (UID: \"e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9\") " Oct 31 13:02:23.497113 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.496309 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9-plugins-config-volume" pod "e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9" (UID: "e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:02:23.497113 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.496745 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/399ac99e-25a1-44d3-b774-04991fbe4f47-plugins-config-volume" pod "399ac99e-25a1-44d3-b774-04991fbe4f47" (UID: "399ac99e-25a1-44d3-b774-04991fbe4f47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:02:23.508201 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.507862 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:23.516765 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.515863 6841 generic.go:356] "Generic (PLEG): container finished" podID="e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9" containerID="75ac6696050ce370e2cc573e1ff405d72e03cd5e5156be3f54f57bca703dbb51" exitCode=0 Oct 31 13:02:23.516765 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.516020 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-cff4fd7fd-dgj5x" event={"ID":"e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9","Type":"ContainerDied","Data":"75ac6696050ce370e2cc573e1ff405d72e03cd5e5156be3f54f57bca703dbb51"} Oct 31 13:02:23.516765 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.516074 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-cff4fd7fd-dgj5x" event={"ID":"e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9","Type":"ContainerDied","Data":"277c7644d48379121f6a97ae942f771e7f37986cc09f89bf1f991419807d0147"} Oct 31 13:02:23.516765 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.516103 6841 scope.go:122] "RemoveContainer" containerID="75ac6696050ce370e2cc573e1ff405d72e03cd5e5156be3f54f57bca703dbb51" Oct 31 13:02:23.516765 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.516254 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf963-cff4fd7fd-dgj5x" Oct 31 13:02:23.525025 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.521379 6841 generic.go:356] "Generic (PLEG): container finished" podID="4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585" containerID="52c869b4451aace43501bd7ead4968df9fcd0f8a179ab092c608bfef344bc546" exitCode=0 Oct 31 13:02:23.525025 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.521558 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z" event={"ID":"4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585","Type":"ContainerDied","Data":"52c869b4451aace43501bd7ead4968df9fcd0f8a179ab092c608bfef344bc546"} Oct 31 13:02:23.525025 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.521607 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z" event={"ID":"4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585","Type":"ContainerDied","Data":"da81ec7edf4862c08370cfb0043d7b781f7b1ef16f652089b78b2651037549b5"} Oct 31 13:02:23.525025 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.521690 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z" Oct 31 13:02:23.530177 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.529266 6841 generic.go:356] "Generic (PLEG): container finished" podID="399ac99e-25a1-44d3-b774-04991fbe4f47" containerID="21a16ed537f35e4190ec4314338b709b228a3c074a92bd8f82e3767ae1c03261" exitCode=0 Oct 31 13:02:23.530177 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.529363 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-875df4b95-k59g5" event={"ID":"399ac99e-25a1-44d3-b774-04991fbe4f47","Type":"ContainerDied","Data":"21a16ed537f35e4190ec4314338b709b228a3c074a92bd8f82e3767ae1c03261"} Oct 31 13:02:23.530177 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.529413 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-875df4b95-k59g5" event={"ID":"399ac99e-25a1-44d3-b774-04991fbe4f47","Type":"ContainerDied","Data":"c9a6b77cb9469d3e827f1ae69f0dd1b7b58922c973b2c6f879ddc8d4f4115e4b"} Oct 31 13:02:23.530177 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.529495 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-875df4b95-k59g5" Oct 31 13:02:23.582269 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.582122 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/399ac99e-25a1-44d3-b774-04991fbe4f47-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:23.582269 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.582185 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:23.645943 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.635782 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/399ac99e-25a1-44d3-b774-04991fbe4f47-kube-api-access-vl87g" pod "399ac99e-25a1-44d3-b774-04991fbe4f47" (UID: "399ac99e-25a1-44d3-b774-04991fbe4f47"). InnerVolumeSpecName "kube-api-access-vl87g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:23.645943 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.644056 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9-kube-api-access-w4jbx" pod "e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9" (UID: "e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9"). InnerVolumeSpecName "kube-api-access-w4jbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:23.648208 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.648161 6841 scope.go:122] "RemoveContainer" containerID="75ac6696050ce370e2cc573e1ff405d72e03cd5e5156be3f54f57bca703dbb51" Oct 31 13:02:23.649388 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:02:23.649365 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75ac6696050ce370e2cc573e1ff405d72e03cd5e5156be3f54f57bca703dbb51\": container with ID starting with 75ac6696050ce370e2cc573e1ff405d72e03cd5e5156be3f54f57bca703dbb51 not found: ID does not exist" containerID="75ac6696050ce370e2cc573e1ff405d72e03cd5e5156be3f54f57bca703dbb51" Oct 31 13:02:23.649582 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.649524 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ac6696050ce370e2cc573e1ff405d72e03cd5e5156be3f54f57bca703dbb51"} err="failed to get container status \"75ac6696050ce370e2cc573e1ff405d72e03cd5e5156be3f54f57bca703dbb51\": rpc error: code = NotFound desc = could not find container \"75ac6696050ce370e2cc573e1ff405d72e03cd5e5156be3f54f57bca703dbb51\": container with ID starting with 75ac6696050ce370e2cc573e1ff405d72e03cd5e5156be3f54f57bca703dbb51 not found: ID does not exist" Oct 31 13:02:23.649582 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.649550 6841 scope.go:122] "RemoveContainer" containerID="52c869b4451aace43501bd7ead4968df9fcd0f8a179ab092c608bfef344bc546" Oct 31 13:02:23.658396 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.658370 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8afce0f-eb7a-45c5-b837-2024fd7c563a" path="/var/lib/kubelet/pods/a8afce0f-eb7a-45c5-b837-2024fd7c563a/volumes" Oct 31 13:02:23.659056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.659030 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3fee6c8-5e84-4011-a626-7a3da7239b06" path="/var/lib/kubelet/pods/e3fee6c8-5e84-4011-a626-7a3da7239b06/volumes" Oct 31 13:02:23.670931 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.670909 6841 scope.go:122] "RemoveContainer" containerID="52c869b4451aace43501bd7ead4968df9fcd0f8a179ab092c608bfef344bc546" Oct 31 13:02:23.672147 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:02:23.672116 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52c869b4451aace43501bd7ead4968df9fcd0f8a179ab092c608bfef344bc546\": container with ID starting with 52c869b4451aace43501bd7ead4968df9fcd0f8a179ab092c608bfef344bc546 not found: ID does not exist" containerID="52c869b4451aace43501bd7ead4968df9fcd0f8a179ab092c608bfef344bc546" Oct 31 13:02:23.672250 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.672167 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52c869b4451aace43501bd7ead4968df9fcd0f8a179ab092c608bfef344bc546"} err="failed to get container status \"52c869b4451aace43501bd7ead4968df9fcd0f8a179ab092c608bfef344bc546\": rpc error: code = NotFound desc = could not find container \"52c869b4451aace43501bd7ead4968df9fcd0f8a179ab092c608bfef344bc546\": container with ID starting with 52c869b4451aace43501bd7ead4968df9fcd0f8a179ab092c608bfef344bc546 not found: ID does not exist" Oct 31 13:02:23.672250 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.672194 6841 scope.go:122] "RemoveContainer" containerID="21a16ed537f35e4190ec4314338b709b228a3c074a92bd8f82e3767ae1c03261" Oct 31 13:02:23.683697 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.683672 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl87g\" (UniqueName: \"kubernetes.io/projected/399ac99e-25a1-44d3-b774-04991fbe4f47-kube-api-access-vl87g\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:23.683697 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.683698 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4jbx\" (UniqueName: \"kubernetes.io/projected/e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9-kube-api-access-w4jbx\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:23.687853 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.687824 6841 scope.go:122] "RemoveContainer" containerID="21a16ed537f35e4190ec4314338b709b228a3c074a92bd8f82e3767ae1c03261" Oct 31 13:02:23.688347 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:02:23.688315 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21a16ed537f35e4190ec4314338b709b228a3c074a92bd8f82e3767ae1c03261\": container with ID starting with 21a16ed537f35e4190ec4314338b709b228a3c074a92bd8f82e3767ae1c03261 not found: ID does not exist" containerID="21a16ed537f35e4190ec4314338b709b228a3c074a92bd8f82e3767ae1c03261" Oct 31 13:02:23.688521 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.688350 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21a16ed537f35e4190ec4314338b709b228a3c074a92bd8f82e3767ae1c03261"} err="failed to get container status \"21a16ed537f35e4190ec4314338b709b228a3c074a92bd8f82e3767ae1c03261\": rpc error: code = NotFound desc = could not find container \"21a16ed537f35e4190ec4314338b709b228a3c074a92bd8f82e3767ae1c03261\": container with ID starting with 21a16ed537f35e4190ec4314338b709b228a3c074a92bd8f82e3767ae1c03261 not found: ID does not exist" Oct 31 13:02:23.695775 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.695750 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z"] Oct 31 13:02:23.701845 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.701810 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf963-7fdd4c65b8-xjg8z"] Oct 31 13:02:23.836184 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.836113 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ccf963-cff4fd7fd-dgj5x"] Oct 31 13:02:23.849959 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.849931 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ccf963-cff4fd7fd-dgj5x"] Oct 31 13:02:23.860604 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.860576 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-875df4b95-k59g5"] Oct 31 13:02:23.869008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.868965 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf963-875df4b95-k59g5"] Oct 31 13:02:23.958779 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.958742 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr"] Oct 31 13:02:23.960498 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:23.959008 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr" podUID="6eadb1e5-3fb0-49a8-964b-06d862381d04" containerName="maas-api" containerID="cri-o://1fe0703aa482df70feee2facb1721e8ebfcc0047dc8cfe22b19a27f0e8ca2c2c" gracePeriod=30 Oct 31 13:02:24.107194 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:24.107111 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:24.467772 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:24.467739 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr" Oct 31 13:02:24.536320 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:24.536291 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6eadb1e5-3fb0-49a8-964b-06d862381d04-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eadb1e5-3fb0-49a8-964b-06d862381d04-metrics-certs\") pod \"6eadb1e5-3fb0-49a8-964b-06d862381d04\" (UID: \"6eadb1e5-3fb0-49a8-964b-06d862381d04\") " Oct 31 13:02:24.536516 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:24.536366 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eadb1e5-3fb0-49a8-964b-06d862381d04-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eadb1e5-3fb0-49a8-964b-06d862381d04-openshift-service-ca\") pod \"6eadb1e5-3fb0-49a8-964b-06d862381d04\" (UID: \"6eadb1e5-3fb0-49a8-964b-06d862381d04\") " Oct 31 13:02:24.536516 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:24.536455 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eadb1e5-3fb0-49a8-964b-06d862381d04-kube-api-access-lfrnv\" (UniqueName: \"kubernetes.io/projected/6eadb1e5-3fb0-49a8-964b-06d862381d04-kube-api-access-lfrnv\") pod \"6eadb1e5-3fb0-49a8-964b-06d862381d04\" (UID: \"6eadb1e5-3fb0-49a8-964b-06d862381d04\") " Oct 31 13:02:24.536516 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:24.536508 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6eadb1e5-3fb0-49a8-964b-06d862381d04-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eadb1e5-3fb0-49a8-964b-06d862381d04-maas-api-tls\") pod \"6eadb1e5-3fb0-49a8-964b-06d862381d04\" (UID: \"6eadb1e5-3fb0-49a8-964b-06d862381d04\") " Oct 31 13:02:24.537006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:24.536966 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eadb1e5-3fb0-49a8-964b-06d862381d04-openshift-service-ca" pod "6eadb1e5-3fb0-49a8-964b-06d862381d04" (UID: "6eadb1e5-3fb0-49a8-964b-06d862381d04"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:02:24.537240 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:24.537206 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eadb1e5-3fb0-49a8-964b-06d862381d04-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:24.539347 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:24.539322 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6eadb1e5-3fb0-49a8-964b-06d862381d04-maas-api-tls" pod "6eadb1e5-3fb0-49a8-964b-06d862381d04" (UID: "6eadb1e5-3fb0-49a8-964b-06d862381d04"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:02:24.539595 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:24.539564 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eadb1e5-3fb0-49a8-964b-06d862381d04-kube-api-access-lfrnv" pod "6eadb1e5-3fb0-49a8-964b-06d862381d04" (UID: "6eadb1e5-3fb0-49a8-964b-06d862381d04"). InnerVolumeSpecName "kube-api-access-lfrnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:24.545709 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:24.545681 6841 generic.go:356] "Generic (PLEG): container finished" podID="6eadb1e5-3fb0-49a8-964b-06d862381d04" containerID="1fe0703aa482df70feee2facb1721e8ebfcc0047dc8cfe22b19a27f0e8ca2c2c" exitCode=0 Oct 31 13:02:24.545799 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:24.545781 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr" event={"ID":"6eadb1e5-3fb0-49a8-964b-06d862381d04","Type":"ContainerDied","Data":"1fe0703aa482df70feee2facb1721e8ebfcc0047dc8cfe22b19a27f0e8ca2c2c"} Oct 31 13:02:24.545854 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:24.545840 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr" event={"ID":"6eadb1e5-3fb0-49a8-964b-06d862381d04","Type":"ContainerDied","Data":"5699c2339cadb6e47f074c7c14de753cd5f3b6db1f01f87187cef2e61f27eec2"} Oct 31 13:02:24.545899 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:24.545874 6841 scope.go:122] "RemoveContainer" containerID="1fe0703aa482df70feee2facb1721e8ebfcc0047dc8cfe22b19a27f0e8ca2c2c" Oct 31 13:02:24.546148 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:24.546129 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr" Oct 31 13:02:24.556134 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:24.556105 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6eadb1e5-3fb0-49a8-964b-06d862381d04-metrics-certs" pod "6eadb1e5-3fb0-49a8-964b-06d862381d04" (UID: "6eadb1e5-3fb0-49a8-964b-06d862381d04"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:02:24.570589 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:24.570563 6841 scope.go:122] "RemoveContainer" containerID="1fe0703aa482df70feee2facb1721e8ebfcc0047dc8cfe22b19a27f0e8ca2c2c" Oct 31 13:02:24.570910 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:02:24.570889 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fe0703aa482df70feee2facb1721e8ebfcc0047dc8cfe22b19a27f0e8ca2c2c\": container with ID starting with 1fe0703aa482df70feee2facb1721e8ebfcc0047dc8cfe22b19a27f0e8ca2c2c not found: ID does not exist" containerID="1fe0703aa482df70feee2facb1721e8ebfcc0047dc8cfe22b19a27f0e8ca2c2c" Oct 31 13:02:24.570975 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:24.570922 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fe0703aa482df70feee2facb1721e8ebfcc0047dc8cfe22b19a27f0e8ca2c2c"} err="failed to get container status \"1fe0703aa482df70feee2facb1721e8ebfcc0047dc8cfe22b19a27f0e8ca2c2c\": rpc error: code = NotFound desc = could not find container \"1fe0703aa482df70feee2facb1721e8ebfcc0047dc8cfe22b19a27f0e8ca2c2c\": container with ID starting with 1fe0703aa482df70feee2facb1721e8ebfcc0047dc8cfe22b19a27f0e8ca2c2c not found: ID does not exist" Oct 31 13:02:24.635371 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:24.635229 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:24.657012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:24.645012 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eadb1e5-3fb0-49a8-964b-06d862381d04-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:24.657356 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:24.657327 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eadb1e5-3fb0-49a8-964b-06d862381d04-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:24.657486 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:24.657472 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfrnv\" (UniqueName: \"kubernetes.io/projected/6eadb1e5-3fb0-49a8-964b-06d862381d04-kube-api-access-lfrnv\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:24.878037 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:24.878004 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr"] Oct 31 13:02:24.887225 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:24.887197 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-01e8d2-7df67cb958-l5jvr"] Oct 31 13:02:24.981639 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:24.981605 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:25.190314 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:25.190237 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:25.478936 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:25.478596 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:25.694425 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:25.694386 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="399ac99e-25a1-44d3-b774-04991fbe4f47" path="/var/lib/kubelet/pods/399ac99e-25a1-44d3-b774-04991fbe4f47/volumes" Oct 31 13:02:25.694962 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:25.694938 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585" path="/var/lib/kubelet/pods/4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585/volumes" Oct 31 13:02:25.695517 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:25.695495 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eadb1e5-3fb0-49a8-964b-06d862381d04" path="/var/lib/kubelet/pods/6eadb1e5-3fb0-49a8-964b-06d862381d04/volumes" Oct 31 13:02:25.696091 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:25.696070 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9" path="/var/lib/kubelet/pods/e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9/volumes" Oct 31 13:02:25.714619 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:25.714592 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:25.800615 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:25.800522 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8"] Oct 31 13:02:25.800801 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:25.800772 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" podUID="40d2bf70-21e4-498f-ac76-23246675e3bf" containerName="istio-proxy" containerID="cri-o://e353905af1081da1a8522088240c7514e5ae6e2633109e62d7dbad60a8ccb3c5" gracePeriod=30 Oct 31 13:02:25.866440 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:25.866408 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:02:25.866593 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:25.866466 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" podUID="40d2bf70-21e4-498f-ac76-23246675e3bf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:02:25.958035 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:25.953271 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:26.616155 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:26.615166 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:26.833601 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:26.833570 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:27.080008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:27.076345 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:27.397508 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:27.397330 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:27.436228 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:27.436194 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2"] Oct 31 13:02:27.436418 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:27.436398 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" podUID="ebcb215e-7f84-4653-96e6-3a13c449be73" containerName="istio-proxy" containerID="cri-o://54c965bd8bde25b4e49d2af75f2e8bfdd691ab7c0dbf680572bf5e32ca966e7e" gracePeriod=30 Oct 31 13:02:27.675698 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:27.675554 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:28.116928 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.116536 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:28.381277 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.381185 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:28.655655 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.655571 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:28.743809 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.743778 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-5d98778b55-snvlr"] Oct 31 13:02:28.744479 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.744461 6841 cpu_manager.go:414] "containerMap: removing container" podUID="e3fee6c8-5e84-4011-a626-7a3da7239b06" containerName="payload-processing" Oct 31 13:02:28.744479 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.744479 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3fee6c8-5e84-4011-a626-7a3da7239b06" containerName="payload-processing" Oct 31 13:02:28.744595 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.744506 6841 cpu_manager.go:414] "containerMap: removing container" podUID="e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9" containerName="payload-processing" Oct 31 13:02:28.744595 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.744511 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9" containerName="payload-processing" Oct 31 13:02:28.744595 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.744523 6841 cpu_manager.go:414] "containerMap: removing container" podUID="a8afce0f-eb7a-45c5-b837-2024fd7c563a" containerName="maas-api" Oct 31 13:02:28.744595 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.744529 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8afce0f-eb7a-45c5-b837-2024fd7c563a" containerName="maas-api" Oct 31 13:02:28.744595 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.744539 6841 cpu_manager.go:414] "containerMap: removing container" podUID="1c502702-da0b-45d4-ba77-5ae972c1bd16" containerName="istio-proxy" Oct 31 13:02:28.744595 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.744544 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c502702-da0b-45d4-ba77-5ae972c1bd16" containerName="istio-proxy" Oct 31 13:02:28.744595 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.744553 6841 cpu_manager.go:414] "containerMap: removing container" podUID="4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585" containerName="maas-api" Oct 31 13:02:28.744595 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.744557 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585" containerName="maas-api" Oct 31 13:02:28.744912 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.744601 6841 cpu_manager.go:414] "containerMap: removing container" podUID="3e30e333-c7fd-4aaa-9541-3a1ef29189a4" containerName="istio-proxy" Oct 31 13:02:28.744912 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.744606 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e30e333-c7fd-4aaa-9541-3a1ef29189a4" containerName="istio-proxy" Oct 31 13:02:28.744912 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.744619 6841 cpu_manager.go:414] "containerMap: removing container" podUID="3897f656-b3a9-4d66-b894-3663c88c253d" containerName="istio-proxy" Oct 31 13:02:28.744912 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.744624 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3897f656-b3a9-4d66-b894-3663c88c253d" containerName="istio-proxy" Oct 31 13:02:28.744912 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.744641 6841 cpu_manager.go:414] "containerMap: removing container" podUID="6eadb1e5-3fb0-49a8-964b-06d862381d04" containerName="maas-api" Oct 31 13:02:28.744912 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.744647 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eadb1e5-3fb0-49a8-964b-06d862381d04" containerName="maas-api" Oct 31 13:02:28.744912 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.744654 6841 cpu_manager.go:414] "containerMap: removing container" podUID="1c502702-da0b-45d4-ba77-5ae972c1bd16" containerName="istio-proxy" Oct 31 13:02:28.744912 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.744659 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c502702-da0b-45d4-ba77-5ae972c1bd16" containerName="istio-proxy" Oct 31 13:02:28.744912 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.744665 6841 cpu_manager.go:414] "containerMap: removing container" podUID="399ac99e-25a1-44d3-b774-04991fbe4f47" containerName="payload-pre-processing" Oct 31 13:02:28.744912 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.744670 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="399ac99e-25a1-44d3-b774-04991fbe4f47" containerName="payload-pre-processing" Oct 31 13:02:28.744912 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.744794 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3fee6c8-5e84-4011-a626-7a3da7239b06" containerName="payload-processing" Oct 31 13:02:28.744912 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.744815 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9c41d9c-82e3-440b-82fd-dbe2e5ab8bd9" containerName="payload-processing" Oct 31 13:02:28.744912 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.744825 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8afce0f-eb7a-45c5-b837-2024fd7c563a" containerName="maas-api" Oct 31 13:02:28.744912 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.744834 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c502702-da0b-45d4-ba77-5ae972c1bd16" containerName="istio-proxy" Oct 31 13:02:28.744912 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.744844 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d3e1a2f-5dfd-4a0a-bf35-4aeaeeeac585" containerName="maas-api" Oct 31 13:02:28.744912 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.744855 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e30e333-c7fd-4aaa-9541-3a1ef29189a4" containerName="istio-proxy" Oct 31 13:02:28.744912 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.744863 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="3897f656-b3a9-4d66-b894-3663c88c253d" containerName="istio-proxy" Oct 31 13:02:28.744912 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.744870 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c502702-da0b-45d4-ba77-5ae972c1bd16" containerName="istio-proxy" Oct 31 13:02:28.744912 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.744876 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eadb1e5-3fb0-49a8-964b-06d862381d04" containerName="maas-api" Oct 31 13:02:28.744912 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.744884 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="399ac99e-25a1-44d3-b774-04991fbe4f47" containerName="payload-pre-processing" Oct 31 13:02:28.748495 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.748471 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-5d98778b55-snvlr" Oct 31 13:02:28.760445 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.760136 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-0db792-dockercfg-z927j\"" Oct 31 13:02:28.761611 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.761589 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-0db792\"" Oct 31 13:02:28.766802 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.766782 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-5d98778b55-snvlr"] Oct 31 13:02:28.780761 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.780731 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtp2s\" (UniqueName: \"kubernetes.io/projected/93391f16-8d6c-45a1-9bb6-838d6b65289d-kube-api-access-gtp2s\") pod \"payload-pre-processing-e2e-ipp-b-0db792-5d98778b55-snvlr\" (UID: \"93391f16-8d6c-45a1-9bb6-838d6b65289d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-5d98778b55-snvlr" Oct 31 13:02:28.780963 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.780934 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93391f16-8d6c-45a1-9bb6-838d6b65289d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0db792-5d98778b55-snvlr\" (UID: \"93391f16-8d6c-45a1-9bb6-838d6b65289d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-5d98778b55-snvlr" Oct 31 13:02:28.909066 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.908487 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtp2s\" (UniqueName: \"kubernetes.io/projected/93391f16-8d6c-45a1-9bb6-838d6b65289d-kube-api-access-gtp2s\") pod \"payload-pre-processing-e2e-ipp-b-0db792-5d98778b55-snvlr\" (UID: \"93391f16-8d6c-45a1-9bb6-838d6b65289d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-5d98778b55-snvlr" Oct 31 13:02:28.909066 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.908678 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93391f16-8d6c-45a1-9bb6-838d6b65289d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0db792-5d98778b55-snvlr\" (UID: \"93391f16-8d6c-45a1-9bb6-838d6b65289d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-5d98778b55-snvlr" Oct 31 13:02:28.916005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.909884 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93391f16-8d6c-45a1-9bb6-838d6b65289d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0db792-5d98778b55-snvlr\" (UID: \"93391f16-8d6c-45a1-9bb6-838d6b65289d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-5d98778b55-snvlr" Oct 31 13:02:28.936006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:28.925904 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0db792-54bf4947dd-q7k6g"] Oct 31 13:02:29.033005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:29.032734 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:29.064006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:29.054188 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0db792-54bf4947dd-q7k6g"] Oct 31 13:02:29.064006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:29.054310 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db792-54bf4947dd-q7k6g" Oct 31 13:02:29.064006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:29.063582 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtp2s\" (UniqueName: \"kubernetes.io/projected/93391f16-8d6c-45a1-9bb6-838d6b65289d-kube-api-access-gtp2s\") pod \"payload-pre-processing-e2e-ipp-b-0db792-5d98778b55-snvlr\" (UID: \"93391f16-8d6c-45a1-9bb6-838d6b65289d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-5d98778b55-snvlr" Oct 31 13:02:29.064006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:29.063863 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-5d98778b55-snvlr" Oct 31 13:02:29.146065 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:29.146024 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e238ba43-f409-4888-aff1-b47b000c0211-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0db792-54bf4947dd-q7k6g\" (UID: \"e238ba43-f409-4888-aff1-b47b000c0211\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0db792-54bf4947dd-q7k6g" Oct 31 13:02:29.146221 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:29.146199 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjcfd\" (UniqueName: \"kubernetes.io/projected/e238ba43-f409-4888-aff1-b47b000c0211-kube-api-access-sjcfd\") pod \"payload-processing-e2e-ipp-b-0db792-54bf4947dd-q7k6g\" (UID: \"e238ba43-f409-4888-aff1-b47b000c0211\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0db792-54bf4947dd-q7k6g" Oct 31 13:02:29.247719 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:29.247672 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e238ba43-f409-4888-aff1-b47b000c0211-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0db792-54bf4947dd-q7k6g\" (UID: \"e238ba43-f409-4888-aff1-b47b000c0211\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0db792-54bf4947dd-q7k6g" Oct 31 13:02:29.247906 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:29.247763 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjcfd\" (UniqueName: \"kubernetes.io/projected/e238ba43-f409-4888-aff1-b47b000c0211-kube-api-access-sjcfd\") pod \"payload-processing-e2e-ipp-b-0db792-54bf4947dd-q7k6g\" (UID: \"e238ba43-f409-4888-aff1-b47b000c0211\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0db792-54bf4947dd-q7k6g" Oct 31 13:02:29.248635 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:29.248605 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e238ba43-f409-4888-aff1-b47b000c0211-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0db792-54bf4947dd-q7k6g\" (UID: \"e238ba43-f409-4888-aff1-b47b000c0211\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0db792-54bf4947dd-q7k6g" Oct 31 13:02:29.262101 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:29.262076 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjcfd\" (UniqueName: \"kubernetes.io/projected/e238ba43-f409-4888-aff1-b47b000c0211-kube-api-access-sjcfd\") pod \"payload-processing-e2e-ipp-b-0db792-54bf4947dd-q7k6g\" (UID: \"e238ba43-f409-4888-aff1-b47b000c0211\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0db792-54bf4947dd-q7k6g" Oct 31 13:02:29.422120 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:29.421464 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db792-54bf4947dd-q7k6g" Oct 31 13:02:29.446660 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:29.446379 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-5d98778b55-snvlr"] Oct 31 13:02:29.448953 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:29.448749 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:29.448953 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:02:29.448915 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93391f16_8d6c_45a1_9bb6_838d6b65289d.slice/crio-637b0fd96472785eefda8d2e32af1c1924dc302b2030515cbe3c69a8b2e8cce3 WatchSource:0}: Error finding container 637b0fd96472785eefda8d2e32af1c1924dc302b2030515cbe3c69a8b2e8cce3: Status 404 returned error can't find the container with id 637b0fd96472785eefda8d2e32af1c1924dc302b2030515cbe3c69a8b2e8cce3 Oct 31 13:02:29.701465 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:29.701426 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-5d98778b55-snvlr" event={"ID":"93391f16-8d6c-45a1-9bb6-838d6b65289d","Type":"ContainerStarted","Data":"c9c2e355f6a1c2236bc2d61462ca6f60915c524e8fd960f95c7f4ed999440064"} Oct 31 13:02:29.701639 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:29.701500 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-5d98778b55-snvlr" Oct 31 13:02:29.701639 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:29.701526 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-5d98778b55-snvlr" event={"ID":"93391f16-8d6c-45a1-9bb6-838d6b65289d","Type":"ContainerStarted","Data":"637b0fd96472785eefda8d2e32af1c1924dc302b2030515cbe3c69a8b2e8cce3"} Oct 31 13:02:29.723935 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:29.723903 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0db792-54bf4947dd-q7k6g"] Oct 31 13:02:29.724877 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:02:29.724846 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode238ba43_f409_4888_aff1_b47b000c0211.slice/crio-ec6c9180125ec865b7bbdd5b7ddf513b2318f3f69a28fe3020db4c77dc4cbfc2 WatchSource:0}: Error finding container ec6c9180125ec865b7bbdd5b7ddf513b2318f3f69a28fe3020db4c77dc4cbfc2: Status 404 returned error can't find the container with id ec6c9180125ec865b7bbdd5b7ddf513b2318f3f69a28fe3020db4c77dc4cbfc2 Oct 31 13:02:29.726541 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:29.726496 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-5d98778b55-snvlr" podStartSLOduration=1.726485132 podStartE2EDuration="1.726485132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:02:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:29.722771456 +0000 UTC m=+1796.205894362" watchObservedRunningTime="2026-10-31 13:02:29.726485132 +0000 UTC m=+1796.209608033" Oct 31 13:02:30.395006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:30.392121 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:30.712302 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:30.712262 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db792-54bf4947dd-q7k6g" event={"ID":"e238ba43-f409-4888-aff1-b47b000c0211","Type":"ContainerStarted","Data":"817e7c84c84bab3d00079c648ce3648d8785be2f26f7a232f54ec20288cc1f71"} Oct 31 13:02:30.712491 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:30.712329 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db792-54bf4947dd-q7k6g" event={"ID":"e238ba43-f409-4888-aff1-b47b000c0211","Type":"ContainerStarted","Data":"ec6c9180125ec865b7bbdd5b7ddf513b2318f3f69a28fe3020db4c77dc4cbfc2"} Oct 31 13:02:30.712491 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:30.712360 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db792-54bf4947dd-q7k6g" Oct 31 13:02:30.769164 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:30.769113 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db792-54bf4947dd-q7k6g" podStartSLOduration=2.769100393 podStartE2EDuration="2.769100393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:02:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:30.765643449 +0000 UTC m=+1797.248766350" watchObservedRunningTime="2026-10-31 13:02:30.769100393 +0000 UTC m=+1797.252223294" Oct 31 13:02:30.782124 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:30.782075 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:31.180727 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.180696 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:02:31.245689 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.245411 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:31.316587 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.316539 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40d2bf70-21e4-498f-ac76-23246675e3bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d2bf70-21e4-498f-ac76-23246675e3bf-credential-socket\") pod \"40d2bf70-21e4-498f-ac76-23246675e3bf\" (UID: \"40d2bf70-21e4-498f-ac76-23246675e3bf\") " Oct 31 13:02:31.316790 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.316677 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40d2bf70-21e4-498f-ac76-23246675e3bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40d2bf70-21e4-498f-ac76-23246675e3bf-workload-certs\") pod \"40d2bf70-21e4-498f-ac76-23246675e3bf\" (UID: \"40d2bf70-21e4-498f-ac76-23246675e3bf\") " Oct 31 13:02:31.316790 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.316751 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40d2bf70-21e4-498f-ac76-23246675e3bf-istio-token\" (UniqueName: \"kubernetes.io/projected/40d2bf70-21e4-498f-ac76-23246675e3bf-istio-token\") pod \"40d2bf70-21e4-498f-ac76-23246675e3bf\" (UID: \"40d2bf70-21e4-498f-ac76-23246675e3bf\") " Oct 31 13:02:31.317299 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.316952 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40d2bf70-21e4-498f-ac76-23246675e3bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d2bf70-21e4-498f-ac76-23246675e3bf-workload-socket\") pod \"40d2bf70-21e4-498f-ac76-23246675e3bf\" (UID: \"40d2bf70-21e4-498f-ac76-23246675e3bf\") " Oct 31 13:02:31.317299 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.317019 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40d2bf70-21e4-498f-ac76-23246675e3bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40d2bf70-21e4-498f-ac76-23246675e3bf-istio-data\") pod \"40d2bf70-21e4-498f-ac76-23246675e3bf\" (UID: \"40d2bf70-21e4-498f-ac76-23246675e3bf\") " Oct 31 13:02:31.317299 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.317063 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40d2bf70-21e4-498f-ac76-23246675e3bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40d2bf70-21e4-498f-ac76-23246675e3bf-istio-podinfo\") pod \"40d2bf70-21e4-498f-ac76-23246675e3bf\" (UID: \"40d2bf70-21e4-498f-ac76-23246675e3bf\") " Oct 31 13:02:31.317299 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.317161 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40d2bf70-21e4-498f-ac76-23246675e3bf-kube-api-access-5j97b\" (UniqueName: \"kubernetes.io/projected/40d2bf70-21e4-498f-ac76-23246675e3bf-kube-api-access-5j97b\") pod \"40d2bf70-21e4-498f-ac76-23246675e3bf\" (UID: \"40d2bf70-21e4-498f-ac76-23246675e3bf\") " Oct 31 13:02:31.317299 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.317202 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40d2bf70-21e4-498f-ac76-23246675e3bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40d2bf70-21e4-498f-ac76-23246675e3bf-istio-envoy\") pod \"40d2bf70-21e4-498f-ac76-23246675e3bf\" (UID: \"40d2bf70-21e4-498f-ac76-23246675e3bf\") " Oct 31 13:02:31.317637 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.317308 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40d2bf70-21e4-498f-ac76-23246675e3bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40d2bf70-21e4-498f-ac76-23246675e3bf-istiod-ca-cert\") pod \"40d2bf70-21e4-498f-ac76-23246675e3bf\" (UID: \"40d2bf70-21e4-498f-ac76-23246675e3bf\") " Oct 31 13:02:31.343494 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.341489 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d2bf70-21e4-498f-ac76-23246675e3bf-workload-socket" pod "40d2bf70-21e4-498f-ac76-23246675e3bf" (UID: "40d2bf70-21e4-498f-ac76-23246675e3bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:31.343494 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.341773 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d2bf70-21e4-498f-ac76-23246675e3bf-istio-data" pod "40d2bf70-21e4-498f-ac76-23246675e3bf" (UID: "40d2bf70-21e4-498f-ac76-23246675e3bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:31.343494 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.341846 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40d2bf70-21e4-498f-ac76-23246675e3bf-istio-podinfo" pod "40d2bf70-21e4-498f-ac76-23246675e3bf" (UID: "40d2bf70-21e4-498f-ac76-23246675e3bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:02:31.343494 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.342926 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40d2bf70-21e4-498f-ac76-23246675e3bf-istiod-ca-cert" pod "40d2bf70-21e4-498f-ac76-23246675e3bf" (UID: "40d2bf70-21e4-498f-ac76-23246675e3bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:02:31.343494 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.343008 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d2bf70-21e4-498f-ac76-23246675e3bf-workload-certs" pod "40d2bf70-21e4-498f-ac76-23246675e3bf" (UID: "40d2bf70-21e4-498f-ac76-23246675e3bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:31.344524 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.344490 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d2bf70-21e4-498f-ac76-23246675e3bf-kube-api-access-5j97b" pod "40d2bf70-21e4-498f-ac76-23246675e3bf" (UID: "40d2bf70-21e4-498f-ac76-23246675e3bf"). InnerVolumeSpecName "kube-api-access-5j97b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:31.344657 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.344630 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d2bf70-21e4-498f-ac76-23246675e3bf-credential-socket" pod "40d2bf70-21e4-498f-ac76-23246675e3bf" (UID: "40d2bf70-21e4-498f-ac76-23246675e3bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:31.350245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.346907 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d2bf70-21e4-498f-ac76-23246675e3bf-istio-envoy" pod "40d2bf70-21e4-498f-ac76-23246675e3bf" (UID: "40d2bf70-21e4-498f-ac76-23246675e3bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:31.360024 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.357347 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d2bf70-21e4-498f-ac76-23246675e3bf-istio-token" pod "40d2bf70-21e4-498f-ac76-23246675e3bf" (UID: "40d2bf70-21e4-498f-ac76-23246675e3bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:31.422289 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.422251 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40d2bf70-21e4-498f-ac76-23246675e3bf-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:31.422289 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.422289 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j97b\" (UniqueName: \"kubernetes.io/projected/40d2bf70-21e4-498f-ac76-23246675e3bf-kube-api-access-5j97b\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:31.422703 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.422304 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40d2bf70-21e4-498f-ac76-23246675e3bf-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:31.422703 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.422318 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40d2bf70-21e4-498f-ac76-23246675e3bf-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:31.422703 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.422330 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d2bf70-21e4-498f-ac76-23246675e3bf-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:31.422703 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.422343 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40d2bf70-21e4-498f-ac76-23246675e3bf-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:31.422703 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.422353 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40d2bf70-21e4-498f-ac76-23246675e3bf-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:31.422703 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.422361 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d2bf70-21e4-498f-ac76-23246675e3bf-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:31.422703 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.422370 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40d2bf70-21e4-498f-ac76-23246675e3bf-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:31.537765 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.537657 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt"] Oct 31 13:02:31.538034 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.538002 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" podUID="cd421e14-02ab-4072-918a-e3c6077263c8" containerName="istio-proxy" containerID="cri-o://44c37590ceca231b5b5ba8c3e595cee0f52dee2b714a0671f8cad56d504b48dd" gracePeriod=30 Oct 31 13:02:31.790052 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.777674 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:31.900194 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.898839 6841 generic.go:356] "Generic (PLEG): container finished" podID="40d2bf70-21e4-498f-ac76-23246675e3bf" containerID="e353905af1081da1a8522088240c7514e5ae6e2633109e62d7dbad60a8ccb3c5" exitCode=0 Oct 31 13:02:31.900194 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.899892 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" Oct 31 13:02:31.900447 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.900261 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" event={"ID":"40d2bf70-21e4-498f-ac76-23246675e3bf","Type":"ContainerDied","Data":"e353905af1081da1a8522088240c7514e5ae6e2633109e62d7dbad60a8ccb3c5"} Oct 31 13:02:31.900447 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.900318 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8" event={"ID":"40d2bf70-21e4-498f-ac76-23246675e3bf","Type":"ContainerDied","Data":"f4261dbb1e9b169705265ccb74ee4d61f9e677a6a5d438a39941e6dd93882484"} Oct 31 13:02:31.900447 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.900349 6841 scope.go:122] "RemoveContainer" containerID="e353905af1081da1a8522088240c7514e5ae6e2633109e62d7dbad60a8ccb3c5" Oct 31 13:02:31.921214 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:02:31.921176 6841 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d2bf70_21e4_498f_ac76_23246675e3bf.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d2bf70_21e4_498f_ac76_23246675e3bf.slice/crio-f4261dbb1e9b169705265ccb74ee4d61f9e677a6a5d438a39941e6dd93882484\": RecentStats: unable to find data in memory cache]" Oct 31 13:02:31.921429 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:02:31.921211 6841 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d2bf70_21e4_498f_ac76_23246675e3bf.slice/crio-f4261dbb1e9b169705265ccb74ee4d61f9e677a6a5d438a39941e6dd93882484\": RecentStats: unable to find data in memory cache]" Oct 31 13:02:31.936673 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.936649 6841 scope.go:122] "RemoveContainer" containerID="1a09d688c41bd1d2cbce3a2c6f137d38aa0ff6b46ca662008a8325740ec7dacc" Oct 31 13:02:31.942575 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.942535 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8"] Oct 31 13:02:31.948563 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.948532 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-bc9ae8-openshift-default-978b96fdc-g89g8"] Oct 31 13:02:31.951570 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.951550 6841 scope.go:122] "RemoveContainer" containerID="e353905af1081da1a8522088240c7514e5ae6e2633109e62d7dbad60a8ccb3c5" Oct 31 13:02:31.951861 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:02:31.951839 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e353905af1081da1a8522088240c7514e5ae6e2633109e62d7dbad60a8ccb3c5\": container with ID starting with e353905af1081da1a8522088240c7514e5ae6e2633109e62d7dbad60a8ccb3c5 not found: ID does not exist" containerID="e353905af1081da1a8522088240c7514e5ae6e2633109e62d7dbad60a8ccb3c5" Oct 31 13:02:31.951905 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.951869 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e353905af1081da1a8522088240c7514e5ae6e2633109e62d7dbad60a8ccb3c5"} err="failed to get container status \"e353905af1081da1a8522088240c7514e5ae6e2633109e62d7dbad60a8ccb3c5\": rpc error: code = NotFound desc = could not find container \"e353905af1081da1a8522088240c7514e5ae6e2633109e62d7dbad60a8ccb3c5\": container with ID starting with e353905af1081da1a8522088240c7514e5ae6e2633109e62d7dbad60a8ccb3c5 not found: ID does not exist" Oct 31 13:02:31.951905 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.951888 6841 scope.go:122] "RemoveContainer" containerID="1a09d688c41bd1d2cbce3a2c6f137d38aa0ff6b46ca662008a8325740ec7dacc" Oct 31 13:02:31.952184 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:02:31.952146 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a09d688c41bd1d2cbce3a2c6f137d38aa0ff6b46ca662008a8325740ec7dacc\": container with ID starting with 1a09d688c41bd1d2cbce3a2c6f137d38aa0ff6b46ca662008a8325740ec7dacc not found: ID does not exist" containerID="1a09d688c41bd1d2cbce3a2c6f137d38aa0ff6b46ca662008a8325740ec7dacc" Oct 31 13:02:31.952184 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:31.952171 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a09d688c41bd1d2cbce3a2c6f137d38aa0ff6b46ca662008a8325740ec7dacc"} err="failed to get container status \"1a09d688c41bd1d2cbce3a2c6f137d38aa0ff6b46ca662008a8325740ec7dacc\": rpc error: code = NotFound desc = could not find container \"1a09d688c41bd1d2cbce3a2c6f137d38aa0ff6b46ca662008a8325740ec7dacc\": container with ID starting with 1a09d688c41bd1d2cbce3a2c6f137d38aa0ff6b46ca662008a8325740ec7dacc not found: ID does not exist" Oct 31 13:02:32.241936 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.241900 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:32.588083 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.588042 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:32.730681 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.730656 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:02:32.806039 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.805941 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebcb215e-7f84-4653-96e6-3a13c449be73-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebcb215e-7f84-4653-96e6-3a13c449be73-istio-envoy\") pod \"ebcb215e-7f84-4653-96e6-3a13c449be73\" (UID: \"ebcb215e-7f84-4653-96e6-3a13c449be73\") " Oct 31 13:02:32.806191 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.806111 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebcb215e-7f84-4653-96e6-3a13c449be73-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebcb215e-7f84-4653-96e6-3a13c449be73-istiod-ca-cert\") pod \"ebcb215e-7f84-4653-96e6-3a13c449be73\" (UID: \"ebcb215e-7f84-4653-96e6-3a13c449be73\") " Oct 31 13:02:32.806235 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.806190 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebcb215e-7f84-4653-96e6-3a13c449be73-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebcb215e-7f84-4653-96e6-3a13c449be73-istio-data\") pod \"ebcb215e-7f84-4653-96e6-3a13c449be73\" (UID: \"ebcb215e-7f84-4653-96e6-3a13c449be73\") " Oct 31 13:02:32.806268 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.806245 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebcb215e-7f84-4653-96e6-3a13c449be73-istio-token\" (UniqueName: \"kubernetes.io/projected/ebcb215e-7f84-4653-96e6-3a13c449be73-istio-token\") pod \"ebcb215e-7f84-4653-96e6-3a13c449be73\" (UID: \"ebcb215e-7f84-4653-96e6-3a13c449be73\") " Oct 31 13:02:32.806305 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.806281 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebcb215e-7f84-4653-96e6-3a13c449be73-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebcb215e-7f84-4653-96e6-3a13c449be73-credential-socket\") pod \"ebcb215e-7f84-4653-96e6-3a13c449be73\" (UID: \"ebcb215e-7f84-4653-96e6-3a13c449be73\") " Oct 31 13:02:32.806337 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.806313 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebcb215e-7f84-4653-96e6-3a13c449be73-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebcb215e-7f84-4653-96e6-3a13c449be73-workload-socket\") pod \"ebcb215e-7f84-4653-96e6-3a13c449be73\" (UID: \"ebcb215e-7f84-4653-96e6-3a13c449be73\") " Oct 31 13:02:32.806367 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.806339 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebcb215e-7f84-4653-96e6-3a13c449be73-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebcb215e-7f84-4653-96e6-3a13c449be73-workload-certs\") pod \"ebcb215e-7f84-4653-96e6-3a13c449be73\" (UID: \"ebcb215e-7f84-4653-96e6-3a13c449be73\") " Oct 31 13:02:32.806434 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.806417 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebcb215e-7f84-4653-96e6-3a13c449be73-kube-api-access-pkh5r\" (UniqueName: \"kubernetes.io/projected/ebcb215e-7f84-4653-96e6-3a13c449be73-kube-api-access-pkh5r\") pod \"ebcb215e-7f84-4653-96e6-3a13c449be73\" (UID: \"ebcb215e-7f84-4653-96e6-3a13c449be73\") " Oct 31 13:02:32.806490 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.806476 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ebcb215e-7f84-4653-96e6-3a13c449be73-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebcb215e-7f84-4653-96e6-3a13c449be73-istio-podinfo\") pod \"ebcb215e-7f84-4653-96e6-3a13c449be73\" (UID: \"ebcb215e-7f84-4653-96e6-3a13c449be73\") " Oct 31 13:02:32.807888 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.807856 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebcb215e-7f84-4653-96e6-3a13c449be73-istiod-ca-cert" pod "ebcb215e-7f84-4653-96e6-3a13c449be73" (UID: "ebcb215e-7f84-4653-96e6-3a13c449be73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:02:32.808080 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.808055 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebcb215e-7f84-4653-96e6-3a13c449be73-credential-socket" pod "ebcb215e-7f84-4653-96e6-3a13c449be73" (UID: "ebcb215e-7f84-4653-96e6-3a13c449be73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:32.808265 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.808244 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebcb215e-7f84-4653-96e6-3a13c449be73-workload-socket" pod "ebcb215e-7f84-4653-96e6-3a13c449be73" (UID: "ebcb215e-7f84-4653-96e6-3a13c449be73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:32.808446 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.808385 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebcb215e-7f84-4653-96e6-3a13c449be73-istio-data" pod "ebcb215e-7f84-4653-96e6-3a13c449be73" (UID: "ebcb215e-7f84-4653-96e6-3a13c449be73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:32.808446 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.808443 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebcb215e-7f84-4653-96e6-3a13c449be73-workload-certs" pod "ebcb215e-7f84-4653-96e6-3a13c449be73" (UID: "ebcb215e-7f84-4653-96e6-3a13c449be73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:32.810261 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.810231 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebcb215e-7f84-4653-96e6-3a13c449be73-istio-podinfo" pod "ebcb215e-7f84-4653-96e6-3a13c449be73" (UID: "ebcb215e-7f84-4653-96e6-3a13c449be73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:02:32.810529 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.810505 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebcb215e-7f84-4653-96e6-3a13c449be73-istio-token" pod "ebcb215e-7f84-4653-96e6-3a13c449be73" (UID: "ebcb215e-7f84-4653-96e6-3a13c449be73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:32.811389 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.811367 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebcb215e-7f84-4653-96e6-3a13c449be73-istio-envoy" pod "ebcb215e-7f84-4653-96e6-3a13c449be73" (UID: "ebcb215e-7f84-4653-96e6-3a13c449be73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:32.811533 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.811512 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebcb215e-7f84-4653-96e6-3a13c449be73-kube-api-access-pkh5r" pod "ebcb215e-7f84-4653-96e6-3a13c449be73" (UID: "ebcb215e-7f84-4653-96e6-3a13c449be73"). InnerVolumeSpecName "kube-api-access-pkh5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:32.860094 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.860061 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6"] Oct 31 13:02:32.861454 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.861423 6841 cpu_manager.go:414] "containerMap: removing container" podUID="ebcb215e-7f84-4653-96e6-3a13c449be73" containerName="istio-proxy" Oct 31 13:02:32.861454 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.861453 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebcb215e-7f84-4653-96e6-3a13c449be73" containerName="istio-proxy" Oct 31 13:02:32.861454 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.861472 6841 cpu_manager.go:414] "containerMap: removing container" podUID="40d2bf70-21e4-498f-ac76-23246675e3bf" containerName="istio-proxy" Oct 31 13:02:32.861765 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.861481 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40d2bf70-21e4-498f-ac76-23246675e3bf" containerName="istio-proxy" Oct 31 13:02:32.861765 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.861510 6841 cpu_manager.go:414] "containerMap: removing container" podUID="ebcb215e-7f84-4653-96e6-3a13c449be73" containerName="istio-proxy" Oct 31 13:02:32.861765 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.861519 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebcb215e-7f84-4653-96e6-3a13c449be73" containerName="istio-proxy" Oct 31 13:02:32.861765 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.861757 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebcb215e-7f84-4653-96e6-3a13c449be73" containerName="istio-proxy" Oct 31 13:02:32.861765 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.861775 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="40d2bf70-21e4-498f-ac76-23246675e3bf" containerName="istio-proxy" Oct 31 13:02:32.862265 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.861790 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebcb215e-7f84-4653-96e6-3a13c449be73" containerName="istio-proxy" Oct 31 13:02:32.866418 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.866392 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:02:32.880770 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.879520 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-12056b-openshift-default-dockercfg-skq4c\"" Oct 31 13:02:32.892150 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.887039 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6"] Oct 31 13:02:32.915004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.912520 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13c0b6da-6006-44b6-8677-d16ffd433df2-istiod-ca-cert\") pod \"e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6\" (UID: \"13c0b6da-6006-44b6-8677-d16ffd433df2\") " pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:02:32.915004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.912585 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gqqg\" (UniqueName: \"kubernetes.io/projected/13c0b6da-6006-44b6-8677-d16ffd433df2-kube-api-access-7gqqg\") pod \"e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6\" (UID: \"13c0b6da-6006-44b6-8677-d16ffd433df2\") " pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:02:32.915004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.912636 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13c0b6da-6006-44b6-8677-d16ffd433df2-workload-certs\") pod \"e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6\" (UID: \"13c0b6da-6006-44b6-8677-d16ffd433df2\") " pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:02:32.915004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.912712 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13c0b6da-6006-44b6-8677-d16ffd433df2-istio-podinfo\") pod \"e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6\" (UID: \"13c0b6da-6006-44b6-8677-d16ffd433df2\") " pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:02:32.915004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.912756 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13c0b6da-6006-44b6-8677-d16ffd433df2-istio-envoy\") pod \"e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6\" (UID: \"13c0b6da-6006-44b6-8677-d16ffd433df2\") " pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:02:32.915004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.912811 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13c0b6da-6006-44b6-8677-d16ffd433df2-istio-data\") pod \"e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6\" (UID: \"13c0b6da-6006-44b6-8677-d16ffd433df2\") " pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:02:32.915004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.912856 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13c0b6da-6006-44b6-8677-d16ffd433df2-istio-token\") pod \"e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6\" (UID: \"13c0b6da-6006-44b6-8677-d16ffd433df2\") " pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:02:32.915004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.912975 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c0b6da-6006-44b6-8677-d16ffd433df2-workload-socket\") pod \"e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6\" (UID: \"13c0b6da-6006-44b6-8677-d16ffd433df2\") " pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:02:32.915004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.913045 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c0b6da-6006-44b6-8677-d16ffd433df2-credential-socket\") pod \"e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6\" (UID: \"13c0b6da-6006-44b6-8677-d16ffd433df2\") " pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:02:32.915004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.913165 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebcb215e-7f84-4653-96e6-3a13c449be73-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:32.915004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.913180 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebcb215e-7f84-4653-96e6-3a13c449be73-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:32.915004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.913200 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebcb215e-7f84-4653-96e6-3a13c449be73-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:32.915004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.913213 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebcb215e-7f84-4653-96e6-3a13c449be73-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:32.915004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.913227 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebcb215e-7f84-4653-96e6-3a13c449be73-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:32.915004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.913241 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebcb215e-7f84-4653-96e6-3a13c449be73-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:32.915004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.913258 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkh5r\" (UniqueName: \"kubernetes.io/projected/ebcb215e-7f84-4653-96e6-3a13c449be73-kube-api-access-pkh5r\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:32.915004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.913271 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebcb215e-7f84-4653-96e6-3a13c449be73-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:32.915004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.913284 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebcb215e-7f84-4653-96e6-3a13c449be73-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:32.932543 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.931701 6841 generic.go:356] "Generic (PLEG): container finished" podID="ebcb215e-7f84-4653-96e6-3a13c449be73" containerID="54c965bd8bde25b4e49d2af75f2e8bfdd691ab7c0dbf680572bf5e32ca966e7e" exitCode=0 Oct 31 13:02:32.932543 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.931796 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" event={"ID":"ebcb215e-7f84-4653-96e6-3a13c449be73","Type":"ContainerDied","Data":"54c965bd8bde25b4e49d2af75f2e8bfdd691ab7c0dbf680572bf5e32ca966e7e"} Oct 31 13:02:32.932543 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.931808 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" Oct 31 13:02:32.932543 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.931880 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2" event={"ID":"ebcb215e-7f84-4653-96e6-3a13c449be73","Type":"ContainerDied","Data":"234dd507286c3ed7eb92d571434469a7197c7d9258885a6b8925f7fc8caa63ee"} Oct 31 13:02:32.932543 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:32.931918 6841 scope.go:122] "RemoveContainer" containerID="54c965bd8bde25b4e49d2af75f2e8bfdd691ab7c0dbf680572bf5e32ca966e7e" Oct 31 13:02:33.046467 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.028315 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13c0b6da-6006-44b6-8677-d16ffd433df2-istio-token\") pod \"e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6\" (UID: \"13c0b6da-6006-44b6-8677-d16ffd433df2\") " pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:02:33.046467 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.028455 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c0b6da-6006-44b6-8677-d16ffd433df2-workload-socket\") pod \"e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6\" (UID: \"13c0b6da-6006-44b6-8677-d16ffd433df2\") " pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:02:33.046467 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.028508 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c0b6da-6006-44b6-8677-d16ffd433df2-credential-socket\") pod \"e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6\" (UID: \"13c0b6da-6006-44b6-8677-d16ffd433df2\") " pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:02:33.046467 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.028598 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13c0b6da-6006-44b6-8677-d16ffd433df2-istiod-ca-cert\") pod \"e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6\" (UID: \"13c0b6da-6006-44b6-8677-d16ffd433df2\") " pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:02:33.046467 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.028639 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gqqg\" (UniqueName: \"kubernetes.io/projected/13c0b6da-6006-44b6-8677-d16ffd433df2-kube-api-access-7gqqg\") pod \"e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6\" (UID: \"13c0b6da-6006-44b6-8677-d16ffd433df2\") " pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:02:33.046467 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.028682 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13c0b6da-6006-44b6-8677-d16ffd433df2-workload-certs\") pod \"e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6\" (UID: \"13c0b6da-6006-44b6-8677-d16ffd433df2\") " pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:02:33.046467 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.028757 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13c0b6da-6006-44b6-8677-d16ffd433df2-istio-podinfo\") pod \"e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6\" (UID: \"13c0b6da-6006-44b6-8677-d16ffd433df2\") " pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:02:33.046467 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.028804 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13c0b6da-6006-44b6-8677-d16ffd433df2-istio-envoy\") pod \"e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6\" (UID: \"13c0b6da-6006-44b6-8677-d16ffd433df2\") " pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:02:33.046467 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.028860 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13c0b6da-6006-44b6-8677-d16ffd433df2-istio-data\") pod \"e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6\" (UID: \"13c0b6da-6006-44b6-8677-d16ffd433df2\") " pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:02:33.046467 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.040712 6841 scope.go:122] "RemoveContainer" containerID="f86a3b89d8a0a8abe80b4cab3de2a83e592196d5d72d48535306d7013fb4da73" Oct 31 13:02:33.046467 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.041698 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13c0b6da-6006-44b6-8677-d16ffd433df2-workload-certs\") pod \"e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6\" (UID: \"13c0b6da-6006-44b6-8677-d16ffd433df2\") " pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:02:33.046467 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.041841 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13c0b6da-6006-44b6-8677-d16ffd433df2-istiod-ca-cert\") pod \"e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6\" (UID: \"13c0b6da-6006-44b6-8677-d16ffd433df2\") " pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:02:33.046467 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.042275 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c0b6da-6006-44b6-8677-d16ffd433df2-workload-socket\") pod \"e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6\" (UID: \"13c0b6da-6006-44b6-8677-d16ffd433df2\") " pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:02:33.046467 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.042368 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c0b6da-6006-44b6-8677-d16ffd433df2-credential-socket\") pod \"e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6\" (UID: \"13c0b6da-6006-44b6-8677-d16ffd433df2\") " pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:02:33.090152 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.068068 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13c0b6da-6006-44b6-8677-d16ffd433df2-istio-data\") pod \"e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6\" (UID: \"13c0b6da-6006-44b6-8677-d16ffd433df2\") " pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:02:33.090152 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.088063 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:33.153150 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.150910 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2"] Oct 31 13:02:33.177035 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.173744 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gqqg\" (UniqueName: \"kubernetes.io/projected/13c0b6da-6006-44b6-8677-d16ffd433df2-kube-api-access-7gqqg\") pod \"e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6\" (UID: \"13c0b6da-6006-44b6-8677-d16ffd433df2\") " pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:02:33.177035 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.174036 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13c0b6da-6006-44b6-8677-d16ffd433df2-istio-envoy\") pod \"e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6\" (UID: \"13c0b6da-6006-44b6-8677-d16ffd433df2\") " pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:02:33.177035 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.174552 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13c0b6da-6006-44b6-8677-d16ffd433df2-istio-podinfo\") pod \"e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6\" (UID: \"13c0b6da-6006-44b6-8677-d16ffd433df2\") " pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:02:33.222047 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.221935 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13c0b6da-6006-44b6-8677-d16ffd433df2-istio-token\") pod \"e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6\" (UID: \"13c0b6da-6006-44b6-8677-d16ffd433df2\") " pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:02:33.249245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.245555 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-ccf963-openshift-default-5768d56967-c8bk2"] Oct 31 13:02:33.249815 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.249507 6841 scope.go:122] "RemoveContainer" containerID="54c965bd8bde25b4e49d2af75f2e8bfdd691ab7c0dbf680572bf5e32ca966e7e" Oct 31 13:02:33.250325 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:02:33.250050 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54c965bd8bde25b4e49d2af75f2e8bfdd691ab7c0dbf680572bf5e32ca966e7e\": container with ID starting with 54c965bd8bde25b4e49d2af75f2e8bfdd691ab7c0dbf680572bf5e32ca966e7e not found: ID does not exist" containerID="54c965bd8bde25b4e49d2af75f2e8bfdd691ab7c0dbf680572bf5e32ca966e7e" Oct 31 13:02:33.250325 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.250080 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54c965bd8bde25b4e49d2af75f2e8bfdd691ab7c0dbf680572bf5e32ca966e7e"} err="failed to get container status \"54c965bd8bde25b4e49d2af75f2e8bfdd691ab7c0dbf680572bf5e32ca966e7e\": rpc error: code = NotFound desc = could not find container \"54c965bd8bde25b4e49d2af75f2e8bfdd691ab7c0dbf680572bf5e32ca966e7e\": container with ID starting with 54c965bd8bde25b4e49d2af75f2e8bfdd691ab7c0dbf680572bf5e32ca966e7e not found: ID does not exist" Oct 31 13:02:33.250325 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.250105 6841 scope.go:122] "RemoveContainer" containerID="f86a3b89d8a0a8abe80b4cab3de2a83e592196d5d72d48535306d7013fb4da73" Oct 31 13:02:33.250717 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:02:33.250635 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f86a3b89d8a0a8abe80b4cab3de2a83e592196d5d72d48535306d7013fb4da73\": container with ID starting with f86a3b89d8a0a8abe80b4cab3de2a83e592196d5d72d48535306d7013fb4da73 not found: ID does not exist" containerID="f86a3b89d8a0a8abe80b4cab3de2a83e592196d5d72d48535306d7013fb4da73" Oct 31 13:02:33.250717 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.250667 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f86a3b89d8a0a8abe80b4cab3de2a83e592196d5d72d48535306d7013fb4da73"} err="failed to get container status \"f86a3b89d8a0a8abe80b4cab3de2a83e592196d5d72d48535306d7013fb4da73\": rpc error: code = NotFound desc = could not find container \"f86a3b89d8a0a8abe80b4cab3de2a83e592196d5d72d48535306d7013fb4da73\": container with ID starting with f86a3b89d8a0a8abe80b4cab3de2a83e592196d5d72d48535306d7013fb4da73 not found: ID does not exist" Oct 31 13:02:33.512851 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.511598 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:02:33.524577 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.522746 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm"] Oct 31 13:02:33.524577 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.523008 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" podUID="164b7cbb-c80d-4709-b5d0-5bdba8070ce6" containerName="istio-proxy" containerID="cri-o://c9a43971a23e2288e79ffc3c02714ba9bc0cc61090695b688fd66c15be4a6c98" gracePeriod=30 Oct 31 13:02:33.527503 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.527476 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh"] Oct 31 13:02:33.527722 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.527698 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" podUID="1d9c49d7-4979-4d5c-b94e-d477215dd938" containerName="istio-proxy" containerID="cri-o://892f364691264adda080aa0dba36fe773fbd1c42c7a07167ecf3f3b675cbd2c7" gracePeriod=30 Oct 31 13:02:33.560517 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.556349 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:33.647045 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.646970 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40d2bf70-21e4-498f-ac76-23246675e3bf" path="/var/lib/kubelet/pods/40d2bf70-21e4-498f-ac76-23246675e3bf/volumes" Oct 31 13:02:33.647590 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.647569 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebcb215e-7f84-4653-96e6-3a13c449be73" path="/var/lib/kubelet/pods/ebcb215e-7f84-4653-96e6-3a13c449be73/volumes" Oct 31 13:02:33.822156 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:02:33.822119 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13c0b6da_6006_44b6_8677_d16ffd433df2.slice/crio-a86ee5435d933dba7cbadcb64dd181a23ddd6affda06ff73154217205ded0c57 WatchSource:0}: Error finding container a86ee5435d933dba7cbadcb64dd181a23ddd6affda06ff73154217205ded0c57: Status 404 returned error can't find the container with id a86ee5435d933dba7cbadcb64dd181a23ddd6affda06ff73154217205ded0c57 Oct 31 13:02:33.822963 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.822935 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6"] Oct 31 13:02:33.824916 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.824876 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:02:33.825025 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.824945 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:02:33.825025 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.824974 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:02:33.955685 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:33.955646 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" event={"ID":"13c0b6da-6006-44b6-8677-d16ffd433df2","Type":"ContainerStarted","Data":"a86ee5435d933dba7cbadcb64dd181a23ddd6affda06ff73154217205ded0c57"} Oct 31 13:02:34.036428 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:34.036384 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:34.653292 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:34.653255 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:34.861452 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:34.861413 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:34.964376 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:34.964336 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" event={"ID":"13c0b6da-6006-44b6-8677-d16ffd433df2","Type":"ContainerStarted","Data":"4973ccb4472adf76b2d03bb79267b4580ef815a6276c6e778ad6376100b09d2d"} Oct 31 13:02:34.992947 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:34.992888 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" podStartSLOduration=2.992874379 podStartE2EDuration="2.992874379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:02:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:34.988359903 +0000 UTC m=+1801.471482813" watchObservedRunningTime="2026-10-31 13:02:34.992874379 +0000 UTC m=+1801.475997279" Oct 31 13:02:35.241404 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:35.241315 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:35.301890 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:35.301857 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-6f58649d67-qjsfw"] Oct 31 13:02:35.302584 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:35.302567 6841 cpu_manager.go:414] "containerMap: removing container" podUID="40d2bf70-21e4-498f-ac76-23246675e3bf" containerName="istio-proxy" Oct 31 13:02:35.302649 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:35.302585 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40d2bf70-21e4-498f-ac76-23246675e3bf" containerName="istio-proxy" Oct 31 13:02:35.302707 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:35.302698 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="40d2bf70-21e4-498f-ac76-23246675e3bf" containerName="istio-proxy" Oct 31 13:02:35.306649 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:35.306628 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-6f58649d67-qjsfw" Oct 31 13:02:35.315428 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:35.315402 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4989f9\"" Oct 31 13:02:35.318705 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:35.318662 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-6f58649d67-qjsfw"] Oct 31 13:02:35.406647 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:35.406606 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f47h\" (UniqueName: \"kubernetes.io/projected/d03bab9c-ade2-43da-8bb1-37fc53fd93f9-kube-api-access-2f47h\") pod \"payload-pre-processing-e2e-ipp-a-4989f9-6f58649d67-qjsfw\" (UID: \"d03bab9c-ade2-43da-8bb1-37fc53fd93f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-6f58649d67-qjsfw" Oct 31 13:02:35.406933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:35.406912 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d03bab9c-ade2-43da-8bb1-37fc53fd93f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4989f9-6f58649d67-qjsfw\" (UID: \"d03bab9c-ade2-43da-8bb1-37fc53fd93f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-6f58649d67-qjsfw" Oct 31 13:02:35.476791 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:35.476751 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:35.525510 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:35.525357 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:02:35.527896 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:35.526676 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d03bab9c-ade2-43da-8bb1-37fc53fd93f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4989f9-6f58649d67-qjsfw\" (UID: \"d03bab9c-ade2-43da-8bb1-37fc53fd93f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-6f58649d67-qjsfw" Oct 31 13:02:35.528217 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:35.528198 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f47h\" (UniqueName: \"kubernetes.io/projected/d03bab9c-ade2-43da-8bb1-37fc53fd93f9-kube-api-access-2f47h\") pod \"payload-pre-processing-e2e-ipp-a-4989f9-6f58649d67-qjsfw\" (UID: \"d03bab9c-ade2-43da-8bb1-37fc53fd93f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-6f58649d67-qjsfw" Oct 31 13:02:35.528881 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:35.527764 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d03bab9c-ade2-43da-8bb1-37fc53fd93f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4989f9-6f58649d67-qjsfw\" (UID: \"d03bab9c-ade2-43da-8bb1-37fc53fd93f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-6f58649d67-qjsfw" Oct 31 13:02:35.569478 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:35.569440 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f47h\" (UniqueName: \"kubernetes.io/projected/d03bab9c-ade2-43da-8bb1-37fc53fd93f9-kube-api-access-2f47h\") pod \"payload-pre-processing-e2e-ipp-a-4989f9-6f58649d67-qjsfw\" (UID: \"d03bab9c-ade2-43da-8bb1-37fc53fd93f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-6f58649d67-qjsfw" Oct 31 13:02:35.615793 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:35.615765 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:02:35.655098 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:35.625413 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-6f58649d67-qjsfw" Oct 31 13:02:35.756779 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:35.756747 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4989f9-67d6485cdd-xndwk"] Oct 31 13:02:35.762636 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:35.762614 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4989f9-67d6485cdd-xndwk"] Oct 31 13:02:35.762760 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:35.762716 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4989f9-67d6485cdd-xndwk" Oct 31 13:02:35.768127 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:35.768103 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4989f9-dockercfg-795jh\"" Oct 31 13:02:35.838915 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:35.838879 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd92l\" (UniqueName: \"kubernetes.io/projected/e024b9a0-c5a3-4850-914a-fe60e0c6f50e-kube-api-access-qd92l\") pod \"payload-processing-e2e-ipp-a-4989f9-67d6485cdd-xndwk\" (UID: \"e024b9a0-c5a3-4850-914a-fe60e0c6f50e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4989f9-67d6485cdd-xndwk" Oct 31 13:02:35.839130 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:35.839104 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e024b9a0-c5a3-4850-914a-fe60e0c6f50e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4989f9-67d6485cdd-xndwk\" (UID: \"e024b9a0-c5a3-4850-914a-fe60e0c6f50e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4989f9-67d6485cdd-xndwk" Oct 31 13:02:35.940971 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:35.940940 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e024b9a0-c5a3-4850-914a-fe60e0c6f50e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4989f9-67d6485cdd-xndwk\" (UID: \"e024b9a0-c5a3-4850-914a-fe60e0c6f50e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4989f9-67d6485cdd-xndwk" Oct 31 13:02:35.941198 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:35.941177 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd92l\" (UniqueName: \"kubernetes.io/projected/e024b9a0-c5a3-4850-914a-fe60e0c6f50e-kube-api-access-qd92l\") pod \"payload-processing-e2e-ipp-a-4989f9-67d6485cdd-xndwk\" (UID: \"e024b9a0-c5a3-4850-914a-fe60e0c6f50e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4989f9-67d6485cdd-xndwk" Oct 31 13:02:35.941698 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:35.941678 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e024b9a0-c5a3-4850-914a-fe60e0c6f50e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4989f9-67d6485cdd-xndwk\" (UID: \"e024b9a0-c5a3-4850-914a-fe60e0c6f50e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4989f9-67d6485cdd-xndwk" Oct 31 13:02:35.973401 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:35.973372 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd92l\" (UniqueName: \"kubernetes.io/projected/e024b9a0-c5a3-4850-914a-fe60e0c6f50e-kube-api-access-qd92l\") pod \"payload-processing-e2e-ipp-a-4989f9-67d6485cdd-xndwk\" (UID: \"e024b9a0-c5a3-4850-914a-fe60e0c6f50e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4989f9-67d6485cdd-xndwk" Oct 31 13:02:35.976731 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:35.976709 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:02:35.977599 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:35.977579 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:02:36.128230 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.127333 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4989f9-67d6485cdd-xndwk" Oct 31 13:02:36.140416 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.140293 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-6f58649d67-qjsfw"] Oct 31 13:02:36.262716 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.262682 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:36.457579 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.457554 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4989f9-67d6485cdd-xndwk"] Oct 31 13:02:36.459223 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:02:36.459195 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode024b9a0_c5a3_4850_914a_fe60e0c6f50e.slice/crio-21328748cb76e12e8dffd16d90e91dc9b57168332c319a2b305b0e6f075e319e WatchSource:0}: Error finding container 21328748cb76e12e8dffd16d90e91dc9b57168332c319a2b305b0e6f075e319e: Status 404 returned error can't find the container with id 21328748cb76e12e8dffd16d90e91dc9b57168332c319a2b305b0e6f075e319e Oct 31 13:02:36.675754 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.675717 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:36.826211 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.826180 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:02:36.885113 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.885076 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:36.966224 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.966135 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd421e14-02ab-4072-918a-e3c6077263c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd421e14-02ab-4072-918a-e3c6077263c8-istiod-ca-cert\") pod \"cd421e14-02ab-4072-918a-e3c6077263c8\" (UID: \"cd421e14-02ab-4072-918a-e3c6077263c8\") " Oct 31 13:02:36.966224 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.966191 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd421e14-02ab-4072-918a-e3c6077263c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd421e14-02ab-4072-918a-e3c6077263c8-credential-socket\") pod \"cd421e14-02ab-4072-918a-e3c6077263c8\" (UID: \"cd421e14-02ab-4072-918a-e3c6077263c8\") " Oct 31 13:02:36.966430 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.966232 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd421e14-02ab-4072-918a-e3c6077263c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd421e14-02ab-4072-918a-e3c6077263c8-workload-socket\") pod \"cd421e14-02ab-4072-918a-e3c6077263c8\" (UID: \"cd421e14-02ab-4072-918a-e3c6077263c8\") " Oct 31 13:02:36.966430 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.966290 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd421e14-02ab-4072-918a-e3c6077263c8-istio-token\" (UniqueName: \"kubernetes.io/projected/cd421e14-02ab-4072-918a-e3c6077263c8-istio-token\") pod \"cd421e14-02ab-4072-918a-e3c6077263c8\" (UID: \"cd421e14-02ab-4072-918a-e3c6077263c8\") " Oct 31 13:02:36.966430 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.966317 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd421e14-02ab-4072-918a-e3c6077263c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd421e14-02ab-4072-918a-e3c6077263c8-istio-data\") pod \"cd421e14-02ab-4072-918a-e3c6077263c8\" (UID: \"cd421e14-02ab-4072-918a-e3c6077263c8\") " Oct 31 13:02:36.966586 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.966497 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd421e14-02ab-4072-918a-e3c6077263c8-credential-socket" pod "cd421e14-02ab-4072-918a-e3c6077263c8" (UID: "cd421e14-02ab-4072-918a-e3c6077263c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:36.966643 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.966578 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd421e14-02ab-4072-918a-e3c6077263c8-istiod-ca-cert" pod "cd421e14-02ab-4072-918a-e3c6077263c8" (UID: "cd421e14-02ab-4072-918a-e3c6077263c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:02:36.966693 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.966578 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd421e14-02ab-4072-918a-e3c6077263c8-workload-socket" pod "cd421e14-02ab-4072-918a-e3c6077263c8" (UID: "cd421e14-02ab-4072-918a-e3c6077263c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:36.966693 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.966633 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd421e14-02ab-4072-918a-e3c6077263c8-istio-data" pod "cd421e14-02ab-4072-918a-e3c6077263c8" (UID: "cd421e14-02ab-4072-918a-e3c6077263c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:36.966875 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.966855 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd421e14-02ab-4072-918a-e3c6077263c8-kube-api-access-tllhz\" (UniqueName: \"kubernetes.io/projected/cd421e14-02ab-4072-918a-e3c6077263c8-kube-api-access-tllhz\") pod \"cd421e14-02ab-4072-918a-e3c6077263c8\" (UID: \"cd421e14-02ab-4072-918a-e3c6077263c8\") " Oct 31 13:02:36.966950 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.966934 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd421e14-02ab-4072-918a-e3c6077263c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd421e14-02ab-4072-918a-e3c6077263c8-workload-certs\") pod \"cd421e14-02ab-4072-918a-e3c6077263c8\" (UID: \"cd421e14-02ab-4072-918a-e3c6077263c8\") " Oct 31 13:02:36.967095 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.967075 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd421e14-02ab-4072-918a-e3c6077263c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd421e14-02ab-4072-918a-e3c6077263c8-istio-envoy\") pod \"cd421e14-02ab-4072-918a-e3c6077263c8\" (UID: \"cd421e14-02ab-4072-918a-e3c6077263c8\") " Oct 31 13:02:36.967359 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.967201 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd421e14-02ab-4072-918a-e3c6077263c8-workload-certs" pod "cd421e14-02ab-4072-918a-e3c6077263c8" (UID: "cd421e14-02ab-4072-918a-e3c6077263c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:36.967451 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.967428 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd421e14-02ab-4072-918a-e3c6077263c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd421e14-02ab-4072-918a-e3c6077263c8-istio-podinfo\") pod \"cd421e14-02ab-4072-918a-e3c6077263c8\" (UID: \"cd421e14-02ab-4072-918a-e3c6077263c8\") " Oct 31 13:02:36.968058 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.968037 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd421e14-02ab-4072-918a-e3c6077263c8-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:36.968164 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.968060 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd421e14-02ab-4072-918a-e3c6077263c8-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:36.968164 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.968075 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd421e14-02ab-4072-918a-e3c6077263c8-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:36.968164 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.968090 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd421e14-02ab-4072-918a-e3c6077263c8-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:36.968164 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.968104 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd421e14-02ab-4072-918a-e3c6077263c8-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:36.969237 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.969209 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd421e14-02ab-4072-918a-e3c6077263c8-kube-api-access-tllhz" pod "cd421e14-02ab-4072-918a-e3c6077263c8" (UID: "cd421e14-02ab-4072-918a-e3c6077263c8"). InnerVolumeSpecName "kube-api-access-tllhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:36.969331 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.969298 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd421e14-02ab-4072-918a-e3c6077263c8-istio-token" pod "cd421e14-02ab-4072-918a-e3c6077263c8" (UID: "cd421e14-02ab-4072-918a-e3c6077263c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:36.969395 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.969376 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd421e14-02ab-4072-918a-e3c6077263c8-istio-podinfo" pod "cd421e14-02ab-4072-918a-e3c6077263c8" (UID: "cd421e14-02ab-4072-918a-e3c6077263c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:02:36.969496 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.969482 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd421e14-02ab-4072-918a-e3c6077263c8-istio-envoy" pod "cd421e14-02ab-4072-918a-e3c6077263c8" (UID: "cd421e14-02ab-4072-918a-e3c6077263c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:36.989463 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.989438 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4989f9-67d6485cdd-xndwk" event={"ID":"e024b9a0-c5a3-4850-914a-fe60e0c6f50e","Type":"ContainerStarted","Data":"d7ef6d4a67dcfb8f9dc0b7aca5be7cf45cac2bf4bb0a6c9ec81cc6fbc86bed52"} Oct 31 13:02:36.989569 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.989494 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4989f9-67d6485cdd-xndwk" event={"ID":"e024b9a0-c5a3-4850-914a-fe60e0c6f50e","Type":"ContainerStarted","Data":"21328748cb76e12e8dffd16d90e91dc9b57168332c319a2b305b0e6f075e319e"} Oct 31 13:02:36.989569 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.989532 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4989f9-67d6485cdd-xndwk" Oct 31 13:02:36.991282 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.991257 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-6f58649d67-qjsfw" event={"ID":"d03bab9c-ade2-43da-8bb1-37fc53fd93f9","Type":"ContainerStarted","Data":"500b0e45b909b37cffe613d3f3b32e10882ba995055d5722fcc0f9233618774f"} Oct 31 13:02:36.991367 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.991303 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-6f58649d67-qjsfw" event={"ID":"d03bab9c-ade2-43da-8bb1-37fc53fd93f9","Type":"ContainerStarted","Data":"399e836d3b2bdff69771bd4cd224e2fe0f127b40b7418833706cae3dea0fb50e"} Oct 31 13:02:36.991367 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.991335 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-6f58649d67-qjsfw" Oct 31 13:02:36.993031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.993006 6841 generic.go:356] "Generic (PLEG): container finished" podID="cd421e14-02ab-4072-918a-e3c6077263c8" containerID="44c37590ceca231b5b5ba8c3e595cee0f52dee2b714a0671f8cad56d504b48dd" exitCode=0 Oct 31 13:02:36.993099 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.993052 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" Oct 31 13:02:36.993099 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.993080 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" event={"ID":"cd421e14-02ab-4072-918a-e3c6077263c8","Type":"ContainerDied","Data":"44c37590ceca231b5b5ba8c3e595cee0f52dee2b714a0671f8cad56d504b48dd"} Oct 31 13:02:36.993170 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.993131 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt" event={"ID":"cd421e14-02ab-4072-918a-e3c6077263c8","Type":"ContainerDied","Data":"03fd4b1ffc0798f75eb763c21600430ba9d2d1873f994f2d35986b86c00f8e8e"} Oct 31 13:02:36.993221 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:36.993182 6841 scope.go:122] "RemoveContainer" containerID="44c37590ceca231b5b5ba8c3e595cee0f52dee2b714a0671f8cad56d504b48dd" Oct 31 13:02:37.003841 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:37.003825 6841 scope.go:122] "RemoveContainer" containerID="5ee31ac384bff02e3313e7c063614a58abdc9a5ab2e8039edaa7438800835cd8" Oct 31 13:02:37.013030 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:37.013012 6841 scope.go:122] "RemoveContainer" containerID="44c37590ceca231b5b5ba8c3e595cee0f52dee2b714a0671f8cad56d504b48dd" Oct 31 13:02:37.013288 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:02:37.013268 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44c37590ceca231b5b5ba8c3e595cee0f52dee2b714a0671f8cad56d504b48dd\": container with ID starting with 44c37590ceca231b5b5ba8c3e595cee0f52dee2b714a0671f8cad56d504b48dd not found: ID does not exist" containerID="44c37590ceca231b5b5ba8c3e595cee0f52dee2b714a0671f8cad56d504b48dd" Oct 31 13:02:37.013379 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:37.013298 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44c37590ceca231b5b5ba8c3e595cee0f52dee2b714a0671f8cad56d504b48dd"} err="failed to get container status \"44c37590ceca231b5b5ba8c3e595cee0f52dee2b714a0671f8cad56d504b48dd\": rpc error: code = NotFound desc = could not find container \"44c37590ceca231b5b5ba8c3e595cee0f52dee2b714a0671f8cad56d504b48dd\": container with ID starting with 44c37590ceca231b5b5ba8c3e595cee0f52dee2b714a0671f8cad56d504b48dd not found: ID does not exist" Oct 31 13:02:37.013379 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:37.013319 6841 scope.go:122] "RemoveContainer" containerID="5ee31ac384bff02e3313e7c063614a58abdc9a5ab2e8039edaa7438800835cd8" Oct 31 13:02:37.013590 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:02:37.013571 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ee31ac384bff02e3313e7c063614a58abdc9a5ab2e8039edaa7438800835cd8\": container with ID starting with 5ee31ac384bff02e3313e7c063614a58abdc9a5ab2e8039edaa7438800835cd8 not found: ID does not exist" containerID="5ee31ac384bff02e3313e7c063614a58abdc9a5ab2e8039edaa7438800835cd8" Oct 31 13:02:37.013650 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:37.013596 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ee31ac384bff02e3313e7c063614a58abdc9a5ab2e8039edaa7438800835cd8"} err="failed to get container status \"5ee31ac384bff02e3313e7c063614a58abdc9a5ab2e8039edaa7438800835cd8\": rpc error: code = NotFound desc = could not find container \"5ee31ac384bff02e3313e7c063614a58abdc9a5ab2e8039edaa7438800835cd8\": container with ID starting with 5ee31ac384bff02e3313e7c063614a58abdc9a5ab2e8039edaa7438800835cd8 not found: ID does not exist" Oct 31 13:02:37.021022 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:37.020971 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-4989f9-67d6485cdd-xndwk" podStartSLOduration=2.020959386 podStartE2EDuration="2.020959386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:37.015515414 +0000 UTC m=+1803.498638314" watchObservedRunningTime="2026-10-31 13:02:37.020959386 +0000 UTC m=+1803.504082289" Oct 31 13:02:37.043283 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:37.043212 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-6f58649d67-qjsfw" podStartSLOduration=2.043199941 podStartE2EDuration="2.043199941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:37.039857798 +0000 UTC m=+1803.522980704" watchObservedRunningTime="2026-10-31 13:02:37.043199941 +0000 UTC m=+1803.526322841" Oct 31 13:02:37.057040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:37.056946 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt"] Oct 31 13:02:37.061302 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:37.061277 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-01e8d2-openshift-default-df6fc88c9-chcvt"] Oct 31 13:02:37.069803 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:37.069778 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd421e14-02ab-4072-918a-e3c6077263c8-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:37.069803 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:37.069803 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tllhz\" (UniqueName: \"kubernetes.io/projected/cd421e14-02ab-4072-918a-e3c6077263c8-kube-api-access-tllhz\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:37.069928 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:37.069814 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd421e14-02ab-4072-918a-e3c6077263c8-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:37.069928 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:37.069833 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd421e14-02ab-4072-918a-e3c6077263c8-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:37.260090 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:37.250901 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:37.646183 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:37.646109 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd421e14-02ab-4072-918a-e3c6077263c8" path="/var/lib/kubelet/pods/cd421e14-02ab-4072-918a-e3c6077263c8/volumes" Oct 31 13:02:37.832022 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:37.831969 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:38.070844 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.070803 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:38.378965 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.378878 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:38.835547 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.835521 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" Oct 31 13:02:38.839147 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.839119 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" Oct 31 13:02:38.910073 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.909924 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-istiod-ca-cert\") pod \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\" (UID: \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\") " Oct 31 13:02:38.910327 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.910303 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-istiod-ca-cert" pod "164b7cbb-c80d-4709-b5d0-5bdba8070ce6" (UID: "164b7cbb-c80d-4709-b5d0-5bdba8070ce6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:02:38.910438 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.910418 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d9c49d7-4979-4d5c-b94e-d477215dd938-kube-api-access-9ggqf\" (UniqueName: \"kubernetes.io/projected/1d9c49d7-4979-4d5c-b94e-d477215dd938-kube-api-access-9ggqf\") pod \"1d9c49d7-4979-4d5c-b94e-d477215dd938\" (UID: \"1d9c49d7-4979-4d5c-b94e-d477215dd938\") " Oct 31 13:02:38.910797 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.910774 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-kube-api-access-jgzb6\" (UniqueName: \"kubernetes.io/projected/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-kube-api-access-jgzb6\") pod \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\" (UID: \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\") " Oct 31 13:02:38.910876 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.910817 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d9c49d7-4979-4d5c-b94e-d477215dd938-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d9c49d7-4979-4d5c-b94e-d477215dd938-workload-certs\") pod \"1d9c49d7-4979-4d5c-b94e-d477215dd938\" (UID: \"1d9c49d7-4979-4d5c-b94e-d477215dd938\") " Oct 31 13:02:38.910876 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.910845 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d9c49d7-4979-4d5c-b94e-d477215dd938-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d9c49d7-4979-4d5c-b94e-d477215dd938-istio-podinfo\") pod \"1d9c49d7-4979-4d5c-b94e-d477215dd938\" (UID: \"1d9c49d7-4979-4d5c-b94e-d477215dd938\") " Oct 31 13:02:38.911003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.910887 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-workload-socket\") pod \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\" (UID: \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\") " Oct 31 13:02:38.911003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.910925 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-workload-certs\") pod \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\" (UID: \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\") " Oct 31 13:02:38.911522 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.911491 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d9c49d7-4979-4d5c-b94e-d477215dd938-istio-token\" (UniqueName: \"kubernetes.io/projected/1d9c49d7-4979-4d5c-b94e-d477215dd938-istio-token\") pod \"1d9c49d7-4979-4d5c-b94e-d477215dd938\" (UID: \"1d9c49d7-4979-4d5c-b94e-d477215dd938\") " Oct 31 13:02:38.911639 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.911541 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-credential-socket\") pod \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\" (UID: \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\") " Oct 31 13:02:38.911639 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.911582 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-istio-podinfo\") pod \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\" (UID: \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\") " Oct 31 13:02:38.911639 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.911610 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-istio-token\" (UniqueName: \"kubernetes.io/projected/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-istio-token\") pod \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\" (UID: \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\") " Oct 31 13:02:38.911896 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.911656 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d9c49d7-4979-4d5c-b94e-d477215dd938-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d9c49d7-4979-4d5c-b94e-d477215dd938-istiod-ca-cert\") pod \"1d9c49d7-4979-4d5c-b94e-d477215dd938\" (UID: \"1d9c49d7-4979-4d5c-b94e-d477215dd938\") " Oct 31 13:02:38.911896 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.911706 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-istio-data\") pod \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\" (UID: \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\") " Oct 31 13:02:38.911896 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.911753 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d9c49d7-4979-4d5c-b94e-d477215dd938-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9c49d7-4979-4d5c-b94e-d477215dd938-credential-socket\") pod \"1d9c49d7-4979-4d5c-b94e-d477215dd938\" (UID: \"1d9c49d7-4979-4d5c-b94e-d477215dd938\") " Oct 31 13:02:38.911896 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.911846 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d9c49d7-4979-4d5c-b94e-d477215dd938-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d9c49d7-4979-4d5c-b94e-d477215dd938-istio-data\") pod \"1d9c49d7-4979-4d5c-b94e-d477215dd938\" (UID: \"1d9c49d7-4979-4d5c-b94e-d477215dd938\") " Oct 31 13:02:38.911896 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.911875 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d9c49d7-4979-4d5c-b94e-d477215dd938-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d9c49d7-4979-4d5c-b94e-d477215dd938-istio-envoy\") pod \"1d9c49d7-4979-4d5c-b94e-d477215dd938\" (UID: \"1d9c49d7-4979-4d5c-b94e-d477215dd938\") " Oct 31 13:02:38.912262 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.911915 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d9c49d7-4979-4d5c-b94e-d477215dd938-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9c49d7-4979-4d5c-b94e-d477215dd938-workload-socket\") pod \"1d9c49d7-4979-4d5c-b94e-d477215dd938\" (UID: \"1d9c49d7-4979-4d5c-b94e-d477215dd938\") " Oct 31 13:02:38.912262 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.911951 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-istio-envoy\") pod \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\" (UID: \"164b7cbb-c80d-4709-b5d0-5bdba8070ce6\") " Oct 31 13:02:38.912262 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.912159 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-workload-socket" pod "164b7cbb-c80d-4709-b5d0-5bdba8070ce6" (UID: "164b7cbb-c80d-4709-b5d0-5bdba8070ce6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:38.912731 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.912706 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:38.912838 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.912748 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:38.913378 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.913030 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9c49d7-4979-4d5c-b94e-d477215dd938-workload-certs" pod "1d9c49d7-4979-4d5c-b94e-d477215dd938" (UID: "1d9c49d7-4979-4d5c-b94e-d477215dd938"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:38.913378 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.913225 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-credential-socket" pod "164b7cbb-c80d-4709-b5d0-5bdba8070ce6" (UID: "164b7cbb-c80d-4709-b5d0-5bdba8070ce6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:38.914031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.913587 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-workload-certs" pod "164b7cbb-c80d-4709-b5d0-5bdba8070ce6" (UID: "164b7cbb-c80d-4709-b5d0-5bdba8070ce6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:38.914429 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.914125 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9c49d7-4979-4d5c-b94e-d477215dd938-istiod-ca-cert" pod "1d9c49d7-4979-4d5c-b94e-d477215dd938" (UID: "1d9c49d7-4979-4d5c-b94e-d477215dd938"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:02:38.914974 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.914636 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-istio-envoy" pod "164b7cbb-c80d-4709-b5d0-5bdba8070ce6" (UID: "164b7cbb-c80d-4709-b5d0-5bdba8070ce6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:38.914974 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.914874 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-istio-data" pod "164b7cbb-c80d-4709-b5d0-5bdba8070ce6" (UID: "164b7cbb-c80d-4709-b5d0-5bdba8070ce6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:38.915154 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.915111 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9c49d7-4979-4d5c-b94e-d477215dd938-credential-socket" pod "1d9c49d7-4979-4d5c-b94e-d477215dd938" (UID: "1d9c49d7-4979-4d5c-b94e-d477215dd938"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:38.915299 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.915275 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9c49d7-4979-4d5c-b94e-d477215dd938-istio-data" pod "1d9c49d7-4979-4d5c-b94e-d477215dd938" (UID: "1d9c49d7-4979-4d5c-b94e-d477215dd938"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:38.915513 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.915491 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9c49d7-4979-4d5c-b94e-d477215dd938-workload-socket" pod "1d9c49d7-4979-4d5c-b94e-d477215dd938" (UID: "1d9c49d7-4979-4d5c-b94e-d477215dd938"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:38.915663 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.915634 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d9c49d7-4979-4d5c-b94e-d477215dd938-kube-api-access-9ggqf" pod "1d9c49d7-4979-4d5c-b94e-d477215dd938" (UID: "1d9c49d7-4979-4d5c-b94e-d477215dd938"). InnerVolumeSpecName "kube-api-access-9ggqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:38.917806 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.917657 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-istio-token" pod "164b7cbb-c80d-4709-b5d0-5bdba8070ce6" (UID: "164b7cbb-c80d-4709-b5d0-5bdba8070ce6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:38.921031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.918698 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-istio-podinfo" pod "164b7cbb-c80d-4709-b5d0-5bdba8070ce6" (UID: "164b7cbb-c80d-4709-b5d0-5bdba8070ce6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:02:38.921031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.919055 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-kube-api-access-jgzb6" pod "164b7cbb-c80d-4709-b5d0-5bdba8070ce6" (UID: "164b7cbb-c80d-4709-b5d0-5bdba8070ce6"). InnerVolumeSpecName "kube-api-access-jgzb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:38.921031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.919195 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d9c49d7-4979-4d5c-b94e-d477215dd938-istio-podinfo" pod "1d9c49d7-4979-4d5c-b94e-d477215dd938" (UID: "1d9c49d7-4979-4d5c-b94e-d477215dd938"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:02:38.921031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.919409 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9c49d7-4979-4d5c-b94e-d477215dd938-istio-envoy" pod "1d9c49d7-4979-4d5c-b94e-d477215dd938" (UID: "1d9c49d7-4979-4d5c-b94e-d477215dd938"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:02:38.921031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:38.921000 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d9c49d7-4979-4d5c-b94e-d477215dd938-istio-token" pod "1d9c49d7-4979-4d5c-b94e-d477215dd938" (UID: "1d9c49d7-4979-4d5c-b94e-d477215dd938"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:39.013704 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.013668 6841 generic.go:356] "Generic (PLEG): container finished" podID="1d9c49d7-4979-4d5c-b94e-d477215dd938" containerID="892f364691264adda080aa0dba36fe773fbd1c42c7a07167ecf3f3b675cbd2c7" exitCode=0 Oct 31 13:02:39.013858 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.013723 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" Oct 31 13:02:39.013858 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.013747 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" event={"ID":"1d9c49d7-4979-4d5c-b94e-d477215dd938","Type":"ContainerDied","Data":"892f364691264adda080aa0dba36fe773fbd1c42c7a07167ecf3f3b675cbd2c7"} Oct 31 13:02:39.014011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.013874 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh" event={"ID":"1d9c49d7-4979-4d5c-b94e-d477215dd938","Type":"ContainerDied","Data":"753e5965c1bad3a5a337dc59137fff4417d033e5bdd9346462628d4351997f62"} Oct 31 13:02:39.014011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.013906 6841 scope.go:122] "RemoveContainer" containerID="892f364691264adda080aa0dba36fe773fbd1c42c7a07167ecf3f3b675cbd2c7" Oct 31 13:02:39.014011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.013963 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d9c49d7-4979-4d5c-b94e-d477215dd938-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:39.014183 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.014168 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d9c49d7-4979-4d5c-b94e-d477215dd938-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:39.014242 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.014189 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9c49d7-4979-4d5c-b94e-d477215dd938-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:39.014242 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.014205 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:39.014242 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.014219 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ggqf\" (UniqueName: \"kubernetes.io/projected/1d9c49d7-4979-4d5c-b94e-d477215dd938-kube-api-access-9ggqf\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:39.014242 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.014233 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgzb6\" (UniqueName: \"kubernetes.io/projected/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-kube-api-access-jgzb6\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:39.014404 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.014248 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d9c49d7-4979-4d5c-b94e-d477215dd938-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:39.014404 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.014262 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d9c49d7-4979-4d5c-b94e-d477215dd938-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:39.014404 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.014276 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:39.014404 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.014291 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d9c49d7-4979-4d5c-b94e-d477215dd938-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:39.014404 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.014304 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:39.014404 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.014318 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:39.014404 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.014331 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:39.014404 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.014344 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d9c49d7-4979-4d5c-b94e-d477215dd938-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:39.014404 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.014357 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/164b7cbb-c80d-4709-b5d0-5bdba8070ce6-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:39.014404 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.014373 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9c49d7-4979-4d5c-b94e-d477215dd938-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:39.017570 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.017543 6841 generic.go:356] "Generic (PLEG): container finished" podID="164b7cbb-c80d-4709-b5d0-5bdba8070ce6" containerID="c9a43971a23e2288e79ffc3c02714ba9bc0cc61090695b688fd66c15be4a6c98" exitCode=0 Oct 31 13:02:39.017694 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.017597 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" Oct 31 13:02:39.017694 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.017630 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" event={"ID":"164b7cbb-c80d-4709-b5d0-5bdba8070ce6","Type":"ContainerDied","Data":"c9a43971a23e2288e79ffc3c02714ba9bc0cc61090695b688fd66c15be4a6c98"} Oct 31 13:02:39.017694 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.017675 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm" event={"ID":"164b7cbb-c80d-4709-b5d0-5bdba8070ce6","Type":"ContainerDied","Data":"16cd9edd182bf7edd61880575bfca8e8c0df01f58cb843e4c8456328447dfcf2"} Oct 31 13:02:39.032381 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.032343 6841 scope.go:122] "RemoveContainer" containerID="892f364691264adda080aa0dba36fe773fbd1c42c7a07167ecf3f3b675cbd2c7" Oct 31 13:02:39.032806 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:02:39.032785 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"892f364691264adda080aa0dba36fe773fbd1c42c7a07167ecf3f3b675cbd2c7\": container with ID starting with 892f364691264adda080aa0dba36fe773fbd1c42c7a07167ecf3f3b675cbd2c7 not found: ID does not exist" containerID="892f364691264adda080aa0dba36fe773fbd1c42c7a07167ecf3f3b675cbd2c7" Oct 31 13:02:39.032917 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.032812 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"892f364691264adda080aa0dba36fe773fbd1c42c7a07167ecf3f3b675cbd2c7"} err="failed to get container status \"892f364691264adda080aa0dba36fe773fbd1c42c7a07167ecf3f3b675cbd2c7\": rpc error: code = NotFound desc = could not find container \"892f364691264adda080aa0dba36fe773fbd1c42c7a07167ecf3f3b675cbd2c7\": container with ID starting with 892f364691264adda080aa0dba36fe773fbd1c42c7a07167ecf3f3b675cbd2c7 not found: ID does not exist" Oct 31 13:02:39.032917 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.032829 6841 scope.go:122] "RemoveContainer" containerID="c9a43971a23e2288e79ffc3c02714ba9bc0cc61090695b688fd66c15be4a6c98" Oct 31 13:02:39.044595 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.044574 6841 scope.go:122] "RemoveContainer" containerID="c9a43971a23e2288e79ffc3c02714ba9bc0cc61090695b688fd66c15be4a6c98" Oct 31 13:02:39.044860 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:02:39.044843 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9a43971a23e2288e79ffc3c02714ba9bc0cc61090695b688fd66c15be4a6c98\": container with ID starting with c9a43971a23e2288e79ffc3c02714ba9bc0cc61090695b688fd66c15be4a6c98 not found: ID does not exist" containerID="c9a43971a23e2288e79ffc3c02714ba9bc0cc61090695b688fd66c15be4a6c98" Oct 31 13:02:39.044933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.044867 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9a43971a23e2288e79ffc3c02714ba9bc0cc61090695b688fd66c15be4a6c98"} err="failed to get container status \"c9a43971a23e2288e79ffc3c02714ba9bc0cc61090695b688fd66c15be4a6c98\": rpc error: code = NotFound desc = could not find container \"c9a43971a23e2288e79ffc3c02714ba9bc0cc61090695b688fd66c15be4a6c98\": container with ID starting with c9a43971a23e2288e79ffc3c02714ba9bc0cc61090695b688fd66c15be4a6c98 not found: ID does not exist" Oct 31 13:02:39.047868 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.047844 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh"] Oct 31 13:02:39.060831 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.060810 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-57749745dc-wtl4l"] Oct 31 13:02:39.061525 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.061507 6841 cpu_manager.go:414] "containerMap: removing container" podUID="cd421e14-02ab-4072-918a-e3c6077263c8" containerName="istio-proxy" Oct 31 13:02:39.061525 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.061524 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd421e14-02ab-4072-918a-e3c6077263c8" containerName="istio-proxy" Oct 31 13:02:39.061652 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.061560 6841 cpu_manager.go:414] "containerMap: removing container" podUID="164b7cbb-c80d-4709-b5d0-5bdba8070ce6" containerName="istio-proxy" Oct 31 13:02:39.061652 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.061565 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="164b7cbb-c80d-4709-b5d0-5bdba8070ce6" containerName="istio-proxy" Oct 31 13:02:39.061652 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.061586 6841 cpu_manager.go:414] "containerMap: removing container" podUID="cd421e14-02ab-4072-918a-e3c6077263c8" containerName="istio-proxy" Oct 31 13:02:39.061652 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.061591 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd421e14-02ab-4072-918a-e3c6077263c8" containerName="istio-proxy" Oct 31 13:02:39.061652 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.061601 6841 cpu_manager.go:414] "containerMap: removing container" podUID="1d9c49d7-4979-4d5c-b94e-d477215dd938" containerName="istio-proxy" Oct 31 13:02:39.061652 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.061606 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d9c49d7-4979-4d5c-b94e-d477215dd938" containerName="istio-proxy" Oct 31 13:02:39.061952 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.061716 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd421e14-02ab-4072-918a-e3c6077263c8" containerName="istio-proxy" Oct 31 13:02:39.061952 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.061724 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d9c49d7-4979-4d5c-b94e-d477215dd938" containerName="istio-proxy" Oct 31 13:02:39.061952 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.061736 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd421e14-02ab-4072-918a-e3c6077263c8" containerName="istio-proxy" Oct 31 13:02:39.061952 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.061745 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="164b7cbb-c80d-4709-b5d0-5bdba8070ce6" containerName="istio-proxy" Oct 31 13:02:39.066847 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.066828 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-r4lgh"] Oct 31 13:02:39.066955 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.066909 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-57749745dc-wtl4l" Oct 31 13:02:39.074136 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.072559 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-12056b-dockercfg-lmf2p\"" Oct 31 13:02:39.077132 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.077000 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-12056b\"" Oct 31 13:02:39.078793 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.078742 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-57749745dc-wtl4l"] Oct 31 13:02:39.084473 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.084447 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm"] Oct 31 13:02:39.089414 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.089390 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-9rtdm"] Oct 31 13:02:39.098390 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.098350 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:39.115259 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.115227 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78bbc9e7-eeda-4b2e-a528-a875d27b9749-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-12056b-57749745dc-wtl4l\" (UID: \"78bbc9e7-eeda-4b2e-a528-a875d27b9749\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-57749745dc-wtl4l" Oct 31 13:02:39.115431 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.115410 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhvfw\" (UniqueName: \"kubernetes.io/projected/78bbc9e7-eeda-4b2e-a528-a875d27b9749-kube-api-access-mhvfw\") pod \"payload-pre-processing-e2e-ipp-a-12056b-57749745dc-wtl4l\" (UID: \"78bbc9e7-eeda-4b2e-a528-a875d27b9749\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-57749745dc-wtl4l" Oct 31 13:02:39.121681 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.121658 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-12056b-66744b5f6f-lw4h4"] Oct 31 13:02:39.126120 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.126101 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-12056b-66744b5f6f-lw4h4" Oct 31 13:02:39.241017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.237331 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-12056b-66744b5f6f-lw4h4"] Oct 31 13:02:39.262007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.260032 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhvfw\" (UniqueName: \"kubernetes.io/projected/78bbc9e7-eeda-4b2e-a528-a875d27b9749-kube-api-access-mhvfw\") pod \"payload-pre-processing-e2e-ipp-a-12056b-57749745dc-wtl4l\" (UID: \"78bbc9e7-eeda-4b2e-a528-a875d27b9749\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-57749745dc-wtl4l" Oct 31 13:02:39.262007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.260192 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6r4p\" (UniqueName: \"kubernetes.io/projected/52a3f1a9-ca45-4157-bfd3-232d9773fe31-kube-api-access-n6r4p\") pod \"payload-processing-e2e-ipp-a-12056b-66744b5f6f-lw4h4\" (UID: \"52a3f1a9-ca45-4157-bfd3-232d9773fe31\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-12056b-66744b5f6f-lw4h4" Oct 31 13:02:39.262007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.260221 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78bbc9e7-eeda-4b2e-a528-a875d27b9749-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-12056b-57749745dc-wtl4l\" (UID: \"78bbc9e7-eeda-4b2e-a528-a875d27b9749\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-57749745dc-wtl4l" Oct 31 13:02:39.262007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.260377 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52a3f1a9-ca45-4157-bfd3-232d9773fe31-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-12056b-66744b5f6f-lw4h4\" (UID: \"52a3f1a9-ca45-4157-bfd3-232d9773fe31\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-12056b-66744b5f6f-lw4h4" Oct 31 13:02:39.271005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.263162 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78bbc9e7-eeda-4b2e-a528-a875d27b9749-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-12056b-57749745dc-wtl4l\" (UID: \"78bbc9e7-eeda-4b2e-a528-a875d27b9749\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-57749745dc-wtl4l" Oct 31 13:02:39.340011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.335532 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:39.362752 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.362712 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6r4p\" (UniqueName: \"kubernetes.io/projected/52a3f1a9-ca45-4157-bfd3-232d9773fe31-kube-api-access-n6r4p\") pod \"payload-processing-e2e-ipp-a-12056b-66744b5f6f-lw4h4\" (UID: \"52a3f1a9-ca45-4157-bfd3-232d9773fe31\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-12056b-66744b5f6f-lw4h4" Oct 31 13:02:39.363182 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.363156 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52a3f1a9-ca45-4157-bfd3-232d9773fe31-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-12056b-66744b5f6f-lw4h4\" (UID: \"52a3f1a9-ca45-4157-bfd3-232d9773fe31\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-12056b-66744b5f6f-lw4h4" Oct 31 13:02:39.364416 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.364339 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52a3f1a9-ca45-4157-bfd3-232d9773fe31-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-12056b-66744b5f6f-lw4h4\" (UID: \"52a3f1a9-ca45-4157-bfd3-232d9773fe31\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-12056b-66744b5f6f-lw4h4" Oct 31 13:02:39.384005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.380147 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhvfw\" (UniqueName: \"kubernetes.io/projected/78bbc9e7-eeda-4b2e-a528-a875d27b9749-kube-api-access-mhvfw\") pod \"payload-pre-processing-e2e-ipp-a-12056b-57749745dc-wtl4l\" (UID: \"78bbc9e7-eeda-4b2e-a528-a875d27b9749\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-57749745dc-wtl4l" Oct 31 13:02:39.416427 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.416395 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-57749745dc-wtl4l" Oct 31 13:02:39.433886 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.430869 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6r4p\" (UniqueName: \"kubernetes.io/projected/52a3f1a9-ca45-4157-bfd3-232d9773fe31-kube-api-access-n6r4p\") pod \"payload-processing-e2e-ipp-a-12056b-66744b5f6f-lw4h4\" (UID: \"52a3f1a9-ca45-4157-bfd3-232d9773fe31\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-12056b-66744b5f6f-lw4h4" Oct 31 13:02:39.442332 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.442281 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-12056b-66744b5f6f-lw4h4" Oct 31 13:02:39.553365 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.553242 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:39.647553 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.647518 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="164b7cbb-c80d-4709-b5d0-5bdba8070ce6" path="/var/lib/kubelet/pods/164b7cbb-c80d-4709-b5d0-5bdba8070ce6/volumes" Oct 31 13:02:39.648256 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.648235 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d9c49d7-4979-4d5c-b94e-d477215dd938" path="/var/lib/kubelet/pods/1d9c49d7-4979-4d5c-b94e-d477215dd938/volumes" Oct 31 13:02:39.809327 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.809299 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-57749745dc-wtl4l"] Oct 31 13:02:39.809723 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:02:39.809697 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78bbc9e7_eeda_4b2e_a528_a875d27b9749.slice/crio-85946f4621f20672763dab969633cc9a99696b99b60c2ae3f8a4d08c411cd3ca WatchSource:0}: Error finding container 85946f4621f20672763dab969633cc9a99696b99b60c2ae3f8a4d08c411cd3ca: Status 404 returned error can't find the container with id 85946f4621f20672763dab969633cc9a99696b99b60c2ae3f8a4d08c411cd3ca Oct 31 13:02:39.840814 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:39.840787 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-12056b-66744b5f6f-lw4h4"] Oct 31 13:02:39.841148 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:02:39.841100 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a3f1a9_ca45_4157_bfd3_232d9773fe31.slice/crio-81cbd2b6752401cbc9ef1a39f8f85ab52132e388896315104c6e2123240b57ec WatchSource:0}: Error finding container 81cbd2b6752401cbc9ef1a39f8f85ab52132e388896315104c6e2123240b57ec: Status 404 returned error can't find the container with id 81cbd2b6752401cbc9ef1a39f8f85ab52132e388896315104c6e2123240b57ec Oct 31 13:02:40.028137 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:40.028103 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-12056b-66744b5f6f-lw4h4" event={"ID":"52a3f1a9-ca45-4157-bfd3-232d9773fe31","Type":"ContainerStarted","Data":"d7dad08df8c6ea6e576b8930feb7f0f05e53d73af046ce713193c3a63ab08139"} Oct 31 13:02:40.028316 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:40.028154 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-12056b-66744b5f6f-lw4h4" event={"ID":"52a3f1a9-ca45-4157-bfd3-232d9773fe31","Type":"ContainerStarted","Data":"81cbd2b6752401cbc9ef1a39f8f85ab52132e388896315104c6e2123240b57ec"} Oct 31 13:02:40.028316 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:40.028215 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-12056b-66744b5f6f-lw4h4" Oct 31 13:02:40.029881 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:40.029855 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-57749745dc-wtl4l" event={"ID":"78bbc9e7-eeda-4b2e-a528-a875d27b9749","Type":"ContainerStarted","Data":"dba1f6a2bb8fb4c92039fb0e13e73a7842e2c2eae48691eb89f64336d19fd80a"} Oct 31 13:02:40.030020 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:40.029903 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-57749745dc-wtl4l" event={"ID":"78bbc9e7-eeda-4b2e-a528-a875d27b9749","Type":"ContainerStarted","Data":"85946f4621f20672763dab969633cc9a99696b99b60c2ae3f8a4d08c411cd3ca"} Oct 31 13:02:40.030020 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:40.029931 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-57749745dc-wtl4l" Oct 31 13:02:40.048751 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:40.048710 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-12056b-66744b5f6f-lw4h4" podStartSLOduration=1.048697236 podStartE2EDuration="1.048697236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:02:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:40.046069887 +0000 UTC m=+1806.529192791" watchObservedRunningTime="2026-10-31 13:02:40.048697236 +0000 UTC m=+1806.531820161" Oct 31 13:02:40.069744 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:40.069670 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-57749745dc-wtl4l" podStartSLOduration=1.069657313 podStartE2EDuration="1.069657313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:02:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:40.063784317 +0000 UTC m=+1806.546907217" watchObservedRunningTime="2026-10-31 13:02:40.069657313 +0000 UTC m=+1806.552780215" Oct 31 13:02:40.430950 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:40.430190 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:40.714943 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:40.714915 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-5d98778b55-snvlr" Oct 31 13:02:41.257845 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:41.257809 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:41.902150 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:41.902123 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db792-54bf4947dd-q7k6g" Oct 31 13:02:42.051070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:42.051034 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:42.253554 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:42.253513 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:42.471276 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:42.471242 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:42.864926 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:42.864882 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-64bd66bc8b-9kvnt"] Oct 31 13:02:42.865183 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:42.865156 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-64bd66bc8b-9kvnt" podUID="96cc2f18-6920-405d-9186-a70543773377" containerName="payload-processing" containerID="cri-o://2cefc7c8d9ee6722b70f6924169424f3f6e785c9398026859b1a71a9dba6b203" gracePeriod=30 Oct 31 13:02:42.880002 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:42.879960 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-794f7bb6dc-4dcjj"] Oct 31 13:02:42.880210 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:42.880176 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-794f7bb6dc-4dcjj" podUID="f30ce016-5c64-4a2c-9b4e-a942e53c1e59" containerName="payload-pre-processing" containerID="cri-o://e60e30e2f815b45949a75656d57420a90232341c8c5c24b2d1085d2c24b5271b" gracePeriod=30 Oct 31 13:02:42.939752 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:42.939715 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:43.082021 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:43.080345 6841 generic.go:356] "Generic (PLEG): container finished" podID="f30ce016-5c64-4a2c-9b4e-a942e53c1e59" containerID="e60e30e2f815b45949a75656d57420a90232341c8c5c24b2d1085d2c24b5271b" exitCode=0 Oct 31 13:02:43.082021 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:43.080574 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-794f7bb6dc-4dcjj" event={"ID":"f30ce016-5c64-4a2c-9b4e-a942e53c1e59","Type":"ContainerDied","Data":"e60e30e2f815b45949a75656d57420a90232341c8c5c24b2d1085d2c24b5271b"} Oct 31 13:02:43.133084 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:43.108310 6841 generic.go:356] "Generic (PLEG): container finished" podID="96cc2f18-6920-405d-9186-a70543773377" containerID="2cefc7c8d9ee6722b70f6924169424f3f6e785c9398026859b1a71a9dba6b203" exitCode=0 Oct 31 13:02:43.133084 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:43.108468 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-64bd66bc8b-9kvnt" event={"ID":"96cc2f18-6920-405d-9186-a70543773377","Type":"ContainerDied","Data":"2cefc7c8d9ee6722b70f6924169424f3f6e785c9398026859b1a71a9dba6b203"} Oct 31 13:02:43.201006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:43.186046 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:43.397945 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:43.397916 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-64bd66bc8b-9kvnt" Oct 31 13:02:43.467580 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:43.467545 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-794f7bb6dc-4dcjj" Oct 31 13:02:43.558345 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:43.558309 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f30ce016-5c64-4a2c-9b4e-a942e53c1e59-kube-api-access-95r45\" (UniqueName: \"kubernetes.io/projected/f30ce016-5c64-4a2c-9b4e-a942e53c1e59-kube-api-access-95r45\") pod \"f30ce016-5c64-4a2c-9b4e-a942e53c1e59\" (UID: \"f30ce016-5c64-4a2c-9b4e-a942e53c1e59\") " Oct 31 13:02:43.559206 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:43.559166 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96cc2f18-6920-405d-9186-a70543773377-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96cc2f18-6920-405d-9186-a70543773377-plugins-config-volume\") pod \"96cc2f18-6920-405d-9186-a70543773377\" (UID: \"96cc2f18-6920-405d-9186-a70543773377\") " Oct 31 13:02:43.559523 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:43.559495 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96cc2f18-6920-405d-9186-a70543773377-kube-api-access-zpl2g\" (UniqueName: \"kubernetes.io/projected/96cc2f18-6920-405d-9186-a70543773377-kube-api-access-zpl2g\") pod \"96cc2f18-6920-405d-9186-a70543773377\" (UID: \"96cc2f18-6920-405d-9186-a70543773377\") " Oct 31 13:02:43.559746 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:43.559728 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f30ce016-5c64-4a2c-9b4e-a942e53c1e59-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f30ce016-5c64-4a2c-9b4e-a942e53c1e59-plugins-config-volume\") pod \"f30ce016-5c64-4a2c-9b4e-a942e53c1e59\" (UID: \"f30ce016-5c64-4a2c-9b4e-a942e53c1e59\") " Oct 31 13:02:43.561141 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:43.561115 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f30ce016-5c64-4a2c-9b4e-a942e53c1e59-plugins-config-volume" pod "f30ce016-5c64-4a2c-9b4e-a942e53c1e59" (UID: "f30ce016-5c64-4a2c-9b4e-a942e53c1e59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:02:43.561709 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:43.561686 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96cc2f18-6920-405d-9186-a70543773377-plugins-config-volume" pod "96cc2f18-6920-405d-9186-a70543773377" (UID: "96cc2f18-6920-405d-9186-a70543773377"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:02:43.580761 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:43.577543 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f30ce016-5c64-4a2c-9b4e-a942e53c1e59-kube-api-access-95r45" pod "f30ce016-5c64-4a2c-9b4e-a942e53c1e59" (UID: "f30ce016-5c64-4a2c-9b4e-a942e53c1e59"). InnerVolumeSpecName "kube-api-access-95r45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:43.580761 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:43.580596 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:43.582157 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:43.582122 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96cc2f18-6920-405d-9186-a70543773377-kube-api-access-zpl2g" pod "96cc2f18-6920-405d-9186-a70543773377" (UID: "96cc2f18-6920-405d-9186-a70543773377"). InnerVolumeSpecName "kube-api-access-zpl2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:43.661697 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:43.661671 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96cc2f18-6920-405d-9186-a70543773377-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:43.661697 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:43.661696 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpl2g\" (UniqueName: \"kubernetes.io/projected/96cc2f18-6920-405d-9186-a70543773377-kube-api-access-zpl2g\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:43.661814 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:43.661707 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f30ce016-5c64-4a2c-9b4e-a942e53c1e59-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:43.661814 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:43.661717 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95r45\" (UniqueName: \"kubernetes.io/projected/f30ce016-5c64-4a2c-9b4e-a942e53c1e59-kube-api-access-95r45\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:44.068160 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.068128 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln"] Oct 31 13:02:44.069141 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.069093 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln" podUID="09c52086-c0c4-4429-9d45-4cc93d125c77" containerName="maas-api" containerID="cri-o://185ebe897c682dd925c7365389e784a6cd418b4ba8a9ac0722a6ea54e1a7bd4f" gracePeriod=30 Oct 31 13:02:44.120232 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.120203 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-794f7bb6dc-4dcjj" Oct 31 13:02:44.120381 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.120230 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-794f7bb6dc-4dcjj" event={"ID":"f30ce016-5c64-4a2c-9b4e-a942e53c1e59","Type":"ContainerDied","Data":"56e8b573f7b487558905601b3495328b14f2fa8d7ffb240349238288c54c4948"} Oct 31 13:02:44.120381 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.120295 6841 scope.go:122] "RemoveContainer" containerID="e60e30e2f815b45949a75656d57420a90232341c8c5c24b2d1085d2c24b5271b" Oct 31 13:02:44.121756 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.121738 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-64bd66bc8b-9kvnt" Oct 31 13:02:44.121839 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.121745 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-64bd66bc8b-9kvnt" event={"ID":"96cc2f18-6920-405d-9186-a70543773377","Type":"ContainerDied","Data":"b1ed172254a28b1d3b4dce27c78439c8c27949b884313d3be6918cd460fac0a6"} Oct 31 13:02:44.184875 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.184847 6841 scope.go:122] "RemoveContainer" containerID="2cefc7c8d9ee6722b70f6924169424f3f6e785c9398026859b1a71a9dba6b203" Oct 31 13:02:44.188174 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.188146 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-794f7bb6dc-4dcjj"] Oct 31 13:02:44.191931 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.191902 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-794f7bb6dc-4dcjj"] Oct 31 13:02:44.226862 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.226810 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-64bd66bc8b-9kvnt"] Oct 31 13:02:44.230055 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.230028 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3c8c51-64bd66bc8b-9kvnt"] Oct 31 13:02:44.280008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.279960 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-6589fb84db-9frk4"] Oct 31 13:02:44.280255 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.280227 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-6589fb84db-9frk4" podUID="fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b" containerName="payload-processing" containerID="cri-o://7b409ccf2816ad178319f5aab252d5d76c2409071143a78622bcc9aa40ab7a43" gracePeriod=30 Oct 31 13:02:44.354806 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.354772 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:44.405372 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.405349 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln" Oct 31 13:02:44.553363 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.553339 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-6589fb84db-9frk4" Oct 31 13:02:44.592021 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.591529 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09c52086-c0c4-4429-9d45-4cc93d125c77-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09c52086-c0c4-4429-9d45-4cc93d125c77-openshift-service-ca\") pod \"09c52086-c0c4-4429-9d45-4cc93d125c77\" (UID: \"09c52086-c0c4-4429-9d45-4cc93d125c77\") " Oct 31 13:02:44.592021 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.591603 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09c52086-c0c4-4429-9d45-4cc93d125c77-kube-api-access-9w6dh\" (UniqueName: \"kubernetes.io/projected/09c52086-c0c4-4429-9d45-4cc93d125c77-kube-api-access-9w6dh\") pod \"09c52086-c0c4-4429-9d45-4cc93d125c77\" (UID: \"09c52086-c0c4-4429-9d45-4cc93d125c77\") " Oct 31 13:02:44.592021 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.591761 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09c52086-c0c4-4429-9d45-4cc93d125c77-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09c52086-c0c4-4429-9d45-4cc93d125c77-maas-api-tls\") pod \"09c52086-c0c4-4429-9d45-4cc93d125c77\" (UID: \"09c52086-c0c4-4429-9d45-4cc93d125c77\") " Oct 31 13:02:44.608836 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.608364 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09c52086-c0c4-4429-9d45-4cc93d125c77-openshift-service-ca" pod "09c52086-c0c4-4429-9d45-4cc93d125c77" (UID: "09c52086-c0c4-4429-9d45-4cc93d125c77"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:02:44.621279 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.621049 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09c52086-c0c4-4429-9d45-4cc93d125c77-metrics-certs\" (UniqueName: \"kubernetes.io/secret/09c52086-c0c4-4429-9d45-4cc93d125c77-metrics-certs\") pod \"09c52086-c0c4-4429-9d45-4cc93d125c77\" (UID: \"09c52086-c0c4-4429-9d45-4cc93d125c77\") " Oct 31 13:02:44.623014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.621893 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c52086-c0c4-4429-9d45-4cc93d125c77-kube-api-access-9w6dh" pod "09c52086-c0c4-4429-9d45-4cc93d125c77" (UID: "09c52086-c0c4-4429-9d45-4cc93d125c77"). InnerVolumeSpecName "kube-api-access-9w6dh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:44.627010 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.624095 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09c52086-c0c4-4429-9d45-4cc93d125c77-maas-api-tls" pod "09c52086-c0c4-4429-9d45-4cc93d125c77" (UID: "09c52086-c0c4-4429-9d45-4cc93d125c77"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:02:44.627010 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.626277 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09c52086-c0c4-4429-9d45-4cc93d125c77-metrics-certs" pod "09c52086-c0c4-4429-9d45-4cc93d125c77" (UID: "09c52086-c0c4-4429-9d45-4cc93d125c77"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:02:44.636004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.632675 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:44.645294 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.645177 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09c52086-c0c4-4429-9d45-4cc93d125c77-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:44.760056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.746304 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b-kube-api-access-c5dws\" (UniqueName: \"kubernetes.io/projected/fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b-kube-api-access-c5dws\") pod \"fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b\" (UID: \"fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b\") " Oct 31 13:02:44.760056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.746548 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b-plugins-config-volume\") pod \"fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b\" (UID: \"fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b\") " Oct 31 13:02:44.760056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.747402 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w6dh\" (UniqueName: \"kubernetes.io/projected/09c52086-c0c4-4429-9d45-4cc93d125c77-kube-api-access-9w6dh\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:44.760056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.747425 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09c52086-c0c4-4429-9d45-4cc93d125c77-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:44.760056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.747439 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09c52086-c0c4-4429-9d45-4cc93d125c77-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:44.760056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.747862 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b-plugins-config-volume" pod "fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b" (UID: "fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:02:44.781003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.778283 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b-kube-api-access-c5dws" pod "fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b" (UID: "fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b"). InnerVolumeSpecName "kube-api-access-c5dws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:44.849134 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.849099 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5dws\" (UniqueName: \"kubernetes.io/projected/fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b-kube-api-access-c5dws\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:44.849134 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.849130 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:44.957296 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:44.957259 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:45.131233 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:45.131153 6841 generic.go:356] "Generic (PLEG): container finished" podID="09c52086-c0c4-4429-9d45-4cc93d125c77" containerID="185ebe897c682dd925c7365389e784a6cd418b4ba8a9ac0722a6ea54e1a7bd4f" exitCode=0 Oct 31 13:02:45.131233 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:45.131212 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln" Oct 31 13:02:45.131233 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:45.131213 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln" event={"ID":"09c52086-c0c4-4429-9d45-4cc93d125c77","Type":"ContainerDied","Data":"185ebe897c682dd925c7365389e784a6cd418b4ba8a9ac0722a6ea54e1a7bd4f"} Oct 31 13:02:45.131520 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:45.131299 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln" event={"ID":"09c52086-c0c4-4429-9d45-4cc93d125c77","Type":"ContainerDied","Data":"97fd92bb7318d66664db444e1ea0caf7044ca48a7ca6fd343ff5e093e711ddab"} Oct 31 13:02:45.131520 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:45.131325 6841 scope.go:122] "RemoveContainer" containerID="185ebe897c682dd925c7365389e784a6cd418b4ba8a9ac0722a6ea54e1a7bd4f" Oct 31 13:02:45.132728 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:45.132707 6841 generic.go:356] "Generic (PLEG): container finished" podID="fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b" containerID="7b409ccf2816ad178319f5aab252d5d76c2409071143a78622bcc9aa40ab7a43" exitCode=0 Oct 31 13:02:45.132813 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:45.132757 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-6589fb84db-9frk4" Oct 31 13:02:45.132813 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:45.132792 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-6589fb84db-9frk4" event={"ID":"fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b","Type":"ContainerDied","Data":"7b409ccf2816ad178319f5aab252d5d76c2409071143a78622bcc9aa40ab7a43"} Oct 31 13:02:45.132905 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:45.132837 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-6589fb84db-9frk4" event={"ID":"fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b","Type":"ContainerDied","Data":"ed8002984da64326c4f4b2f19457d1356ca3a1a4ee75d1ba2a5be1e574a6ec7d"} Oct 31 13:02:45.143285 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:45.143264 6841 scope.go:122] "RemoveContainer" containerID="185ebe897c682dd925c7365389e784a6cd418b4ba8a9ac0722a6ea54e1a7bd4f" Oct 31 13:02:45.143581 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:02:45.143559 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"185ebe897c682dd925c7365389e784a6cd418b4ba8a9ac0722a6ea54e1a7bd4f\": container with ID starting with 185ebe897c682dd925c7365389e784a6cd418b4ba8a9ac0722a6ea54e1a7bd4f not found: ID does not exist" containerID="185ebe897c682dd925c7365389e784a6cd418b4ba8a9ac0722a6ea54e1a7bd4f" Oct 31 13:02:45.143644 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:45.143588 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"185ebe897c682dd925c7365389e784a6cd418b4ba8a9ac0722a6ea54e1a7bd4f"} err="failed to get container status \"185ebe897c682dd925c7365389e784a6cd418b4ba8a9ac0722a6ea54e1a7bd4f\": rpc error: code = NotFound desc = could not find container \"185ebe897c682dd925c7365389e784a6cd418b4ba8a9ac0722a6ea54e1a7bd4f\": container with ID starting with 185ebe897c682dd925c7365389e784a6cd418b4ba8a9ac0722a6ea54e1a7bd4f not found: ID does not exist" Oct 31 13:02:45.143644 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:45.143603 6841 scope.go:122] "RemoveContainer" containerID="7b409ccf2816ad178319f5aab252d5d76c2409071143a78622bcc9aa40ab7a43" Oct 31 13:02:45.152687 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:45.152670 6841 scope.go:122] "RemoveContainer" containerID="7b409ccf2816ad178319f5aab252d5d76c2409071143a78622bcc9aa40ab7a43" Oct 31 13:02:45.152949 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:02:45.152929 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b409ccf2816ad178319f5aab252d5d76c2409071143a78622bcc9aa40ab7a43\": container with ID starting with 7b409ccf2816ad178319f5aab252d5d76c2409071143a78622bcc9aa40ab7a43 not found: ID does not exist" containerID="7b409ccf2816ad178319f5aab252d5d76c2409071143a78622bcc9aa40ab7a43" Oct 31 13:02:45.153167 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:45.152956 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b409ccf2816ad178319f5aab252d5d76c2409071143a78622bcc9aa40ab7a43"} err="failed to get container status \"7b409ccf2816ad178319f5aab252d5d76c2409071143a78622bcc9aa40ab7a43\": rpc error: code = NotFound desc = could not find container \"7b409ccf2816ad178319f5aab252d5d76c2409071143a78622bcc9aa40ab7a43\": container with ID starting with 7b409ccf2816ad178319f5aab252d5d76c2409071143a78622bcc9aa40ab7a43 not found: ID does not exist" Oct 31 13:02:45.157080 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:45.157059 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln"] Oct 31 13:02:45.161504 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:45.161479 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-48dcc1-5bc6f9b45d-fchln"] Oct 31 13:02:45.162090 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:45.162068 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:45.174051 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:45.173896 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-6589fb84db-9frk4"] Oct 31 13:02:45.177126 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:45.177102 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-6589fb84db-9frk4"] Oct 31 13:02:45.642447 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:45.642416 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09c52086-c0c4-4429-9d45-4cc93d125c77" path="/var/lib/kubelet/pods/09c52086-c0c4-4429-9d45-4cc93d125c77/volumes" Oct 31 13:02:45.642845 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:45.642831 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96cc2f18-6920-405d-9186-a70543773377" path="/var/lib/kubelet/pods/96cc2f18-6920-405d-9186-a70543773377/volumes" Oct 31 13:02:45.643226 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:45.643213 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f30ce016-5c64-4a2c-9b4e-a942e53c1e59" path="/var/lib/kubelet/pods/f30ce016-5c64-4a2c-9b4e-a942e53c1e59/volumes" Oct 31 13:02:45.643657 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:45.643636 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b" path="/var/lib/kubelet/pods/fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b/volumes" Oct 31 13:02:46.080309 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:46.079571 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:46.664719 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:46.664686 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb"] Oct 31 13:02:46.665392 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:46.665373 6841 cpu_manager.go:414] "containerMap: removing container" podUID="09c52086-c0c4-4429-9d45-4cc93d125c77" containerName="maas-api" Oct 31 13:02:46.665392 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:46.665391 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09c52086-c0c4-4429-9d45-4cc93d125c77" containerName="maas-api" Oct 31 13:02:46.665546 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:46.665414 6841 cpu_manager.go:414] "containerMap: removing container" podUID="f30ce016-5c64-4a2c-9b4e-a942e53c1e59" containerName="payload-pre-processing" Oct 31 13:02:46.665546 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:46.665419 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f30ce016-5c64-4a2c-9b4e-a942e53c1e59" containerName="payload-pre-processing" Oct 31 13:02:46.665546 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:46.665429 6841 cpu_manager.go:414] "containerMap: removing container" podUID="fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b" containerName="payload-processing" Oct 31 13:02:46.665546 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:46.665434 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b" containerName="payload-processing" Oct 31 13:02:46.665546 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:46.665457 6841 cpu_manager.go:414] "containerMap: removing container" podUID="96cc2f18-6920-405d-9186-a70543773377" containerName="payload-processing" Oct 31 13:02:46.665546 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:46.665462 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96cc2f18-6920-405d-9186-a70543773377" containerName="payload-processing" Oct 31 13:02:46.665837 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:46.665564 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="09c52086-c0c4-4429-9d45-4cc93d125c77" containerName="maas-api" Oct 31 13:02:46.665837 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:46.665574 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="f30ce016-5c64-4a2c-9b4e-a942e53c1e59" containerName="payload-pre-processing" Oct 31 13:02:46.665837 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:46.665579 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe562fe6-8f04-4d55-ab6a-4ad4aaa8fb4b" containerName="payload-processing" Oct 31 13:02:46.665837 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:46.665590 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="96cc2f18-6920-405d-9186-a70543773377" containerName="payload-processing" Oct 31 13:02:46.672070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:46.672048 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb" Oct 31 13:02:46.680560 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:46.680531 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-12056b-serving-cert\"" Oct 31 13:02:46.682430 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:46.682404 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb"] Oct 31 13:02:46.774921 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:46.774891 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb\" (UID: \"3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb" Oct 31 13:02:46.775068 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:46.774931 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb\" (UID: \"3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb" Oct 31 13:02:46.775068 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:46.775046 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f-metrics-certs\") pod \"maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb\" (UID: \"3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb" Oct 31 13:02:46.775220 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:46.775205 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stcvf\" (UniqueName: \"kubernetes.io/projected/3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f-kube-api-access-stcvf\") pod \"maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb\" (UID: \"3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb" Oct 31 13:02:46.815611 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:46.815575 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:46.886685 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:46.882100 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stcvf\" (UniqueName: \"kubernetes.io/projected/3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f-kube-api-access-stcvf\") pod \"maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb\" (UID: \"3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb" Oct 31 13:02:46.886685 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:46.882257 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb\" (UID: \"3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb" Oct 31 13:02:46.886685 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:46.882305 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb\" (UID: \"3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb" Oct 31 13:02:46.886685 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:46.882373 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f-metrics-certs\") pod \"maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb\" (UID: \"3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb" Oct 31 13:02:46.886685 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:46.883858 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb\" (UID: \"3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb" Oct 31 13:02:46.901030 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:46.896610 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f-metrics-certs\") pod \"maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb\" (UID: \"3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb" Oct 31 13:02:46.902020 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:46.901221 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb\" (UID: \"3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb" Oct 31 13:02:46.911001 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:46.907759 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stcvf\" (UniqueName: \"kubernetes.io/projected/3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f-kube-api-access-stcvf\") pod \"maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb\" (UID: \"3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb" Oct 31 13:02:46.988686 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:46.988657 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb" Oct 31 13:02:47.013267 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:47.013234 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:47.599587 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:47.591354 6841 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 13:02:47.599587 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:47.596201 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb"] Oct 31 13:02:47.837709 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:47.837584 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:48.003154 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:48.003123 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-6f58649d67-qjsfw" Oct 31 13:02:48.003358 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:48.003342 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4989f9-67d6485cdd-xndwk" Oct 31 13:02:48.357840 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:48.357744 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb" event={"ID":"3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f","Type":"ContainerStarted","Data":"696785d1ecbccadc9acfaea4a4a25c125f9b24e033991b4804301a5c1b83c24d"} Oct 31 13:02:48.358034 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:48.357850 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb" Oct 31 13:02:48.358034 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:48.357880 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb" event={"ID":"3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f","Type":"ContainerStarted","Data":"aace8892791ab1b5063014a4f257400a05bcb03eadf01f49c9cce3ae2b9824ff"} Oct 31 13:02:48.378226 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:48.378179 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb" podStartSLOduration=1.880949972 podStartE2EDuration="2.378163661s" totalImagesPullingTime="497.213689ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:02:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:48.377681083 +0000 UTC m=+1814.860803984" watchObservedRunningTime="2026-10-31 13:02:48.378163661 +0000 UTC m=+1814.861286581" Oct 31 13:02:48.805596 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:48.805558 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:49.544315 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:49.544280 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:50.035487 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:02:50.035442 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:02:50.149173 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:50.149137 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:50.502006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:50.496861 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:50.671586 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:50.671466 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf"] Oct 31 13:02:50.678177 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:50.678148 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:02:50.699307 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:50.699278 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf"] Oct 31 13:02:50.703362 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:50.703339 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-f4cdd9-openshift-default-dockercfg-pdkrw\"" Oct 31 13:02:50.768011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:50.765060 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/916a1b9e-516f-4cde-955a-71a255d469a4-istio-podinfo\") pod \"e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf\" (UID: \"916a1b9e-516f-4cde-955a-71a255d469a4\") " pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:02:50.768011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:50.765142 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/916a1b9e-516f-4cde-955a-71a255d469a4-istiod-ca-cert\") pod \"e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf\" (UID: \"916a1b9e-516f-4cde-955a-71a255d469a4\") " pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:02:50.768011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:50.765235 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjvg7\" (UniqueName: \"kubernetes.io/projected/916a1b9e-516f-4cde-955a-71a255d469a4-kube-api-access-mjvg7\") pod \"e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf\" (UID: \"916a1b9e-516f-4cde-955a-71a255d469a4\") " pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:02:50.768011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:50.765260 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/916a1b9e-516f-4cde-955a-71a255d469a4-istio-token\") pod \"e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf\" (UID: \"916a1b9e-516f-4cde-955a-71a255d469a4\") " pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:02:50.768011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:50.765328 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/916a1b9e-516f-4cde-955a-71a255d469a4-istio-envoy\") pod \"e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf\" (UID: \"916a1b9e-516f-4cde-955a-71a255d469a4\") " pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:02:50.768011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:50.765380 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/916a1b9e-516f-4cde-955a-71a255d469a4-workload-socket\") pod \"e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf\" (UID: \"916a1b9e-516f-4cde-955a-71a255d469a4\") " pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:02:50.768011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:50.765423 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/916a1b9e-516f-4cde-955a-71a255d469a4-istio-data\") pod \"e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf\" (UID: \"916a1b9e-516f-4cde-955a-71a255d469a4\") " pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:02:50.768011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:50.765516 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/916a1b9e-516f-4cde-955a-71a255d469a4-credential-socket\") pod \"e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf\" (UID: \"916a1b9e-516f-4cde-955a-71a255d469a4\") " pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:02:50.768011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:50.765542 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/916a1b9e-516f-4cde-955a-71a255d469a4-workload-certs\") pod \"e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf\" (UID: \"916a1b9e-516f-4cde-955a-71a255d469a4\") " pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:02:50.845430 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:50.842820 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:50.866370 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:50.866337 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/916a1b9e-516f-4cde-955a-71a255d469a4-istio-envoy\") pod \"e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf\" (UID: \"916a1b9e-516f-4cde-955a-71a255d469a4\") " pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:02:50.866541 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:50.866393 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/916a1b9e-516f-4cde-955a-71a255d469a4-workload-socket\") pod \"e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf\" (UID: \"916a1b9e-516f-4cde-955a-71a255d469a4\") " pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:02:50.866541 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:50.866433 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/916a1b9e-516f-4cde-955a-71a255d469a4-istio-data\") pod \"e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf\" (UID: \"916a1b9e-516f-4cde-955a-71a255d469a4\") " pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:02:50.866541 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:50.866508 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/916a1b9e-516f-4cde-955a-71a255d469a4-credential-socket\") pod \"e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf\" (UID: \"916a1b9e-516f-4cde-955a-71a255d469a4\") " pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:02:50.866712 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:50.866540 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/916a1b9e-516f-4cde-955a-71a255d469a4-workload-certs\") pod \"e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf\" (UID: \"916a1b9e-516f-4cde-955a-71a255d469a4\") " pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:02:50.866712 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:50.866634 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/916a1b9e-516f-4cde-955a-71a255d469a4-istio-podinfo\") pod \"e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf\" (UID: \"916a1b9e-516f-4cde-955a-71a255d469a4\") " pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:02:50.866712 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:50.866685 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/916a1b9e-516f-4cde-955a-71a255d469a4-istiod-ca-cert\") pod \"e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf\" (UID: \"916a1b9e-516f-4cde-955a-71a255d469a4\") " pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:02:50.866864 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:50.866786 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjvg7\" (UniqueName: \"kubernetes.io/projected/916a1b9e-516f-4cde-955a-71a255d469a4-kube-api-access-mjvg7\") pod \"e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf\" (UID: \"916a1b9e-516f-4cde-955a-71a255d469a4\") " pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:02:50.866864 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:50.866814 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/916a1b9e-516f-4cde-955a-71a255d469a4-istio-token\") pod \"e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf\" (UID: \"916a1b9e-516f-4cde-955a-71a255d469a4\") " pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:02:50.866965 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:50.866929 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/916a1b9e-516f-4cde-955a-71a255d469a4-workload-socket\") pod \"e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf\" (UID: \"916a1b9e-516f-4cde-955a-71a255d469a4\") " pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:02:50.867038 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:50.866964 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/916a1b9e-516f-4cde-955a-71a255d469a4-istio-data\") pod \"e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf\" (UID: \"916a1b9e-516f-4cde-955a-71a255d469a4\") " pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:02:50.867142 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:50.867120 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/916a1b9e-516f-4cde-955a-71a255d469a4-workload-certs\") pod \"e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf\" (UID: \"916a1b9e-516f-4cde-955a-71a255d469a4\") " pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:02:50.867200 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:50.867148 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/916a1b9e-516f-4cde-955a-71a255d469a4-credential-socket\") pod \"e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf\" (UID: \"916a1b9e-516f-4cde-955a-71a255d469a4\") " pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:02:50.867723 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:50.867697 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/916a1b9e-516f-4cde-955a-71a255d469a4-istiod-ca-cert\") pod \"e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf\" (UID: \"916a1b9e-516f-4cde-955a-71a255d469a4\") " pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:02:50.868946 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:50.868926 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/916a1b9e-516f-4cde-955a-71a255d469a4-istio-envoy\") pod \"e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf\" (UID: \"916a1b9e-516f-4cde-955a-71a255d469a4\") " pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:02:50.869328 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:50.869296 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/916a1b9e-516f-4cde-955a-71a255d469a4-istio-podinfo\") pod \"e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf\" (UID: \"916a1b9e-516f-4cde-955a-71a255d469a4\") " pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:02:50.894758 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:50.894736 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/916a1b9e-516f-4cde-955a-71a255d469a4-istio-token\") pod \"e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf\" (UID: \"916a1b9e-516f-4cde-955a-71a255d469a4\") " pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:02:50.897019 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:50.896969 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjvg7\" (UniqueName: \"kubernetes.io/projected/916a1b9e-516f-4cde-955a-71a255d469a4-kube-api-access-mjvg7\") pod \"e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf\" (UID: \"916a1b9e-516f-4cde-955a-71a255d469a4\") " pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:02:51.007914 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:51.007881 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:02:51.041779 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:51.041753 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-57749745dc-wtl4l" Oct 31 13:02:51.041957 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:51.041826 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-12056b-66744b5f6f-lw4h4" Oct 31 13:02:51.208307 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:51.208267 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:51.329609 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:51.329576 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf"] Oct 31 13:02:51.334859 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:51.334824 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:02:51.334958 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:51.334898 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:02:51.334958 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:51.334924 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:02:51.383372 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:51.383342 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" event={"ID":"916a1b9e-516f-4cde-955a-71a255d469a4","Type":"ContainerStarted","Data":"81c53440a0c1d8f3e78cb835fa3b9495b78249244a87fa3da0129b888ee044e9"} Oct 31 13:02:51.556296 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:51.556255 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:52.400112 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:52.400074 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" event={"ID":"916a1b9e-516f-4cde-955a-71a255d469a4","Type":"ContainerStarted","Data":"054b0b9b69e2378d9f4cb1886539b7e4433b732d211813368c9e24f47fb8874d"} Oct 31 13:02:52.431676 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:52.431621 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" podStartSLOduration=2.431603315 podStartE2EDuration="2.431603315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:02:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:52.427471807 +0000 UTC m=+1818.910594718" watchObservedRunningTime="2026-10-31 13:02:52.431603315 +0000 UTC m=+1818.914726217" Oct 31 13:02:52.624417 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:52.624380 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:53.008711 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:53.008678 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:02:53.014528 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:53.014503 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:02:53.436397 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:53.436306 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:02:53.449330 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:53.449299 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:02:53.579859 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:53.579820 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:53.890081 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:53.890045 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:54.225462 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:54.225394 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:54.370652 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:54.370621 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb" Oct 31 13:02:54.472079 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:54.472021 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:54.884589 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:54.884547 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:55.157742 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:55.157654 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:55.306636 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:55.306605 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj"] Oct 31 13:02:55.306865 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:55.306844 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" podUID="ba9a674c-9aec-46ad-9600-56952a8128c0" containerName="istio-proxy" containerID="cri-o://4e581b0af62adc108f1ad054a96625d9cd7b08771a1b54cc98cec23286c1935e" gracePeriod=30 Oct 31 13:02:55.444069 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:55.442783 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:55.705161 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:55.705071 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:56.019494 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:56.019400 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:56.392339 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:56.392252 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:56.459380 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:56.459343 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-688b5d7bc-vkv7k"] Oct 31 13:02:56.464281 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:56.464263 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-688b5d7bc-vkv7k" Oct 31 13:02:56.475577 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:56.475549 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f4cdd9-dockercfg-qfr4z\"" Oct 31 13:02:56.475744 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:56.475576 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f4cdd9\"" Oct 31 13:02:56.479121 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:56.479099 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-688b5d7bc-vkv7k"] Oct 31 13:02:56.493950 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:56.493916 6841 patch_prober.go:29] interesting pod/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:02:56.494146 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:56.493995 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" podUID="ba9a674c-9aec-46ad-9600-56952a8128c0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:02:56.612014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:56.609878 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkdnn\" (UniqueName: \"kubernetes.io/projected/5a44908b-d8ff-45cd-b505-d19ce41fbba9-kube-api-access-mkdnn\") pod \"payload-pre-processing-e2e-ipp-b-f4cdd9-688b5d7bc-vkv7k\" (UID: \"5a44908b-d8ff-45cd-b505-d19ce41fbba9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-688b5d7bc-vkv7k" Oct 31 13:02:56.612904 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:56.612861 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a44908b-d8ff-45cd-b505-d19ce41fbba9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f4cdd9-688b5d7bc-vkv7k\" (UID: \"5a44908b-d8ff-45cd-b505-d19ce41fbba9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-688b5d7bc-vkv7k" Oct 31 13:02:56.746038 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:56.715446 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a44908b-d8ff-45cd-b505-d19ce41fbba9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f4cdd9-688b5d7bc-vkv7k\" (UID: \"5a44908b-d8ff-45cd-b505-d19ce41fbba9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-688b5d7bc-vkv7k" Oct 31 13:02:56.746038 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:56.715645 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkdnn\" (UniqueName: \"kubernetes.io/projected/5a44908b-d8ff-45cd-b505-d19ce41fbba9-kube-api-access-mkdnn\") pod \"payload-pre-processing-e2e-ipp-b-f4cdd9-688b5d7bc-vkv7k\" (UID: \"5a44908b-d8ff-45cd-b505-d19ce41fbba9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-688b5d7bc-vkv7k" Oct 31 13:02:56.754216 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:56.753394 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a44908b-d8ff-45cd-b505-d19ce41fbba9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f4cdd9-688b5d7bc-vkv7k\" (UID: \"5a44908b-d8ff-45cd-b505-d19ce41fbba9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-688b5d7bc-vkv7k" Oct 31 13:02:56.759282 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:56.759199 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:56.771302 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:56.770020 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkdnn\" (UniqueName: \"kubernetes.io/projected/5a44908b-d8ff-45cd-b505-d19ce41fbba9-kube-api-access-mkdnn\") pod \"payload-pre-processing-e2e-ipp-b-f4cdd9-688b5d7bc-vkv7k\" (UID: \"5a44908b-d8ff-45cd-b505-d19ce41fbba9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-688b5d7bc-vkv7k" Oct 31 13:02:56.771302 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:56.770154 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-94c96c5b7-54j27"] Oct 31 13:02:56.803681 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:56.803640 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-688b5d7bc-vkv7k" Oct 31 13:02:56.805123 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:56.805089 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-94c96c5b7-54j27" Oct 31 13:02:56.806253 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:56.806217 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-94c96c5b7-54j27"] Oct 31 13:02:56.920800 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:56.920756 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c35a13a-2806-4a74-bd75-af12f5cca5ad-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f4cdd9-94c96c5b7-54j27\" (UID: \"6c35a13a-2806-4a74-bd75-af12f5cca5ad\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-94c96c5b7-54j27" Oct 31 13:02:56.920990 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:56.920894 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdsfx\" (UniqueName: \"kubernetes.io/projected/6c35a13a-2806-4a74-bd75-af12f5cca5ad-kube-api-access-xdsfx\") pod \"payload-processing-e2e-ipp-b-f4cdd9-94c96c5b7-54j27\" (UID: \"6c35a13a-2806-4a74-bd75-af12f5cca5ad\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-94c96c5b7-54j27" Oct 31 13:02:57.022888 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:57.022802 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c35a13a-2806-4a74-bd75-af12f5cca5ad-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f4cdd9-94c96c5b7-54j27\" (UID: \"6c35a13a-2806-4a74-bd75-af12f5cca5ad\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-94c96c5b7-54j27" Oct 31 13:02:57.022888 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:57.022859 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdsfx\" (UniqueName: \"kubernetes.io/projected/6c35a13a-2806-4a74-bd75-af12f5cca5ad-kube-api-access-xdsfx\") pod \"payload-processing-e2e-ipp-b-f4cdd9-94c96c5b7-54j27\" (UID: \"6c35a13a-2806-4a74-bd75-af12f5cca5ad\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-94c96c5b7-54j27" Oct 31 13:02:57.023610 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:57.023589 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c35a13a-2806-4a74-bd75-af12f5cca5ad-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f4cdd9-94c96c5b7-54j27\" (UID: \"6c35a13a-2806-4a74-bd75-af12f5cca5ad\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-94c96c5b7-54j27" Oct 31 13:02:57.392138 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:57.392069 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdsfx\" (UniqueName: \"kubernetes.io/projected/6c35a13a-2806-4a74-bd75-af12f5cca5ad-kube-api-access-xdsfx\") pod \"payload-processing-e2e-ipp-b-f4cdd9-94c96c5b7-54j27\" (UID: \"6c35a13a-2806-4a74-bd75-af12f5cca5ad\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-94c96c5b7-54j27" Oct 31 13:02:57.409383 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:57.409357 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-688b5d7bc-vkv7k"] Oct 31 13:02:57.410823 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:02:57.410799 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a44908b_d8ff_45cd_b505_d19ce41fbba9.slice/crio-a7d02f7fcd5282a1490e4f1387ccea944102113a12a64747a8e84a12000d2230 WatchSource:0}: Error finding container a7d02f7fcd5282a1490e4f1387ccea944102113a12a64747a8e84a12000d2230: Status 404 returned error can't find the container with id a7d02f7fcd5282a1490e4f1387ccea944102113a12a64747a8e84a12000d2230 Oct 31 13:02:57.454867 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:57.454840 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-94c96c5b7-54j27" Oct 31 13:02:57.598505 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:57.598465 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:57.612222 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:57.612181 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-688b5d7bc-vkv7k" event={"ID":"5a44908b-d8ff-45cd-b505-d19ce41fbba9","Type":"ContainerStarted","Data":"8aba772457023feb68bbb65d9d4353febc5a56af8c3043124e76fdbeab9e86d3"} Oct 31 13:02:57.612405 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:57.612246 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-688b5d7bc-vkv7k" Oct 31 13:02:57.612405 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:57.612270 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-688b5d7bc-vkv7k" event={"ID":"5a44908b-d8ff-45cd-b505-d19ce41fbba9","Type":"ContainerStarted","Data":"a7d02f7fcd5282a1490e4f1387ccea944102113a12a64747a8e84a12000d2230"} Oct 31 13:02:57.644697 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:57.644592 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-688b5d7bc-vkv7k" podStartSLOduration=1.644574352 podStartE2EDuration="1.644574352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:57.64401582 +0000 UTC m=+1824.127138722" watchObservedRunningTime="2026-10-31 13:02:57.644574352 +0000 UTC m=+1824.127697253" Oct 31 13:02:57.765425 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:57.765052 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-94c96c5b7-54j27"] Oct 31 13:02:58.051404 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.051370 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf"] Oct 31 13:02:58.056188 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.056158 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:02:58.065530 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.065506 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-49bffb-openshift-default-dockercfg-7qsc9\"" Oct 31 13:02:58.075239 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.075212 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf"] Oct 31 13:02:58.147597 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.140232 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaeed36d-9d97-46f0-93f7-955981f85ede-workload-certs\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf\" (UID: \"eaeed36d-9d97-46f0-93f7-955981f85ede\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:02:58.147597 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.140296 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcgrh\" (UniqueName: \"kubernetes.io/projected/eaeed36d-9d97-46f0-93f7-955981f85ede-kube-api-access-vcgrh\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf\" (UID: \"eaeed36d-9d97-46f0-93f7-955981f85ede\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:02:58.147597 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.140332 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaeed36d-9d97-46f0-93f7-955981f85ede-istio-data\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf\" (UID: \"eaeed36d-9d97-46f0-93f7-955981f85ede\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:02:58.147597 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.140507 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaeed36d-9d97-46f0-93f7-955981f85ede-istiod-ca-cert\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf\" (UID: \"eaeed36d-9d97-46f0-93f7-955981f85ede\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:02:58.147597 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.140536 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaeed36d-9d97-46f0-93f7-955981f85ede-workload-socket\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf\" (UID: \"eaeed36d-9d97-46f0-93f7-955981f85ede\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:02:58.147597 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.140577 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaeed36d-9d97-46f0-93f7-955981f85ede-credential-socket\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf\" (UID: \"eaeed36d-9d97-46f0-93f7-955981f85ede\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:02:58.147597 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.140642 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaeed36d-9d97-46f0-93f7-955981f85ede-istio-envoy\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf\" (UID: \"eaeed36d-9d97-46f0-93f7-955981f85ede\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:02:58.147597 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.140670 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaeed36d-9d97-46f0-93f7-955981f85ede-istio-token\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf\" (UID: \"eaeed36d-9d97-46f0-93f7-955981f85ede\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:02:58.147597 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.140742 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaeed36d-9d97-46f0-93f7-955981f85ede-istio-podinfo\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf\" (UID: \"eaeed36d-9d97-46f0-93f7-955981f85ede\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:02:58.243008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.241691 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaeed36d-9d97-46f0-93f7-955981f85ede-istiod-ca-cert\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf\" (UID: \"eaeed36d-9d97-46f0-93f7-955981f85ede\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:02:58.243008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.241746 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaeed36d-9d97-46f0-93f7-955981f85ede-workload-socket\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf\" (UID: \"eaeed36d-9d97-46f0-93f7-955981f85ede\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:02:58.243008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.241783 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaeed36d-9d97-46f0-93f7-955981f85ede-credential-socket\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf\" (UID: \"eaeed36d-9d97-46f0-93f7-955981f85ede\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:02:58.243008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.241844 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaeed36d-9d97-46f0-93f7-955981f85ede-istio-envoy\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf\" (UID: \"eaeed36d-9d97-46f0-93f7-955981f85ede\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:02:58.243008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.241872 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaeed36d-9d97-46f0-93f7-955981f85ede-istio-token\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf\" (UID: \"eaeed36d-9d97-46f0-93f7-955981f85ede\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:02:58.243008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.241940 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaeed36d-9d97-46f0-93f7-955981f85ede-istio-podinfo\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf\" (UID: \"eaeed36d-9d97-46f0-93f7-955981f85ede\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:02:58.243008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.242050 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaeed36d-9d97-46f0-93f7-955981f85ede-workload-certs\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf\" (UID: \"eaeed36d-9d97-46f0-93f7-955981f85ede\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:02:58.243008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.242088 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcgrh\" (UniqueName: \"kubernetes.io/projected/eaeed36d-9d97-46f0-93f7-955981f85ede-kube-api-access-vcgrh\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf\" (UID: \"eaeed36d-9d97-46f0-93f7-955981f85ede\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:02:58.243008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.242118 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaeed36d-9d97-46f0-93f7-955981f85ede-istio-data\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf\" (UID: \"eaeed36d-9d97-46f0-93f7-955981f85ede\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:02:58.243008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.242582 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaeed36d-9d97-46f0-93f7-955981f85ede-istio-data\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf\" (UID: \"eaeed36d-9d97-46f0-93f7-955981f85ede\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:02:58.243008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.242816 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaeed36d-9d97-46f0-93f7-955981f85ede-credential-socket\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf\" (UID: \"eaeed36d-9d97-46f0-93f7-955981f85ede\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:02:58.246179 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.244241 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaeed36d-9d97-46f0-93f7-955981f85ede-istiod-ca-cert\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf\" (UID: \"eaeed36d-9d97-46f0-93f7-955981f85ede\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:02:58.246179 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.244504 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaeed36d-9d97-46f0-93f7-955981f85ede-workload-certs\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf\" (UID: \"eaeed36d-9d97-46f0-93f7-955981f85ede\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:02:58.246179 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.245847 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:58.298007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.283898 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcgrh\" (UniqueName: \"kubernetes.io/projected/eaeed36d-9d97-46f0-93f7-955981f85ede-kube-api-access-vcgrh\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf\" (UID: \"eaeed36d-9d97-46f0-93f7-955981f85ede\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:02:58.298007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.292334 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaeed36d-9d97-46f0-93f7-955981f85ede-istio-envoy\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf\" (UID: \"eaeed36d-9d97-46f0-93f7-955981f85ede\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:02:58.300041 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.299967 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaeed36d-9d97-46f0-93f7-955981f85ede-workload-socket\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf\" (UID: \"eaeed36d-9d97-46f0-93f7-955981f85ede\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:02:58.317146 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.311243 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaeed36d-9d97-46f0-93f7-955981f85ede-istio-token\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf\" (UID: \"eaeed36d-9d97-46f0-93f7-955981f85ede\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:02:58.327010 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.324585 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaeed36d-9d97-46f0-93f7-955981f85ede-istio-podinfo\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf\" (UID: \"eaeed36d-9d97-46f0-93f7-955981f85ede\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:02:58.372009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.371282 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:02:58.475226 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.475198 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:58.620813 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.620770 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-94c96c5b7-54j27" event={"ID":"6c35a13a-2806-4a74-bd75-af12f5cca5ad","Type":"ContainerStarted","Data":"dc3558a9f3674917b2a6fa6c594a65213dec50ec99d721426db92034c782e85c"} Oct 31 13:02:58.621048 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.620828 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-94c96c5b7-54j27" event={"ID":"6c35a13a-2806-4a74-bd75-af12f5cca5ad","Type":"ContainerStarted","Data":"79ef69f9dda2468e734fb336d8264ec76afcf555a6fdd8d966afaeb76cb2f5cb"} Oct 31 13:02:58.621048 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.620937 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-94c96c5b7-54j27" Oct 31 13:02:58.641376 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.641327 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-94c96c5b7-54j27" podStartSLOduration=2.641313111 podStartE2EDuration="2.641313111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:58.639306747 +0000 UTC m=+1825.122429672" watchObservedRunningTime="2026-10-31 13:02:58.641313111 +0000 UTC m=+1825.124436009" Oct 31 13:02:58.700052 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.700020 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:58.713128 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.713098 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf"] Oct 31 13:02:58.717944 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:02:58.717918 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaeed36d_9d97_46f0_93f7_955981f85ede.slice/crio-4c3687ac4242896937337119c76aec8fc4bacad1b73abae0e8deb3df7b626bab WatchSource:0}: Error finding container 4c3687ac4242896937337119c76aec8fc4bacad1b73abae0e8deb3df7b626bab: Status 404 returned error can't find the container with id 4c3687ac4242896937337119c76aec8fc4bacad1b73abae0e8deb3df7b626bab Oct 31 13:02:58.720747 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.720717 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:02:58.720842 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.720782 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:02:58.720842 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.720812 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:02:58.839944 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.838301 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css"] Oct 31 13:02:58.843076 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.843054 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:02:58.855799 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.855771 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07b82d57-b897-421f-b06a-85d23a3d3f3f-istiod-ca-cert\") pod \"e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css\" (UID: \"07b82d57-b897-421f-b06a-85d23a3d3f3f\") " pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:02:58.855957 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.855857 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07b82d57-b897-421f-b06a-85d23a3d3f3f-istio-podinfo\") pod \"e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css\" (UID: \"07b82d57-b897-421f-b06a-85d23a3d3f3f\") " pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:02:58.856057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.855970 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dglzq\" (UniqueName: \"kubernetes.io/projected/07b82d57-b897-421f-b06a-85d23a3d3f3f-kube-api-access-dglzq\") pod \"e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css\" (UID: \"07b82d57-b897-421f-b06a-85d23a3d3f3f\") " pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:02:58.856147 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.856129 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07b82d57-b897-421f-b06a-85d23a3d3f3f-workload-certs\") pod \"e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css\" (UID: \"07b82d57-b897-421f-b06a-85d23a3d3f3f\") " pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:02:58.856215 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.856178 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07b82d57-b897-421f-b06a-85d23a3d3f3f-istio-envoy\") pod \"e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css\" (UID: \"07b82d57-b897-421f-b06a-85d23a3d3f3f\") " pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:02:58.856277 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.856252 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07b82d57-b897-421f-b06a-85d23a3d3f3f-istio-data\") pod \"e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css\" (UID: \"07b82d57-b897-421f-b06a-85d23a3d3f3f\") " pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:02:58.856332 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.856311 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b82d57-b897-421f-b06a-85d23a3d3f3f-workload-socket\") pod \"e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css\" (UID: \"07b82d57-b897-421f-b06a-85d23a3d3f3f\") " pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:02:58.856380 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.856361 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b82d57-b897-421f-b06a-85d23a3d3f3f-credential-socket\") pod \"e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css\" (UID: \"07b82d57-b897-421f-b06a-85d23a3d3f3f\") " pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:02:58.856475 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.856448 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07b82d57-b897-421f-b06a-85d23a3d3f3f-istio-token\") pod \"e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css\" (UID: \"07b82d57-b897-421f-b06a-85d23a3d3f3f\") " pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:02:58.860118 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.860097 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-28e4df-openshift-default-dockercfg-msn5b\"" Oct 31 13:02:58.908686 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:58.908653 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css"] Oct 31 13:02:59.139559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:59.120459 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:59.139559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:59.123282 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07b82d57-b897-421f-b06a-85d23a3d3f3f-istio-envoy\") pod \"e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css\" (UID: \"07b82d57-b897-421f-b06a-85d23a3d3f3f\") " pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:02:59.139559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:59.123327 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07b82d57-b897-421f-b06a-85d23a3d3f3f-istio-data\") pod \"e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css\" (UID: \"07b82d57-b897-421f-b06a-85d23a3d3f3f\") " pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:02:59.139559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:59.123744 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07b82d57-b897-421f-b06a-85d23a3d3f3f-istio-data\") pod \"e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css\" (UID: \"07b82d57-b897-421f-b06a-85d23a3d3f3f\") " pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:02:59.139559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:59.124529 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b82d57-b897-421f-b06a-85d23a3d3f3f-workload-socket\") pod \"e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css\" (UID: \"07b82d57-b897-421f-b06a-85d23a3d3f3f\") " pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:02:59.139559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:59.124607 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b82d57-b897-421f-b06a-85d23a3d3f3f-credential-socket\") pod \"e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css\" (UID: \"07b82d57-b897-421f-b06a-85d23a3d3f3f\") " pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:02:59.139559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:59.124645 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07b82d57-b897-421f-b06a-85d23a3d3f3f-istio-token\") pod \"e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css\" (UID: \"07b82d57-b897-421f-b06a-85d23a3d3f3f\") " pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:02:59.139559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:59.124757 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07b82d57-b897-421f-b06a-85d23a3d3f3f-istiod-ca-cert\") pod \"e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css\" (UID: \"07b82d57-b897-421f-b06a-85d23a3d3f3f\") " pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:02:59.139559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:59.124838 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07b82d57-b897-421f-b06a-85d23a3d3f3f-istio-podinfo\") pod \"e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css\" (UID: \"07b82d57-b897-421f-b06a-85d23a3d3f3f\") " pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:02:59.139559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:59.124936 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dglzq\" (UniqueName: \"kubernetes.io/projected/07b82d57-b897-421f-b06a-85d23a3d3f3f-kube-api-access-dglzq\") pod \"e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css\" (UID: \"07b82d57-b897-421f-b06a-85d23a3d3f3f\") " pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:02:59.139559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:59.125107 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07b82d57-b897-421f-b06a-85d23a3d3f3f-workload-certs\") pod \"e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css\" (UID: \"07b82d57-b897-421f-b06a-85d23a3d3f3f\") " pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:02:59.139559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:59.125458 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07b82d57-b897-421f-b06a-85d23a3d3f3f-workload-certs\") pod \"e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css\" (UID: \"07b82d57-b897-421f-b06a-85d23a3d3f3f\") " pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:02:59.139559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:59.125695 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b82d57-b897-421f-b06a-85d23a3d3f3f-workload-socket\") pod \"e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css\" (UID: \"07b82d57-b897-421f-b06a-85d23a3d3f3f\") " pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:02:59.139559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:59.125927 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b82d57-b897-421f-b06a-85d23a3d3f3f-credential-socket\") pod \"e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css\" (UID: \"07b82d57-b897-421f-b06a-85d23a3d3f3f\") " pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:02:59.139559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:59.126848 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07b82d57-b897-421f-b06a-85d23a3d3f3f-istiod-ca-cert\") pod \"e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css\" (UID: \"07b82d57-b897-421f-b06a-85d23a3d3f3f\") " pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:02:59.295907 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:59.295490 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07b82d57-b897-421f-b06a-85d23a3d3f3f-istio-token\") pod \"e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css\" (UID: \"07b82d57-b897-421f-b06a-85d23a3d3f3f\") " pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:02:59.296518 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:59.296489 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07b82d57-b897-421f-b06a-85d23a3d3f3f-istio-podinfo\") pod \"e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css\" (UID: \"07b82d57-b897-421f-b06a-85d23a3d3f3f\") " pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:02:59.299284 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:59.297304 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07b82d57-b897-421f-b06a-85d23a3d3f3f-istio-envoy\") pod \"e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css\" (UID: \"07b82d57-b897-421f-b06a-85d23a3d3f3f\") " pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:02:59.299948 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:59.299317 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dglzq\" (UniqueName: \"kubernetes.io/projected/07b82d57-b897-421f-b06a-85d23a3d3f3f-kube-api-access-dglzq\") pod \"e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css\" (UID: \"07b82d57-b897-421f-b06a-85d23a3d3f3f\") " pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:02:59.383387 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:59.383343 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:59.459825 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:59.459796 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:02:59.635441 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:59.635405 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" event={"ID":"eaeed36d-9d97-46f0-93f7-955981f85ede","Type":"ContainerStarted","Data":"6eb378320491bf178a5472201b386cb6791e8a9f90659ef35458d9cb320bf8eb"} Oct 31 13:02:59.635594 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:59.635468 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" event={"ID":"eaeed36d-9d97-46f0-93f7-955981f85ede","Type":"ContainerStarted","Data":"4c3687ac4242896937337119c76aec8fc4bacad1b73abae0e8deb3df7b626bab"} Oct 31 13:02:59.667139 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:59.667077 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" podStartSLOduration=2.667063254 podStartE2EDuration="2.667063254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:59.665385644 +0000 UTC m=+1826.148508546" watchObservedRunningTime="2026-10-31 13:02:59.667063254 +0000 UTC m=+1826.150186224" Oct 31 13:02:59.756112 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:59.756082 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-65f69c944c-g26k2"] Oct 31 13:02:59.756455 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:59.756355 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-65f69c944c-g26k2" podUID="fe022e44-83a1-4f5e-b2be-dc6438d8631b" containerName="payload-pre-processing" containerID="cri-o://4e4263adcb3a30a5423ee74a8fa565d6d4bfb835f48e0a5f10c6cd19c076ac4d" gracePeriod=30 Oct 31 13:02:59.759191 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:02:59.759158 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b82d57_b897_421f_b06a_85d23a3d3f3f.slice/crio-9649641d437dea0bc23ffb13a4890e958f53013c01905cfa74f9ebcc5e255dd0 WatchSource:0}: Error finding container 9649641d437dea0bc23ffb13a4890e958f53013c01905cfa74f9ebcc5e255dd0: Status 404 returned error can't find the container with id 9649641d437dea0bc23ffb13a4890e958f53013c01905cfa74f9ebcc5e255dd0 Oct 31 13:02:59.761222 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:59.761194 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css"] Oct 31 13:02:59.762633 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:59.762595 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:02:59.762973 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:59.762678 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:02:59.762973 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:59.762720 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:02:59.814414 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:59.814385 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:02:59.889447 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:02:59.889418 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-65f69c944c-g26k2" podUID="fe022e44-83a1-4f5e-b2be-dc6438d8631b" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.114:9004: connect: connection refused" Oct 31 13:03:00.111198 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.103881 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:00.309300 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.309270 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-65f69c944c-g26k2" Oct 31 13:03:00.372323 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.372232 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:03:00.377555 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.377529 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:03:00.432142 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.430645 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe022e44-83a1-4f5e-b2be-dc6438d8631b-kube-api-access-6kftd\" (UniqueName: \"kubernetes.io/projected/fe022e44-83a1-4f5e-b2be-dc6438d8631b-kube-api-access-6kftd\") pod \"fe022e44-83a1-4f5e-b2be-dc6438d8631b\" (UID: \"fe022e44-83a1-4f5e-b2be-dc6438d8631b\") " Oct 31 13:03:00.432142 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.430964 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe022e44-83a1-4f5e-b2be-dc6438d8631b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe022e44-83a1-4f5e-b2be-dc6438d8631b-plugins-config-volume\") pod \"fe022e44-83a1-4f5e-b2be-dc6438d8631b\" (UID: \"fe022e44-83a1-4f5e-b2be-dc6438d8631b\") " Oct 31 13:03:00.433023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.432719 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe022e44-83a1-4f5e-b2be-dc6438d8631b-plugins-config-volume" pod "fe022e44-83a1-4f5e-b2be-dc6438d8631b" (UID: "fe022e44-83a1-4f5e-b2be-dc6438d8631b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:03:00.445067 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.444958 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe022e44-83a1-4f5e-b2be-dc6438d8631b-kube-api-access-6kftd" pod "fe022e44-83a1-4f5e-b2be-dc6438d8631b" (UID: "fe022e44-83a1-4f5e-b2be-dc6438d8631b"). InnerVolumeSpecName "kube-api-access-6kftd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:03:00.473409 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.473366 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:00.532801 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.532766 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe022e44-83a1-4f5e-b2be-dc6438d8631b-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:00.532801 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.532796 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kftd\" (UniqueName: \"kubernetes.io/projected/fe022e44-83a1-4f5e-b2be-dc6438d8631b-kube-api-access-6kftd\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:00.621566 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.621542 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:03:00.642950 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.642874 6841 generic.go:356] "Generic (PLEG): container finished" podID="ba9a674c-9aec-46ad-9600-56952a8128c0" containerID="4e581b0af62adc108f1ad054a96625d9cd7b08771a1b54cc98cec23286c1935e" exitCode=0 Oct 31 13:03:00.642950 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.642920 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" Oct 31 13:03:00.643203 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.642954 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" event={"ID":"ba9a674c-9aec-46ad-9600-56952a8128c0","Type":"ContainerDied","Data":"4e581b0af62adc108f1ad054a96625d9cd7b08771a1b54cc98cec23286c1935e"} Oct 31 13:03:00.643203 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.643035 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj" event={"ID":"ba9a674c-9aec-46ad-9600-56952a8128c0","Type":"ContainerDied","Data":"96546c1ecac358b406fc03d5309147c5f83157ef7ebbb95b9b70b62e662488e7"} Oct 31 13:03:00.643203 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.643050 6841 scope.go:122] "RemoveContainer" containerID="4e581b0af62adc108f1ad054a96625d9cd7b08771a1b54cc98cec23286c1935e" Oct 31 13:03:00.644844 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.644817 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" event={"ID":"07b82d57-b897-421f-b06a-85d23a3d3f3f","Type":"ContainerStarted","Data":"d8485f67ccfb9b121df99422d918b740b14c9068fbc5336a6807e756fe74fa29"} Oct 31 13:03:00.644967 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.644869 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" event={"ID":"07b82d57-b897-421f-b06a-85d23a3d3f3f","Type":"ContainerStarted","Data":"9649641d437dea0bc23ffb13a4890e958f53013c01905cfa74f9ebcc5e255dd0"} Oct 31 13:03:00.646263 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.646244 6841 generic.go:356] "Generic (PLEG): container finished" podID="fe022e44-83a1-4f5e-b2be-dc6438d8631b" containerID="4e4263adcb3a30a5423ee74a8fa565d6d4bfb835f48e0a5f10c6cd19c076ac4d" exitCode=0 Oct 31 13:03:00.646345 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.646282 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-65f69c944c-g26k2" Oct 31 13:03:00.646345 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.646329 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-65f69c944c-g26k2" event={"ID":"fe022e44-83a1-4f5e-b2be-dc6438d8631b","Type":"ContainerDied","Data":"4e4263adcb3a30a5423ee74a8fa565d6d4bfb835f48e0a5f10c6cd19c076ac4d"} Oct 31 13:03:00.646440 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.646376 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-65f69c944c-g26k2" event={"ID":"fe022e44-83a1-4f5e-b2be-dc6438d8631b","Type":"ContainerDied","Data":"ed2f9b642fbdf56d7177f0d2cd16f39d0c000a42c9899fce70b9d3f3f9a7e902"} Oct 31 13:03:00.646607 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.646580 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:03:00.647573 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.647542 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:03:00.656057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.656040 6841 scope.go:122] "RemoveContainer" containerID="c94e46db8065457675b759a07fa1b7b20fc687e393dd3c812bd1dab9da8264bd" Oct 31 13:03:00.669303 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.669280 6841 scope.go:122] "RemoveContainer" containerID="4e581b0af62adc108f1ad054a96625d9cd7b08771a1b54cc98cec23286c1935e" Oct 31 13:03:00.669656 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:03:00.669636 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e581b0af62adc108f1ad054a96625d9cd7b08771a1b54cc98cec23286c1935e\": container with ID starting with 4e581b0af62adc108f1ad054a96625d9cd7b08771a1b54cc98cec23286c1935e not found: ID does not exist" containerID="4e581b0af62adc108f1ad054a96625d9cd7b08771a1b54cc98cec23286c1935e" Oct 31 13:03:00.669722 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.669667 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e581b0af62adc108f1ad054a96625d9cd7b08771a1b54cc98cec23286c1935e"} err="failed to get container status \"4e581b0af62adc108f1ad054a96625d9cd7b08771a1b54cc98cec23286c1935e\": rpc error: code = NotFound desc = could not find container \"4e581b0af62adc108f1ad054a96625d9cd7b08771a1b54cc98cec23286c1935e\": container with ID starting with 4e581b0af62adc108f1ad054a96625d9cd7b08771a1b54cc98cec23286c1935e not found: ID does not exist" Oct 31 13:03:00.669722 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.669686 6841 scope.go:122] "RemoveContainer" containerID="c94e46db8065457675b759a07fa1b7b20fc687e393dd3c812bd1dab9da8264bd" Oct 31 13:03:00.670011 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:03:00.669977 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c94e46db8065457675b759a07fa1b7b20fc687e393dd3c812bd1dab9da8264bd\": container with ID starting with c94e46db8065457675b759a07fa1b7b20fc687e393dd3c812bd1dab9da8264bd not found: ID does not exist" containerID="c94e46db8065457675b759a07fa1b7b20fc687e393dd3c812bd1dab9da8264bd" Oct 31 13:03:00.670091 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.670018 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c94e46db8065457675b759a07fa1b7b20fc687e393dd3c812bd1dab9da8264bd"} err="failed to get container status \"c94e46db8065457675b759a07fa1b7b20fc687e393dd3c812bd1dab9da8264bd\": rpc error: code = NotFound desc = could not find container \"c94e46db8065457675b759a07fa1b7b20fc687e393dd3c812bd1dab9da8264bd\": container with ID starting with c94e46db8065457675b759a07fa1b7b20fc687e393dd3c812bd1dab9da8264bd not found: ID does not exist" Oct 31 13:03:00.670091 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.670034 6841 scope.go:122] "RemoveContainer" containerID="4e4263adcb3a30a5423ee74a8fa565d6d4bfb835f48e0a5f10c6cd19c076ac4d" Oct 31 13:03:00.681374 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.681347 6841 scope.go:122] "RemoveContainer" containerID="4e4263adcb3a30a5423ee74a8fa565d6d4bfb835f48e0a5f10c6cd19c076ac4d" Oct 31 13:03:00.681654 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:03:00.681633 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e4263adcb3a30a5423ee74a8fa565d6d4bfb835f48e0a5f10c6cd19c076ac4d\": container with ID starting with 4e4263adcb3a30a5423ee74a8fa565d6d4bfb835f48e0a5f10c6cd19c076ac4d not found: ID does not exist" containerID="4e4263adcb3a30a5423ee74a8fa565d6d4bfb835f48e0a5f10c6cd19c076ac4d" Oct 31 13:03:00.681705 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.681661 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e4263adcb3a30a5423ee74a8fa565d6d4bfb835f48e0a5f10c6cd19c076ac4d"} err="failed to get container status \"4e4263adcb3a30a5423ee74a8fa565d6d4bfb835f48e0a5f10c6cd19c076ac4d\": rpc error: code = NotFound desc = could not find container \"4e4263adcb3a30a5423ee74a8fa565d6d4bfb835f48e0a5f10c6cd19c076ac4d\": container with ID starting with 4e4263adcb3a30a5423ee74a8fa565d6d4bfb835f48e0a5f10c6cd19c076ac4d not found: ID does not exist" Oct 31 13:03:00.717910 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.717856 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" podStartSLOduration=2.717842003 podStartE2EDuration="2.717842003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:00.714715778 +0000 UTC m=+1827.197838697" watchObservedRunningTime="2026-10-31 13:03:00.717842003 +0000 UTC m=+1827.200964903" Oct 31 13:03:00.735308 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.735276 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba9a674c-9aec-46ad-9600-56952a8128c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba9a674c-9aec-46ad-9600-56952a8128c0-istiod-ca-cert\") pod \"ba9a674c-9aec-46ad-9600-56952a8128c0\" (UID: \"ba9a674c-9aec-46ad-9600-56952a8128c0\") " Oct 31 13:03:00.735498 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.735357 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba9a674c-9aec-46ad-9600-56952a8128c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba9a674c-9aec-46ad-9600-56952a8128c0-workload-certs\") pod \"ba9a674c-9aec-46ad-9600-56952a8128c0\" (UID: \"ba9a674c-9aec-46ad-9600-56952a8128c0\") " Oct 31 13:03:00.735498 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.735392 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba9a674c-9aec-46ad-9600-56952a8128c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba9a674c-9aec-46ad-9600-56952a8128c0-istio-data\") pod \"ba9a674c-9aec-46ad-9600-56952a8128c0\" (UID: \"ba9a674c-9aec-46ad-9600-56952a8128c0\") " Oct 31 13:03:00.735498 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.735456 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba9a674c-9aec-46ad-9600-56952a8128c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba9a674c-9aec-46ad-9600-56952a8128c0-istio-envoy\") pod \"ba9a674c-9aec-46ad-9600-56952a8128c0\" (UID: \"ba9a674c-9aec-46ad-9600-56952a8128c0\") " Oct 31 13:03:00.735684 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.735582 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba9a674c-9aec-46ad-9600-56952a8128c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba9a674c-9aec-46ad-9600-56952a8128c0-workload-socket\") pod \"ba9a674c-9aec-46ad-9600-56952a8128c0\" (UID: \"ba9a674c-9aec-46ad-9600-56952a8128c0\") " Oct 31 13:03:00.735684 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.735630 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba9a674c-9aec-46ad-9600-56952a8128c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba9a674c-9aec-46ad-9600-56952a8128c0-credential-socket\") pod \"ba9a674c-9aec-46ad-9600-56952a8128c0\" (UID: \"ba9a674c-9aec-46ad-9600-56952a8128c0\") " Oct 31 13:03:00.735803 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.735656 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba9a674c-9aec-46ad-9600-56952a8128c0-workload-certs" pod "ba9a674c-9aec-46ad-9600-56952a8128c0" (UID: "ba9a674c-9aec-46ad-9600-56952a8128c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:03:00.735803 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.735678 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba9a674c-9aec-46ad-9600-56952a8128c0-kube-api-access-pnzhn\" (UniqueName: \"kubernetes.io/projected/ba9a674c-9aec-46ad-9600-56952a8128c0-kube-api-access-pnzhn\") pod \"ba9a674c-9aec-46ad-9600-56952a8128c0\" (UID: \"ba9a674c-9aec-46ad-9600-56952a8128c0\") " Oct 31 13:03:00.735803 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.735741 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba9a674c-9aec-46ad-9600-56952a8128c0-istio-data" pod "ba9a674c-9aec-46ad-9600-56952a8128c0" (UID: "ba9a674c-9aec-46ad-9600-56952a8128c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:03:00.735803 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.735757 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba9a674c-9aec-46ad-9600-56952a8128c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba9a674c-9aec-46ad-9600-56952a8128c0-istio-podinfo\") pod \"ba9a674c-9aec-46ad-9600-56952a8128c0\" (UID: \"ba9a674c-9aec-46ad-9600-56952a8128c0\") " Oct 31 13:03:00.735803 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.735766 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba9a674c-9aec-46ad-9600-56952a8128c0-istiod-ca-cert" pod "ba9a674c-9aec-46ad-9600-56952a8128c0" (UID: "ba9a674c-9aec-46ad-9600-56952a8128c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:03:00.736064 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.735851 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba9a674c-9aec-46ad-9600-56952a8128c0-istio-token\" (UniqueName: \"kubernetes.io/projected/ba9a674c-9aec-46ad-9600-56952a8128c0-istio-token\") pod \"ba9a674c-9aec-46ad-9600-56952a8128c0\" (UID: \"ba9a674c-9aec-46ad-9600-56952a8128c0\") " Oct 31 13:03:00.736169 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.736142 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba9a674c-9aec-46ad-9600-56952a8128c0-workload-socket" pod "ba9a674c-9aec-46ad-9600-56952a8128c0" (UID: "ba9a674c-9aec-46ad-9600-56952a8128c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:03:00.736229 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.736075 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba9a674c-9aec-46ad-9600-56952a8128c0-credential-socket" pod "ba9a674c-9aec-46ad-9600-56952a8128c0" (UID: "ba9a674c-9aec-46ad-9600-56952a8128c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:03:00.736887 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.736858 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba9a674c-9aec-46ad-9600-56952a8128c0-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:00.736887 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.736886 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba9a674c-9aec-46ad-9600-56952a8128c0-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:00.737149 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.736910 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba9a674c-9aec-46ad-9600-56952a8128c0-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:00.737149 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.736936 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba9a674c-9aec-46ad-9600-56952a8128c0-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:00.737149 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.736959 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba9a674c-9aec-46ad-9600-56952a8128c0-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:00.738125 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.738098 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba9a674c-9aec-46ad-9600-56952a8128c0-istio-podinfo" pod "ba9a674c-9aec-46ad-9600-56952a8128c0" (UID: "ba9a674c-9aec-46ad-9600-56952a8128c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:03:00.738224 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.738186 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba9a674c-9aec-46ad-9600-56952a8128c0-istio-envoy" pod "ba9a674c-9aec-46ad-9600-56952a8128c0" (UID: "ba9a674c-9aec-46ad-9600-56952a8128c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:03:00.738891 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.738870 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba9a674c-9aec-46ad-9600-56952a8128c0-kube-api-access-pnzhn" pod "ba9a674c-9aec-46ad-9600-56952a8128c0" (UID: "ba9a674c-9aec-46ad-9600-56952a8128c0"). InnerVolumeSpecName "kube-api-access-pnzhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:03:00.739424 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.739400 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba9a674c-9aec-46ad-9600-56952a8128c0-istio-token" pod "ba9a674c-9aec-46ad-9600-56952a8128c0" (UID: "ba9a674c-9aec-46ad-9600-56952a8128c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:03:00.749068 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.749042 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-65f69c944c-g26k2"] Oct 31 13:03:00.751227 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.751204 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-adfd11-65f69c944c-g26k2"] Oct 31 13:03:00.864004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.849218 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnzhn\" (UniqueName: \"kubernetes.io/projected/ba9a674c-9aec-46ad-9600-56952a8128c0-kube-api-access-pnzhn\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:00.864004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.849266 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba9a674c-9aec-46ad-9600-56952a8128c0-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:00.864004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.849281 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba9a674c-9aec-46ad-9600-56952a8128c0-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:00.864004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.849295 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba9a674c-9aec-46ad-9600-56952a8128c0-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:00.975571 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.975538 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj"] Oct 31 13:03:00.979078 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:00.979044 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-3c8c51-openshift-default-f8bdcf6f9-v5gxj"] Oct 31 13:03:01.363264 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:01.363164 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:01.460509 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:01.460472 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:03:01.466320 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:01.466294 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:03:01.614561 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:01.606941 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:01.645230 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:01.645199 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba9a674c-9aec-46ad-9600-56952a8128c0" path="/var/lib/kubelet/pods/ba9a674c-9aec-46ad-9600-56952a8128c0/volumes" Oct 31 13:03:01.645663 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:01.645649 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe022e44-83a1-4f5e-b2be-dc6438d8631b" path="/var/lib/kubelet/pods/fe022e44-83a1-4f5e-b2be-dc6438d8631b/volumes" Oct 31 13:03:01.656786 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:01.656755 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:03:01.657667 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:01.657649 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:03:01.878066 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:01.874901 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:02.147450 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:02.147370 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:02.434352 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:02.433671 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:02.965524 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:02.965487 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:03.085816 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:03.085783 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-12056b-66744b5f6f-lw4h4"] Oct 31 13:03:03.105200 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:03.105171 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-57749745dc-wtl4l"] Oct 31 13:03:03.224543 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:03.224455 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:03.520134 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:03.520046 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:03.789733 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:03.789621 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:04.203922 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:04.203884 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:04.451160 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:04.451114 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:04.968850 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:04.968818 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-28e4df-9ff5f7b6c-fvgw4"] Oct 31 13:03:04.969534 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:04.969520 6841 cpu_manager.go:414] "containerMap: removing container" podUID="ba9a674c-9aec-46ad-9600-56952a8128c0" containerName="istio-proxy" Oct 31 13:03:04.969589 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:04.969536 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba9a674c-9aec-46ad-9600-56952a8128c0" containerName="istio-proxy" Oct 31 13:03:04.969589 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:04.969572 6841 cpu_manager.go:414] "containerMap: removing container" podUID="fe022e44-83a1-4f5e-b2be-dc6438d8631b" containerName="payload-pre-processing" Oct 31 13:03:04.969589 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:04.969580 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe022e44-83a1-4f5e-b2be-dc6438d8631b" containerName="payload-pre-processing" Oct 31 13:03:04.969686 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:04.969601 6841 cpu_manager.go:414] "containerMap: removing container" podUID="ba9a674c-9aec-46ad-9600-56952a8128c0" containerName="istio-proxy" Oct 31 13:03:04.969686 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:04.969606 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba9a674c-9aec-46ad-9600-56952a8128c0" containerName="istio-proxy" Oct 31 13:03:04.969748 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:04.969697 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba9a674c-9aec-46ad-9600-56952a8128c0" containerName="istio-proxy" Oct 31 13:03:04.969748 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:04.969708 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe022e44-83a1-4f5e-b2be-dc6438d8631b" containerName="payload-pre-processing" Oct 31 13:03:04.974781 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:04.974758 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-28e4df-9ff5f7b6c-fvgw4" Oct 31 13:03:04.984365 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:04.983648 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-28e4df\"" Oct 31 13:03:04.986761 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:04.986733 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-28e4df-9ff5f7b6c-fvgw4"] Oct 31 13:03:04.987202 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:04.987179 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-28e4df-dockercfg-s29d4\"" Oct 31 13:03:05.032710 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:05.032676 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:05.053244 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:05.053207 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db09c0fc-2974-4e9a-8c0e-f9edd5528c49-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-28e4df-9ff5f7b6c-fvgw4\" (UID: \"db09c0fc-2974-4e9a-8c0e-f9edd5528c49\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-28e4df-9ff5f7b6c-fvgw4" Oct 31 13:03:05.053536 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:05.053512 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhrxc\" (UniqueName: \"kubernetes.io/projected/db09c0fc-2974-4e9a-8c0e-f9edd5528c49-kube-api-access-qhrxc\") pod \"payload-pre-processing-e2e-praxis-a-28e4df-9ff5f7b6c-fvgw4\" (UID: \"db09c0fc-2974-4e9a-8c0e-f9edd5528c49\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-28e4df-9ff5f7b6c-fvgw4" Oct 31 13:03:05.054426 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:05.054401 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-28e4df-76bf65bc65-72zf6"] Oct 31 13:03:05.055435 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:05.055413 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba9a674c-9aec-46ad-9600-56952a8128c0" containerName="istio-proxy" Oct 31 13:03:05.059220 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:05.059201 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-28e4df-76bf65bc65-72zf6" Oct 31 13:03:05.077673 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:05.077606 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-28e4df-76bf65bc65-72zf6"] Oct 31 13:03:05.256063 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:05.219106 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhrxc\" (UniqueName: \"kubernetes.io/projected/db09c0fc-2974-4e9a-8c0e-f9edd5528c49-kube-api-access-qhrxc\") pod \"payload-pre-processing-e2e-praxis-a-28e4df-9ff5f7b6c-fvgw4\" (UID: \"db09c0fc-2974-4e9a-8c0e-f9edd5528c49\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-28e4df-9ff5f7b6c-fvgw4" Oct 31 13:03:05.256385 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:05.256358 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db09c0fc-2974-4e9a-8c0e-f9edd5528c49-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-28e4df-9ff5f7b6c-fvgw4\" (UID: \"db09c0fc-2974-4e9a-8c0e-f9edd5528c49\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-28e4df-9ff5f7b6c-fvgw4" Oct 31 13:03:05.256527 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:05.256506 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nd4r\" (UniqueName: \"kubernetes.io/projected/30f31282-bb5f-426c-a167-af1e4bb39696-kube-api-access-2nd4r\") pod \"payload-processing-e2e-praxis-a-28e4df-76bf65bc65-72zf6\" (UID: \"30f31282-bb5f-426c-a167-af1e4bb39696\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-28e4df-76bf65bc65-72zf6" Oct 31 13:03:05.256758 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:05.256741 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30f31282-bb5f-426c-a167-af1e4bb39696-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-28e4df-76bf65bc65-72zf6\" (UID: \"30f31282-bb5f-426c-a167-af1e4bb39696\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-28e4df-76bf65bc65-72zf6" Oct 31 13:03:05.290015 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:05.258173 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db09c0fc-2974-4e9a-8c0e-f9edd5528c49-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-28e4df-9ff5f7b6c-fvgw4\" (UID: \"db09c0fc-2974-4e9a-8c0e-f9edd5528c49\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-28e4df-9ff5f7b6c-fvgw4" Oct 31 13:03:05.290275 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:05.273099 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhrxc\" (UniqueName: \"kubernetes.io/projected/db09c0fc-2974-4e9a-8c0e-f9edd5528c49-kube-api-access-qhrxc\") pod \"payload-pre-processing-e2e-praxis-a-28e4df-9ff5f7b6c-fvgw4\" (UID: \"db09c0fc-2974-4e9a-8c0e-f9edd5528c49\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-28e4df-9ff5f7b6c-fvgw4" Oct 31 13:03:05.290389 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:05.284603 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:05.298309 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:05.298272 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-28e4df-9ff5f7b6c-fvgw4" Oct 31 13:03:05.359250 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:05.359203 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nd4r\" (UniqueName: \"kubernetes.io/projected/30f31282-bb5f-426c-a167-af1e4bb39696-kube-api-access-2nd4r\") pod \"payload-processing-e2e-praxis-a-28e4df-76bf65bc65-72zf6\" (UID: \"30f31282-bb5f-426c-a167-af1e4bb39696\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-28e4df-76bf65bc65-72zf6" Oct 31 13:03:05.359440 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:05.359349 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30f31282-bb5f-426c-a167-af1e4bb39696-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-28e4df-76bf65bc65-72zf6\" (UID: \"30f31282-bb5f-426c-a167-af1e4bb39696\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-28e4df-76bf65bc65-72zf6" Oct 31 13:03:05.361322 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:05.360681 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30f31282-bb5f-426c-a167-af1e4bb39696-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-28e4df-76bf65bc65-72zf6\" (UID: \"30f31282-bb5f-426c-a167-af1e4bb39696\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-28e4df-76bf65bc65-72zf6" Oct 31 13:03:05.387143 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:05.387114 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nd4r\" (UniqueName: \"kubernetes.io/projected/30f31282-bb5f-426c-a167-af1e4bb39696-kube-api-access-2nd4r\") pod \"payload-processing-e2e-praxis-a-28e4df-76bf65bc65-72zf6\" (UID: \"30f31282-bb5f-426c-a167-af1e4bb39696\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-28e4df-76bf65bc65-72zf6" Oct 31 13:03:05.672335 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:05.672301 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-28e4df-9ff5f7b6c-fvgw4"] Oct 31 13:03:05.673176 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:05.673157 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-28e4df-76bf65bc65-72zf6" Oct 31 13:03:05.674667 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:03:05.674633 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb09c0fc_2974_4e9a_8c0e_f9edd5528c49.slice/crio-213f0df112b21dce17e584231f0dff0b9fab9afad5f9126ba05b5fc42713b374 WatchSource:0}: Error finding container 213f0df112b21dce17e584231f0dff0b9fab9afad5f9126ba05b5fc42713b374: Status 404 returned error can't find the container with id 213f0df112b21dce17e584231f0dff0b9fab9afad5f9126ba05b5fc42713b374 Oct 31 13:03:05.718869 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:05.718840 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-28e4df-9ff5f7b6c-fvgw4" event={"ID":"db09c0fc-2974-4e9a-8c0e-f9edd5528c49","Type":"ContainerStarted","Data":"213f0df112b21dce17e584231f0dff0b9fab9afad5f9126ba05b5fc42713b374"} Oct 31 13:03:05.866405 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:05.866376 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:05.969507 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:05.969041 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-28e4df-76bf65bc65-72zf6"] Oct 31 13:03:05.973961 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:03:05.973933 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f31282_bb5f_426c_a167_af1e4bb39696.slice/crio-b305cb4830c27d48d2700a5b4965cd5222c5287ffe873e622dd5dfe559664f80 WatchSource:0}: Error finding container b305cb4830c27d48d2700a5b4965cd5222c5287ffe873e622dd5dfe559664f80: Status 404 returned error can't find the container with id b305cb4830c27d48d2700a5b4965cd5222c5287ffe873e622dd5dfe559664f80 Oct 31 13:03:06.745590 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:06.745541 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-28e4df-9ff5f7b6c-fvgw4" event={"ID":"db09c0fc-2974-4e9a-8c0e-f9edd5528c49","Type":"ContainerStarted","Data":"5d1c7b6d321f4883fc295165bbfbd786f36a3c915d2442bda284a89cb2fd1bd4"} Oct 31 13:03:06.748685 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:06.748661 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-28e4df-9ff5f7b6c-fvgw4" Oct 31 13:03:06.752372 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:06.752342 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-28e4df-76bf65bc65-72zf6" event={"ID":"30f31282-bb5f-426c-a167-af1e4bb39696","Type":"ContainerStarted","Data":"10498e593c5142d90a58c68583185bca54c87509c2b007c2972768016b17e763"} Oct 31 13:03:06.752523 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:06.752395 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-28e4df-76bf65bc65-72zf6" event={"ID":"30f31282-bb5f-426c-a167-af1e4bb39696","Type":"ContainerStarted","Data":"b305cb4830c27d48d2700a5b4965cd5222c5287ffe873e622dd5dfe559664f80"} Oct 31 13:03:06.752523 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:06.752428 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-28e4df-76bf65bc65-72zf6" Oct 31 13:03:06.778308 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:06.778258 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-28e4df-9ff5f7b6c-fvgw4" podStartSLOduration=2.778245141 podStartE2EDuration="2.778245141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:06.776042691 +0000 UTC m=+1833.259165593" watchObservedRunningTime="2026-10-31 13:03:06.778245141 +0000 UTC m=+1833.261368069" Oct 31 13:03:06.798907 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:06.798879 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:06.808046 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:06.808008 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-28e4df-76bf65bc65-72zf6" podStartSLOduration=1.80799575 podStartE2EDuration="1.80799575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:06.805217767 +0000 UTC m=+1833.288340690" watchObservedRunningTime="2026-10-31 13:03:06.80799575 +0000 UTC m=+1833.291118642" Oct 31 13:03:07.534475 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:07.534439 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg"] Oct 31 13:03:07.539605 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:07.539582 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg" Oct 31 13:03:07.548440 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:07.548408 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-f4cdd9-serving-cert\"" Oct 31 13:03:07.554793 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:07.554763 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg"] Oct 31 13:03:07.610748 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:07.610716 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dh2v\" (UniqueName: \"kubernetes.io/projected/9b7bbaef-1b8e-4875-a366-b84f3f19e3f5-kube-api-access-4dh2v\") pod \"maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg\" (UID: \"9b7bbaef-1b8e-4875-a366-b84f3f19e3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg" Oct 31 13:03:07.610886 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:07.610759 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b7bbaef-1b8e-4875-a366-b84f3f19e3f5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg\" (UID: \"9b7bbaef-1b8e-4875-a366-b84f3f19e3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg" Oct 31 13:03:07.610886 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:07.610861 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b7bbaef-1b8e-4875-a366-b84f3f19e3f5-metrics-certs\") pod \"maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg\" (UID: \"9b7bbaef-1b8e-4875-a366-b84f3f19e3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg" Oct 31 13:03:07.610973 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:07.610911 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b7bbaef-1b8e-4875-a366-b84f3f19e3f5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg\" (UID: \"9b7bbaef-1b8e-4875-a366-b84f3f19e3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg" Oct 31 13:03:07.696830 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:07.696791 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:07.714180 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:07.714145 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dh2v\" (UniqueName: \"kubernetes.io/projected/9b7bbaef-1b8e-4875-a366-b84f3f19e3f5-kube-api-access-4dh2v\") pod \"maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg\" (UID: \"9b7bbaef-1b8e-4875-a366-b84f3f19e3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg" Oct 31 13:03:07.714541 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:07.714513 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b7bbaef-1b8e-4875-a366-b84f3f19e3f5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg\" (UID: \"9b7bbaef-1b8e-4875-a366-b84f3f19e3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg" Oct 31 13:03:07.714904 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:07.714880 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b7bbaef-1b8e-4875-a366-b84f3f19e3f5-metrics-certs\") pod \"maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg\" (UID: \"9b7bbaef-1b8e-4875-a366-b84f3f19e3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg" Oct 31 13:03:07.715173 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:07.715153 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b7bbaef-1b8e-4875-a366-b84f3f19e3f5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg\" (UID: \"9b7bbaef-1b8e-4875-a366-b84f3f19e3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg" Oct 31 13:03:07.724114 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:03:07.724091 6841 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4cdd9-serving-cert: secret "maas-api-e2e-ipp-b-f4cdd9-serving-cert" not found Oct 31 13:03:07.724874 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:03:07.724612 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b7bbaef-1b8e-4875-a366-b84f3f19e3f5-maas-api-tls podName:9b7bbaef-1b8e-4875-a366-b84f3f19e3f5 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:08.224587941 +0000 UTC m=+1834.707710821 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9b7bbaef-1b8e-4875-a366-b84f3f19e3f5-maas-api-tls") pod "maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg" (UID: "9b7bbaef-1b8e-4875-a366-b84f3f19e3f5") : secret "maas-api-e2e-ipp-b-f4cdd9-serving-cert" not found Oct 31 13:03:07.736026 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:07.735950 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b7bbaef-1b8e-4875-a366-b84f3f19e3f5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg\" (UID: \"9b7bbaef-1b8e-4875-a366-b84f3f19e3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg" Oct 31 13:03:07.742829 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:07.742770 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b7bbaef-1b8e-4875-a366-b84f3f19e3f5-metrics-certs\") pod \"maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg\" (UID: \"9b7bbaef-1b8e-4875-a366-b84f3f19e3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg" Oct 31 13:03:07.754171 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:07.748121 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dh2v\" (UniqueName: \"kubernetes.io/projected/9b7bbaef-1b8e-4875-a366-b84f3f19e3f5-kube-api-access-4dh2v\") pod \"maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg\" (UID: \"9b7bbaef-1b8e-4875-a366-b84f3f19e3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg" Oct 31 13:03:07.802832 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:07.802753 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb"] Oct 31 13:03:07.803027 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:07.803002 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" podUID="91727fdb-0e14-4b48-b74f-05bfd90c0e17" containerName="istio-proxy" containerID="cri-o://7fcac52694ff14c2a80efaf7248ea045280cf904060fdc34e1ad18454948a7af" gracePeriod=30 Oct 31 13:03:07.809769 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:07.809738 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58"] Oct 31 13:03:07.809993 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:07.809959 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" podUID="189ebc59-a986-4e40-a9b1-25a350c5d397" containerName="istio-proxy" containerID="cri-o://646855c7cb04a308807921ad8a07dff98776c8bd427c21a9b7cbaf234565d69a" gracePeriod=30 Oct 31 13:03:07.923826 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:07.923786 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:08.086196 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.086110 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-12056b-66744b5f6f-lw4h4" podUID="52a3f1a9-ca45-4157-bfd3-232d9773fe31" containerName="payload-processing" containerID="cri-o://d7dad08df8c6ea6e576b8930feb7f0f05e53d73af046ce713193c3a63ab08139" gracePeriod=25 Oct 31 13:03:08.105970 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.105927 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-57749745dc-wtl4l" podUID="78bbc9e7-eeda-4b2e-a528-a875d27b9749" containerName="payload-pre-processing" containerID="cri-o://dba1f6a2bb8fb4c92039fb0e13e73a7842e2c2eae48691eb89f64336d19fd80a" gracePeriod=25 Oct 31 13:03:08.174609 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.174572 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:08.283530 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.282624 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b7bbaef-1b8e-4875-a366-b84f3f19e3f5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg\" (UID: \"9b7bbaef-1b8e-4875-a366-b84f3f19e3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg" Oct 31 13:03:08.322021 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.319618 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b7bbaef-1b8e-4875-a366-b84f3f19e3f5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg\" (UID: \"9b7bbaef-1b8e-4875-a366-b84f3f19e3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg" Oct 31 13:03:08.465033 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.464720 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg" Oct 31 13:03:08.514835 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.514496 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-12056b-66744b5f6f-lw4h4" Oct 31 13:03:08.587539 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.587507 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52a3f1a9-ca45-4157-bfd3-232d9773fe31-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52a3f1a9-ca45-4157-bfd3-232d9773fe31-plugins-config-volume\") pod \"52a3f1a9-ca45-4157-bfd3-232d9773fe31\" (UID: \"52a3f1a9-ca45-4157-bfd3-232d9773fe31\") " Oct 31 13:03:08.588413 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.588381 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52a3f1a9-ca45-4157-bfd3-232d9773fe31-plugins-config-volume" pod "52a3f1a9-ca45-4157-bfd3-232d9773fe31" (UID: "52a3f1a9-ca45-4157-bfd3-232d9773fe31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:03:08.612366 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.612344 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-57749745dc-wtl4l" Oct 31 13:03:08.624054 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.624026 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-688b5d7bc-vkv7k" Oct 31 13:03:08.748002 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.745185 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52a3f1a9-ca45-4157-bfd3-232d9773fe31-kube-api-access-n6r4p\" (UniqueName: \"kubernetes.io/projected/52a3f1a9-ca45-4157-bfd3-232d9773fe31-kube-api-access-n6r4p\") pod \"52a3f1a9-ca45-4157-bfd3-232d9773fe31\" (UID: \"52a3f1a9-ca45-4157-bfd3-232d9773fe31\") " Oct 31 13:03:08.748002 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.747675 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52a3f1a9-ca45-4157-bfd3-232d9773fe31-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:08.755438 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.755264 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52a3f1a9-ca45-4157-bfd3-232d9773fe31-kube-api-access-n6r4p" pod "52a3f1a9-ca45-4157-bfd3-232d9773fe31" (UID: "52a3f1a9-ca45-4157-bfd3-232d9773fe31"). InnerVolumeSpecName "kube-api-access-n6r4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:03:08.759400 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.759368 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:08.777017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.776696 6841 generic.go:356] "Generic (PLEG): container finished" podID="52a3f1a9-ca45-4157-bfd3-232d9773fe31" containerID="d7dad08df8c6ea6e576b8930feb7f0f05e53d73af046ce713193c3a63ab08139" exitCode=0 Oct 31 13:03:08.777017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.776786 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-12056b-66744b5f6f-lw4h4" event={"ID":"52a3f1a9-ca45-4157-bfd3-232d9773fe31","Type":"ContainerDied","Data":"d7dad08df8c6ea6e576b8930feb7f0f05e53d73af046ce713193c3a63ab08139"} Oct 31 13:03:08.777017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.776837 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-12056b-66744b5f6f-lw4h4" event={"ID":"52a3f1a9-ca45-4157-bfd3-232d9773fe31","Type":"ContainerDied","Data":"81cbd2b6752401cbc9ef1a39f8f85ab52132e388896315104c6e2123240b57ec"} Oct 31 13:03:08.777017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.776867 6841 scope.go:122] "RemoveContainer" containerID="d7dad08df8c6ea6e576b8930feb7f0f05e53d73af046ce713193c3a63ab08139" Oct 31 13:03:08.781151 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.777407 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-12056b-66744b5f6f-lw4h4" Oct 31 13:03:08.784164 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.784134 6841 generic.go:356] "Generic (PLEG): container finished" podID="78bbc9e7-eeda-4b2e-a528-a875d27b9749" containerID="dba1f6a2bb8fb4c92039fb0e13e73a7842e2c2eae48691eb89f64336d19fd80a" exitCode=0 Oct 31 13:03:08.784311 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.784178 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-57749745dc-wtl4l" Oct 31 13:03:08.784311 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.784232 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-57749745dc-wtl4l" event={"ID":"78bbc9e7-eeda-4b2e-a528-a875d27b9749","Type":"ContainerDied","Data":"dba1f6a2bb8fb4c92039fb0e13e73a7842e2c2eae48691eb89f64336d19fd80a"} Oct 31 13:03:08.784311 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.784272 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-57749745dc-wtl4l" event={"ID":"78bbc9e7-eeda-4b2e-a528-a875d27b9749","Type":"ContainerDied","Data":"85946f4621f20672763dab969633cc9a99696b99b60c2ae3f8a4d08c411cd3ca"} Oct 31 13:03:08.794674 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.794134 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg"] Oct 31 13:03:08.794674 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:03:08.794611 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b7bbaef_1b8e_4875_a366_b84f3f19e3f5.slice/crio-fdbc6732ae019aea4d350fdac61849c4c0e05e17729e50fdab5de16a38de69e7 WatchSource:0}: Error finding container fdbc6732ae019aea4d350fdac61849c4c0e05e17729e50fdab5de16a38de69e7: Status 404 returned error can't find the container with id fdbc6732ae019aea4d350fdac61849c4c0e05e17729e50fdab5de16a38de69e7 Oct 31 13:03:08.830087 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.829890 6841 scope.go:122] "RemoveContainer" containerID="d7dad08df8c6ea6e576b8930feb7f0f05e53d73af046ce713193c3a63ab08139" Oct 31 13:03:08.830336 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:03:08.830310 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7dad08df8c6ea6e576b8930feb7f0f05e53d73af046ce713193c3a63ab08139\": container with ID starting with d7dad08df8c6ea6e576b8930feb7f0f05e53d73af046ce713193c3a63ab08139 not found: ID does not exist" containerID="d7dad08df8c6ea6e576b8930feb7f0f05e53d73af046ce713193c3a63ab08139" Oct 31 13:03:08.830416 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.830346 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7dad08df8c6ea6e576b8930feb7f0f05e53d73af046ce713193c3a63ab08139"} err="failed to get container status \"d7dad08df8c6ea6e576b8930feb7f0f05e53d73af046ce713193c3a63ab08139\": rpc error: code = NotFound desc = could not find container \"d7dad08df8c6ea6e576b8930feb7f0f05e53d73af046ce713193c3a63ab08139\": container with ID starting with d7dad08df8c6ea6e576b8930feb7f0f05e53d73af046ce713193c3a63ab08139 not found: ID does not exist" Oct 31 13:03:08.830416 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.830366 6841 scope.go:122] "RemoveContainer" containerID="dba1f6a2bb8fb4c92039fb0e13e73a7842e2c2eae48691eb89f64336d19fd80a" Oct 31 13:03:08.841540 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.841515 6841 scope.go:122] "RemoveContainer" containerID="dba1f6a2bb8fb4c92039fb0e13e73a7842e2c2eae48691eb89f64336d19fd80a" Oct 31 13:03:08.841841 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:03:08.841821 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dba1f6a2bb8fb4c92039fb0e13e73a7842e2c2eae48691eb89f64336d19fd80a\": container with ID starting with dba1f6a2bb8fb4c92039fb0e13e73a7842e2c2eae48691eb89f64336d19fd80a not found: ID does not exist" containerID="dba1f6a2bb8fb4c92039fb0e13e73a7842e2c2eae48691eb89f64336d19fd80a" Oct 31 13:03:08.841894 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.841850 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dba1f6a2bb8fb4c92039fb0e13e73a7842e2c2eae48691eb89f64336d19fd80a"} err="failed to get container status \"dba1f6a2bb8fb4c92039fb0e13e73a7842e2c2eae48691eb89f64336d19fd80a\": rpc error: code = NotFound desc = could not find container \"dba1f6a2bb8fb4c92039fb0e13e73a7842e2c2eae48691eb89f64336d19fd80a\": container with ID starting with dba1f6a2bb8fb4c92039fb0e13e73a7842e2c2eae48691eb89f64336d19fd80a not found: ID does not exist" Oct 31 13:03:08.848425 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.848401 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78bbc9e7-eeda-4b2e-a528-a875d27b9749-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78bbc9e7-eeda-4b2e-a528-a875d27b9749-plugins-config-volume\") pod \"78bbc9e7-eeda-4b2e-a528-a875d27b9749\" (UID: \"78bbc9e7-eeda-4b2e-a528-a875d27b9749\") " Oct 31 13:03:08.848525 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.848473 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78bbc9e7-eeda-4b2e-a528-a875d27b9749-kube-api-access-mhvfw\" (UniqueName: \"kubernetes.io/projected/78bbc9e7-eeda-4b2e-a528-a875d27b9749-kube-api-access-mhvfw\") pod \"78bbc9e7-eeda-4b2e-a528-a875d27b9749\" (UID: \"78bbc9e7-eeda-4b2e-a528-a875d27b9749\") " Oct 31 13:03:08.848757 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.848738 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78bbc9e7-eeda-4b2e-a528-a875d27b9749-plugins-config-volume" pod "78bbc9e7-eeda-4b2e-a528-a875d27b9749" (UID: "78bbc9e7-eeda-4b2e-a528-a875d27b9749"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:03:08.849159 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.849136 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78bbc9e7-eeda-4b2e-a528-a875d27b9749-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:08.849258 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.849171 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6r4p\" (UniqueName: \"kubernetes.io/projected/52a3f1a9-ca45-4157-bfd3-232d9773fe31-kube-api-access-n6r4p\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:08.850766 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.850746 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78bbc9e7-eeda-4b2e-a528-a875d27b9749-kube-api-access-mhvfw" pod "78bbc9e7-eeda-4b2e-a528-a875d27b9749" (UID: "78bbc9e7-eeda-4b2e-a528-a875d27b9749"). InnerVolumeSpecName "kube-api-access-mhvfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:03:08.865168 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.865141 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-12056b-66744b5f6f-lw4h4"] Oct 31 13:03:08.869124 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.869103 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-12056b-66744b5f6f-lw4h4"] Oct 31 13:03:08.950172 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:08.950145 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhvfw\" (UniqueName: \"kubernetes.io/projected/78bbc9e7-eeda-4b2e-a528-a875d27b9749-kube-api-access-mhvfw\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:09.110019 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:09.109968 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-57749745dc-wtl4l"] Oct 31 13:03:09.115763 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:09.115736 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-57749745dc-wtl4l"] Oct 31 13:03:09.676144 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:09.676111 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52a3f1a9-ca45-4157-bfd3-232d9773fe31" path="/var/lib/kubelet/pods/52a3f1a9-ca45-4157-bfd3-232d9773fe31/volumes" Oct 31 13:03:09.676563 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:09.676549 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78bbc9e7-eeda-4b2e-a528-a875d27b9749" path="/var/lib/kubelet/pods/78bbc9e7-eeda-4b2e-a528-a875d27b9749/volumes" Oct 31 13:03:09.676911 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:09.676889 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-94c96c5b7-54j27" Oct 31 13:03:09.795930 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:09.795885 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg" event={"ID":"9b7bbaef-1b8e-4875-a366-b84f3f19e3f5","Type":"ContainerStarted","Data":"ac8e35f4b0e863b0c2983e44483b85b32f814ea4f8c81e2c5108064892d63eab"} Oct 31 13:03:09.796353 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:09.795945 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg" event={"ID":"9b7bbaef-1b8e-4875-a366-b84f3f19e3f5","Type":"ContainerStarted","Data":"fdbc6732ae019aea4d350fdac61849c4c0e05e17729e50fdab5de16a38de69e7"} Oct 31 13:03:09.796353 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:09.796037 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg" Oct 31 13:03:09.872431 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:09.872380 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg" podStartSLOduration=2.420347248 podStartE2EDuration="2.872365748s" totalImagesPullingTime="452.0185ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:09.869721572 +0000 UTC m=+1836.352844472" watchObservedRunningTime="2026-10-31 13:03:09.872365748 +0000 UTC m=+1836.355488651" Oct 31 13:03:10.297005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:10.283336 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:11.426192 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:11.425771 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:11.667423 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:11.667384 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:12.067700 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:12.067659 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:12.117354 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:12.117327 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-6bbff8b447-7827t"] Oct 31 13:03:12.118040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:12.118024 6841 cpu_manager.go:414] "containerMap: removing container" podUID="78bbc9e7-eeda-4b2e-a528-a875d27b9749" containerName="payload-pre-processing" Oct 31 13:03:12.118084 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:12.118042 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78bbc9e7-eeda-4b2e-a528-a875d27b9749" containerName="payload-pre-processing" Oct 31 13:03:12.118116 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:12.118084 6841 cpu_manager.go:414] "containerMap: removing container" podUID="52a3f1a9-ca45-4157-bfd3-232d9773fe31" containerName="payload-processing" Oct 31 13:03:12.118116 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:12.118091 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52a3f1a9-ca45-4157-bfd3-232d9773fe31" containerName="payload-processing" Oct 31 13:03:12.118200 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:12.118191 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="52a3f1a9-ca45-4157-bfd3-232d9773fe31" containerName="payload-processing" Oct 31 13:03:12.118233 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:12.118206 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="78bbc9e7-eeda-4b2e-a528-a875d27b9749" containerName="payload-pre-processing" Oct 31 13:03:12.123300 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:12.123280 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-6bbff8b447-7827t" Oct 31 13:03:12.134072 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:12.134052 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-12056b\"" Oct 31 13:03:12.138125 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:12.138102 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-6bbff8b447-7827t"] Oct 31 13:03:12.209383 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:12.209353 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hlrj\" (UniqueName: \"kubernetes.io/projected/faafb6b6-9a5a-4a60-a327-1629d2c49868-kube-api-access-5hlrj\") pod \"payload-pre-processing-e2e-ipp-a-12056b-6bbff8b447-7827t\" (UID: \"faafb6b6-9a5a-4a60-a327-1629d2c49868\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-6bbff8b447-7827t" Oct 31 13:03:12.209569 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:12.209549 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faafb6b6-9a5a-4a60-a327-1629d2c49868-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-12056b-6bbff8b447-7827t\" (UID: \"faafb6b6-9a5a-4a60-a327-1629d2c49868\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-6bbff8b447-7827t" Oct 31 13:03:12.312042 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:12.311024 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faafb6b6-9a5a-4a60-a327-1629d2c49868-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-12056b-6bbff8b447-7827t\" (UID: \"faafb6b6-9a5a-4a60-a327-1629d2c49868\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-6bbff8b447-7827t" Oct 31 13:03:12.312042 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:12.311324 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hlrj\" (UniqueName: \"kubernetes.io/projected/faafb6b6-9a5a-4a60-a327-1629d2c49868-kube-api-access-5hlrj\") pod \"payload-pre-processing-e2e-ipp-a-12056b-6bbff8b447-7827t\" (UID: \"faafb6b6-9a5a-4a60-a327-1629d2c49868\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-6bbff8b447-7827t" Oct 31 13:03:12.356053 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:12.337874 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:12.356053 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:12.340619 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faafb6b6-9a5a-4a60-a327-1629d2c49868-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-12056b-6bbff8b447-7827t\" (UID: \"faafb6b6-9a5a-4a60-a327-1629d2c49868\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-6bbff8b447-7827t" Oct 31 13:03:12.356053 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:12.346050 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-12056b-54cd8754db-r8p4d"] Oct 31 13:03:12.567042 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:12.565479 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-12056b-54cd8754db-r8p4d"] Oct 31 13:03:12.567042 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:12.565625 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-12056b-54cd8754db-r8p4d" Oct 31 13:03:12.584254 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:12.568543 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hlrj\" (UniqueName: \"kubernetes.io/projected/faafb6b6-9a5a-4a60-a327-1629d2c49868-kube-api-access-5hlrj\") pod \"payload-pre-processing-e2e-ipp-a-12056b-6bbff8b447-7827t\" (UID: \"faafb6b6-9a5a-4a60-a327-1629d2c49868\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-6bbff8b447-7827t" Oct 31 13:03:12.585493 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:12.585457 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwqvs\" (UniqueName: \"kubernetes.io/projected/6a4859e8-5b79-4f2a-afe6-0b26a64cce08-kube-api-access-lwqvs\") pod \"payload-processing-e2e-ipp-a-12056b-54cd8754db-r8p4d\" (UID: \"6a4859e8-5b79-4f2a-afe6-0b26a64cce08\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-12056b-54cd8754db-r8p4d" Oct 31 13:03:12.585717 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:12.585676 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a4859e8-5b79-4f2a-afe6-0b26a64cce08-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-12056b-54cd8754db-r8p4d\" (UID: \"6a4859e8-5b79-4f2a-afe6-0b26a64cce08\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-12056b-54cd8754db-r8p4d" Oct 31 13:03:12.589460 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:12.589439 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-12056b-dockercfg-rvx7h\"" Oct 31 13:03:12.696728 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:12.689092 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a4859e8-5b79-4f2a-afe6-0b26a64cce08-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-12056b-54cd8754db-r8p4d\" (UID: \"6a4859e8-5b79-4f2a-afe6-0b26a64cce08\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-12056b-54cd8754db-r8p4d" Oct 31 13:03:12.696728 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:12.689244 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwqvs\" (UniqueName: \"kubernetes.io/projected/6a4859e8-5b79-4f2a-afe6-0b26a64cce08-kube-api-access-lwqvs\") pod \"payload-processing-e2e-ipp-a-12056b-54cd8754db-r8p4d\" (UID: \"6a4859e8-5b79-4f2a-afe6-0b26a64cce08\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-12056b-54cd8754db-r8p4d" Oct 31 13:03:12.698090 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:12.697707 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a4859e8-5b79-4f2a-afe6-0b26a64cce08-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-12056b-54cd8754db-r8p4d\" (UID: \"6a4859e8-5b79-4f2a-afe6-0b26a64cce08\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-12056b-54cd8754db-r8p4d" Oct 31 13:03:12.740503 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:12.740220 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-6bbff8b447-7827t" Oct 31 13:03:12.741786 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:12.741721 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwqvs\" (UniqueName: \"kubernetes.io/projected/6a4859e8-5b79-4f2a-afe6-0b26a64cce08-kube-api-access-lwqvs\") pod \"payload-processing-e2e-ipp-a-12056b-54cd8754db-r8p4d\" (UID: \"6a4859e8-5b79-4f2a-afe6-0b26a64cce08\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-12056b-54cd8754db-r8p4d" Oct 31 13:03:12.767001 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:12.766946 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:12.904569 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:12.904546 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-12056b-54cd8754db-r8p4d" Oct 31 13:03:13.105218 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.105186 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-6bbff8b447-7827t"] Oct 31 13:03:13.108857 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:03:13.108802 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaafb6b6_9a5a_4a60_a327_1629d2c49868.slice/crio-dd0b58eb4ff6d4c5702fefc1c0458925464842f1ce97a5eca668f26c8b80814f WatchSource:0}: Error finding container dd0b58eb4ff6d4c5702fefc1c0458925464842f1ce97a5eca668f26c8b80814f: Status 404 returned error can't find the container with id dd0b58eb4ff6d4c5702fefc1c0458925464842f1ce97a5eca668f26c8b80814f Oct 31 13:03:13.209866 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.209844 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" Oct 31 13:03:13.212317 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.212295 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" Oct 31 13:03:13.278106 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.278075 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-12056b-54cd8754db-r8p4d"] Oct 31 13:03:13.300720 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.300689 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91727fdb-0e14-4b48-b74f-05bfd90c0e17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91727fdb-0e14-4b48-b74f-05bfd90c0e17-istio-envoy\") pod \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\" (UID: \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\") " Oct 31 13:03:13.300890 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.300790 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/189ebc59-a986-4e40-a9b1-25a350c5d397-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/189ebc59-a986-4e40-a9b1-25a350c5d397-istio-podinfo\") pod \"189ebc59-a986-4e40-a9b1-25a350c5d397\" (UID: \"189ebc59-a986-4e40-a9b1-25a350c5d397\") " Oct 31 13:03:13.300890 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.300852 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/189ebc59-a986-4e40-a9b1-25a350c5d397-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/189ebc59-a986-4e40-a9b1-25a350c5d397-istiod-ca-cert\") pod \"189ebc59-a986-4e40-a9b1-25a350c5d397\" (UID: \"189ebc59-a986-4e40-a9b1-25a350c5d397\") " Oct 31 13:03:13.301054 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.300915 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91727fdb-0e14-4b48-b74f-05bfd90c0e17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91727fdb-0e14-4b48-b74f-05bfd90c0e17-istio-data\") pod \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\" (UID: \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\") " Oct 31 13:03:13.301054 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.300956 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91727fdb-0e14-4b48-b74f-05bfd90c0e17-kube-api-access-9hnmd\" (UniqueName: \"kubernetes.io/projected/91727fdb-0e14-4b48-b74f-05bfd90c0e17-kube-api-access-9hnmd\") pod \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\" (UID: \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\") " Oct 31 13:03:13.301054 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.301024 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/189ebc59-a986-4e40-a9b1-25a350c5d397-istio-token\" (UniqueName: \"kubernetes.io/projected/189ebc59-a986-4e40-a9b1-25a350c5d397-istio-token\") pod \"189ebc59-a986-4e40-a9b1-25a350c5d397\" (UID: \"189ebc59-a986-4e40-a9b1-25a350c5d397\") " Oct 31 13:03:13.301217 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.301066 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/189ebc59-a986-4e40-a9b1-25a350c5d397-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/189ebc59-a986-4e40-a9b1-25a350c5d397-credential-socket\") pod \"189ebc59-a986-4e40-a9b1-25a350c5d397\" (UID: \"189ebc59-a986-4e40-a9b1-25a350c5d397\") " Oct 31 13:03:13.301217 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.301115 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91727fdb-0e14-4b48-b74f-05bfd90c0e17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91727fdb-0e14-4b48-b74f-05bfd90c0e17-workload-certs\") pod \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\" (UID: \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\") " Oct 31 13:03:13.301217 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.301163 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91727fdb-0e14-4b48-b74f-05bfd90c0e17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91727fdb-0e14-4b48-b74f-05bfd90c0e17-credential-socket\") pod \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\" (UID: \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\") " Oct 31 13:03:13.301412 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.301238 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91727fdb-0e14-4b48-b74f-05bfd90c0e17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91727fdb-0e14-4b48-b74f-05bfd90c0e17-workload-socket\") pod \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\" (UID: \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\") " Oct 31 13:03:13.301412 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.301300 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/189ebc59-a986-4e40-a9b1-25a350c5d397-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/189ebc59-a986-4e40-a9b1-25a350c5d397-istio-envoy\") pod \"189ebc59-a986-4e40-a9b1-25a350c5d397\" (UID: \"189ebc59-a986-4e40-a9b1-25a350c5d397\") " Oct 31 13:03:13.301412 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.301380 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/189ebc59-a986-4e40-a9b1-25a350c5d397-istiod-ca-cert" pod "189ebc59-a986-4e40-a9b1-25a350c5d397" (UID: "189ebc59-a986-4e40-a9b1-25a350c5d397"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:03:13.301565 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.301424 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/189ebc59-a986-4e40-a9b1-25a350c5d397-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/189ebc59-a986-4e40-a9b1-25a350c5d397-istio-data\") pod \"189ebc59-a986-4e40-a9b1-25a350c5d397\" (UID: \"189ebc59-a986-4e40-a9b1-25a350c5d397\") " Oct 31 13:03:13.301565 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.301475 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91727fdb-0e14-4b48-b74f-05bfd90c0e17-istio-token\" (UniqueName: \"kubernetes.io/projected/91727fdb-0e14-4b48-b74f-05bfd90c0e17-istio-token\") pod \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\" (UID: \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\") " Oct 31 13:03:13.301565 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.301500 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/189ebc59-a986-4e40-a9b1-25a350c5d397-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/189ebc59-a986-4e40-a9b1-25a350c5d397-workload-socket\") pod \"189ebc59-a986-4e40-a9b1-25a350c5d397\" (UID: \"189ebc59-a986-4e40-a9b1-25a350c5d397\") " Oct 31 13:03:13.301565 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.301523 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/189ebc59-a986-4e40-a9b1-25a350c5d397-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/189ebc59-a986-4e40-a9b1-25a350c5d397-workload-certs\") pod \"189ebc59-a986-4e40-a9b1-25a350c5d397\" (UID: \"189ebc59-a986-4e40-a9b1-25a350c5d397\") " Oct 31 13:03:13.301565 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.301545 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/189ebc59-a986-4e40-a9b1-25a350c5d397-kube-api-access-2jp4x\" (UniqueName: \"kubernetes.io/projected/189ebc59-a986-4e40-a9b1-25a350c5d397-kube-api-access-2jp4x\") pod \"189ebc59-a986-4e40-a9b1-25a350c5d397\" (UID: \"189ebc59-a986-4e40-a9b1-25a350c5d397\") " Oct 31 13:03:13.301824 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.301590 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91727fdb-0e14-4b48-b74f-05bfd90c0e17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91727fdb-0e14-4b48-b74f-05bfd90c0e17-istio-podinfo\") pod \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\" (UID: \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\") " Oct 31 13:03:13.301824 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.301616 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91727fdb-0e14-4b48-b74f-05bfd90c0e17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91727fdb-0e14-4b48-b74f-05bfd90c0e17-istiod-ca-cert\") pod \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\" (UID: \"91727fdb-0e14-4b48-b74f-05bfd90c0e17\") " Oct 31 13:03:13.302932 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.302250 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91727fdb-0e14-4b48-b74f-05bfd90c0e17-workload-socket" pod "91727fdb-0e14-4b48-b74f-05bfd90c0e17" (UID: "91727fdb-0e14-4b48-b74f-05bfd90c0e17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:03:13.302932 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.302463 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189ebc59-a986-4e40-a9b1-25a350c5d397-credential-socket" pod "189ebc59-a986-4e40-a9b1-25a350c5d397" (UID: "189ebc59-a986-4e40-a9b1-25a350c5d397"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:03:13.302932 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.302625 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189ebc59-a986-4e40-a9b1-25a350c5d397-workload-socket" pod "189ebc59-a986-4e40-a9b1-25a350c5d397" (UID: "189ebc59-a986-4e40-a9b1-25a350c5d397"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:03:13.302932 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.302637 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91727fdb-0e14-4b48-b74f-05bfd90c0e17-workload-certs" pod "91727fdb-0e14-4b48-b74f-05bfd90c0e17" (UID: "91727fdb-0e14-4b48-b74f-05bfd90c0e17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:03:13.302932 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.302732 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189ebc59-a986-4e40-a9b1-25a350c5d397-istio-data" pod "189ebc59-a986-4e40-a9b1-25a350c5d397" (UID: "189ebc59-a986-4e40-a9b1-25a350c5d397"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:03:13.303281 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.303029 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91727fdb-0e14-4b48-b74f-05bfd90c0e17-istio-data" pod "91727fdb-0e14-4b48-b74f-05bfd90c0e17" (UID: "91727fdb-0e14-4b48-b74f-05bfd90c0e17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:03:13.303281 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.303226 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189ebc59-a986-4e40-a9b1-25a350c5d397-workload-certs" pod "189ebc59-a986-4e40-a9b1-25a350c5d397" (UID: "189ebc59-a986-4e40-a9b1-25a350c5d397"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:03:13.303840 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.303648 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91727fdb-0e14-4b48-b74f-05bfd90c0e17-istiod-ca-cert" pod "91727fdb-0e14-4b48-b74f-05bfd90c0e17" (UID: "91727fdb-0e14-4b48-b74f-05bfd90c0e17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:03:13.303840 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.303711 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91727fdb-0e14-4b48-b74f-05bfd90c0e17-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:13.303840 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.303727 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91727fdb-0e14-4b48-b74f-05bfd90c0e17-credential-socket" pod "91727fdb-0e14-4b48-b74f-05bfd90c0e17" (UID: "91727fdb-0e14-4b48-b74f-05bfd90c0e17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:03:13.303840 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.303736 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/189ebc59-a986-4e40-a9b1-25a350c5d397-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:13.303840 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.303751 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/189ebc59-a986-4e40-a9b1-25a350c5d397-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:13.303840 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.303764 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/189ebc59-a986-4e40-a9b1-25a350c5d397-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:13.303840 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.303777 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/189ebc59-a986-4e40-a9b1-25a350c5d397-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:13.303840 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.303790 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91727fdb-0e14-4b48-b74f-05bfd90c0e17-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:13.303840 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.303804 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/189ebc59-a986-4e40-a9b1-25a350c5d397-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:13.303840 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.303817 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91727fdb-0e14-4b48-b74f-05bfd90c0e17-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:13.304417 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.304014 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91727fdb-0e14-4b48-b74f-05bfd90c0e17-istio-envoy" pod "91727fdb-0e14-4b48-b74f-05bfd90c0e17" (UID: "91727fdb-0e14-4b48-b74f-05bfd90c0e17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:03:13.304417 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.304295 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/189ebc59-a986-4e40-a9b1-25a350c5d397-istio-podinfo" pod "189ebc59-a986-4e40-a9b1-25a350c5d397" (UID: "189ebc59-a986-4e40-a9b1-25a350c5d397"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:03:13.306344 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.306322 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91727fdb-0e14-4b48-b74f-05bfd90c0e17-istio-podinfo" pod "91727fdb-0e14-4b48-b74f-05bfd90c0e17" (UID: "91727fdb-0e14-4b48-b74f-05bfd90c0e17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:03:13.307156 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.307112 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91727fdb-0e14-4b48-b74f-05bfd90c0e17-istio-token" pod "91727fdb-0e14-4b48-b74f-05bfd90c0e17" (UID: "91727fdb-0e14-4b48-b74f-05bfd90c0e17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:03:13.307255 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.307159 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189ebc59-a986-4e40-a9b1-25a350c5d397-kube-api-access-2jp4x" pod "189ebc59-a986-4e40-a9b1-25a350c5d397" (UID: "189ebc59-a986-4e40-a9b1-25a350c5d397"). InnerVolumeSpecName "kube-api-access-2jp4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:03:13.307255 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.307177 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189ebc59-a986-4e40-a9b1-25a350c5d397-istio-token" pod "189ebc59-a986-4e40-a9b1-25a350c5d397" (UID: "189ebc59-a986-4e40-a9b1-25a350c5d397"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:03:13.308155 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.308122 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189ebc59-a986-4e40-a9b1-25a350c5d397-istio-envoy" pod "189ebc59-a986-4e40-a9b1-25a350c5d397" (UID: "189ebc59-a986-4e40-a9b1-25a350c5d397"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:03:13.308694 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.308670 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91727fdb-0e14-4b48-b74f-05bfd90c0e17-kube-api-access-9hnmd" pod "91727fdb-0e14-4b48-b74f-05bfd90c0e17" (UID: "91727fdb-0e14-4b48-b74f-05bfd90c0e17"). InnerVolumeSpecName "kube-api-access-9hnmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:03:13.405401 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.405363 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91727fdb-0e14-4b48-b74f-05bfd90c0e17-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:13.405401 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.405403 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jp4x\" (UniqueName: \"kubernetes.io/projected/189ebc59-a986-4e40-a9b1-25a350c5d397-kube-api-access-2jp4x\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:13.405624 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.405419 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91727fdb-0e14-4b48-b74f-05bfd90c0e17-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:13.405624 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.405433 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91727fdb-0e14-4b48-b74f-05bfd90c0e17-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:13.405624 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.405448 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91727fdb-0e14-4b48-b74f-05bfd90c0e17-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:13.405624 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.405461 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/189ebc59-a986-4e40-a9b1-25a350c5d397-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:13.405624 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.405478 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hnmd\" (UniqueName: \"kubernetes.io/projected/91727fdb-0e14-4b48-b74f-05bfd90c0e17-kube-api-access-9hnmd\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:13.405624 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.405493 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/189ebc59-a986-4e40-a9b1-25a350c5d397-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:13.405624 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.405508 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91727fdb-0e14-4b48-b74f-05bfd90c0e17-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:13.405624 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.405523 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/189ebc59-a986-4e40-a9b1-25a350c5d397-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:13.596470 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.591366 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d"] Oct 31 13:03:13.596470 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.592620 6841 cpu_manager.go:414] "containerMap: removing container" podUID="189ebc59-a986-4e40-a9b1-25a350c5d397" containerName="istio-proxy" Oct 31 13:03:13.596470 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.592641 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="189ebc59-a986-4e40-a9b1-25a350c5d397" containerName="istio-proxy" Oct 31 13:03:13.596470 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.592690 6841 cpu_manager.go:414] "containerMap: removing container" podUID="91727fdb-0e14-4b48-b74f-05bfd90c0e17" containerName="istio-proxy" Oct 31 13:03:13.596470 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.592699 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91727fdb-0e14-4b48-b74f-05bfd90c0e17" containerName="istio-proxy" Oct 31 13:03:13.596470 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.592866 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="91727fdb-0e14-4b48-b74f-05bfd90c0e17" containerName="istio-proxy" Oct 31 13:03:13.596470 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.592893 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="189ebc59-a986-4e40-a9b1-25a350c5d397" containerName="istio-proxy" Oct 31 13:03:13.598770 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.598226 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d"] Oct 31 13:03:13.598770 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.598351 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d" Oct 31 13:03:13.604266 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.604241 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-49bffb-serving-cert\"" Oct 31 13:03:13.607744 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.607713 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aed2f8d-a0b0-4416-b07c-e73f310bfb0f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d\" (UID: \"2aed2f8d-a0b0-4416-b07c-e73f310bfb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d" Oct 31 13:03:13.608329 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.608302 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aed2f8d-a0b0-4416-b07c-e73f310bfb0f-metrics-certs\") pod \"maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d\" (UID: \"2aed2f8d-a0b0-4416-b07c-e73f310bfb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d" Oct 31 13:03:13.608421 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.608397 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6z86\" (UniqueName: \"kubernetes.io/projected/2aed2f8d-a0b0-4416-b07c-e73f310bfb0f-kube-api-access-n6z86\") pod \"maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d\" (UID: \"2aed2f8d-a0b0-4416-b07c-e73f310bfb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d" Oct 31 13:03:13.608495 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.608479 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aed2f8d-a0b0-4416-b07c-e73f310bfb0f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d\" (UID: \"2aed2f8d-a0b0-4416-b07c-e73f310bfb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d" Oct 31 13:03:13.710111 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.710077 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aed2f8d-a0b0-4416-b07c-e73f310bfb0f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d\" (UID: \"2aed2f8d-a0b0-4416-b07c-e73f310bfb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d" Oct 31 13:03:13.710256 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.710144 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aed2f8d-a0b0-4416-b07c-e73f310bfb0f-metrics-certs\") pod \"maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d\" (UID: \"2aed2f8d-a0b0-4416-b07c-e73f310bfb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d" Oct 31 13:03:13.710256 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.710169 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6z86\" (UniqueName: \"kubernetes.io/projected/2aed2f8d-a0b0-4416-b07c-e73f310bfb0f-kube-api-access-n6z86\") pod \"maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d\" (UID: \"2aed2f8d-a0b0-4416-b07c-e73f310bfb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d" Oct 31 13:03:13.710256 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.710198 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aed2f8d-a0b0-4416-b07c-e73f310bfb0f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d\" (UID: \"2aed2f8d-a0b0-4416-b07c-e73f310bfb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d" Oct 31 13:03:13.710819 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.710798 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aed2f8d-a0b0-4416-b07c-e73f310bfb0f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d\" (UID: \"2aed2f8d-a0b0-4416-b07c-e73f310bfb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d" Oct 31 13:03:13.712721 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.712698 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aed2f8d-a0b0-4416-b07c-e73f310bfb0f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d\" (UID: \"2aed2f8d-a0b0-4416-b07c-e73f310bfb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d" Oct 31 13:03:13.712801 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.712699 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aed2f8d-a0b0-4416-b07c-e73f310bfb0f-metrics-certs\") pod \"maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d\" (UID: \"2aed2f8d-a0b0-4416-b07c-e73f310bfb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d" Oct 31 13:03:13.748005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.744738 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6z86\" (UniqueName: \"kubernetes.io/projected/2aed2f8d-a0b0-4416-b07c-e73f310bfb0f-kube-api-access-n6z86\") pod \"maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d\" (UID: \"2aed2f8d-a0b0-4416-b07c-e73f310bfb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d" Oct 31 13:03:13.773006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.760526 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:13.876057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.869249 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-6bbff8b447-7827t" event={"ID":"faafb6b6-9a5a-4a60-a327-1629d2c49868","Type":"ContainerStarted","Data":"fd4e6c1e0813d908b1058c97ac39f25c9bf4c945dbeb2c0a938a4fd67c8bf36e"} Oct 31 13:03:13.876057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.869324 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-6bbff8b447-7827t" event={"ID":"faafb6b6-9a5a-4a60-a327-1629d2c49868","Type":"ContainerStarted","Data":"dd0b58eb4ff6d4c5702fefc1c0458925464842f1ce97a5eca668f26c8b80814f"} Oct 31 13:03:13.876057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.870521 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-6bbff8b447-7827t" Oct 31 13:03:13.876057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.872884 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-12056b-54cd8754db-r8p4d" event={"ID":"6a4859e8-5b79-4f2a-afe6-0b26a64cce08","Type":"ContainerStarted","Data":"5413df0b729086d9d32bf9c59ae46670699436492c5be164e43513ccb3a02c60"} Oct 31 13:03:13.876057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.872936 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-12056b-54cd8754db-r8p4d" event={"ID":"6a4859e8-5b79-4f2a-afe6-0b26a64cce08","Type":"ContainerStarted","Data":"71682856657dadec5d5115bf7a6fff7091df7eb04bb54cb84541c04a1f4a3f0d"} Oct 31 13:03:13.876057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.873782 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-12056b-54cd8754db-r8p4d" Oct 31 13:03:13.894004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.888541 6841 generic.go:356] "Generic (PLEG): container finished" podID="91727fdb-0e14-4b48-b74f-05bfd90c0e17" containerID="7fcac52694ff14c2a80efaf7248ea045280cf904060fdc34e1ad18454948a7af" exitCode=0 Oct 31 13:03:13.894004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.888752 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" event={"ID":"91727fdb-0e14-4b48-b74f-05bfd90c0e17","Type":"ContainerDied","Data":"7fcac52694ff14c2a80efaf7248ea045280cf904060fdc34e1ad18454948a7af"} Oct 31 13:03:13.894004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.888810 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" event={"ID":"91727fdb-0e14-4b48-b74f-05bfd90c0e17","Type":"ContainerDied","Data":"739b95588ee20fa27026e5fdb3cb578989aea57dd308363b61844ba02e830e31"} Oct 31 13:03:13.894004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.888840 6841 scope.go:122] "RemoveContainer" containerID="7fcac52694ff14c2a80efaf7248ea045280cf904060fdc34e1ad18454948a7af" Oct 31 13:03:13.894004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.889037 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb" Oct 31 13:03:13.918026 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.916682 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-6bbff8b447-7827t" podStartSLOduration=1.916663828 podStartE2EDuration="1.916663828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:13.911401793 +0000 UTC m=+1840.394524699" watchObservedRunningTime="2026-10-31 13:03:13.916663828 +0000 UTC m=+1840.399786733" Oct 31 13:03:13.921168 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.921136 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d" Oct 31 13:03:13.940575 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.938449 6841 generic.go:356] "Generic (PLEG): container finished" podID="189ebc59-a986-4e40-a9b1-25a350c5d397" containerID="646855c7cb04a308807921ad8a07dff98776c8bd427c21a9b7cbaf234565d69a" exitCode=0 Oct 31 13:03:13.940575 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.938661 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" event={"ID":"189ebc59-a986-4e40-a9b1-25a350c5d397","Type":"ContainerDied","Data":"646855c7cb04a308807921ad8a07dff98776c8bd427c21a9b7cbaf234565d69a"} Oct 31 13:03:13.940575 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.938718 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" event={"ID":"189ebc59-a986-4e40-a9b1-25a350c5d397","Type":"ContainerDied","Data":"4f0cd0617f5cc38d1e5c4a6e7b1b6bd41518368bf2e862ebea853e5bc034e1cf"} Oct 31 13:03:13.940575 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.938817 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58" Oct 31 13:03:13.989070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:13.979576 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-12056b-54cd8754db-r8p4d" podStartSLOduration=1.979557061 podStartE2EDuration="1.979557061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:13.959853519 +0000 UTC m=+1840.442976422" watchObservedRunningTime="2026-10-31 13:03:13.979557061 +0000 UTC m=+1840.462679963" Oct 31 13:03:14.012573 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:14.012542 6841 scope.go:122] "RemoveContainer" containerID="7fcac52694ff14c2a80efaf7248ea045280cf904060fdc34e1ad18454948a7af" Oct 31 13:03:14.012922 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:14.012883 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb"] Oct 31 13:03:14.013110 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:03:14.013082 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fcac52694ff14c2a80efaf7248ea045280cf904060fdc34e1ad18454948a7af\": container with ID starting with 7fcac52694ff14c2a80efaf7248ea045280cf904060fdc34e1ad18454948a7af not found: ID does not exist" containerID="7fcac52694ff14c2a80efaf7248ea045280cf904060fdc34e1ad18454948a7af" Oct 31 13:03:14.013177 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:14.013124 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fcac52694ff14c2a80efaf7248ea045280cf904060fdc34e1ad18454948a7af"} err="failed to get container status \"7fcac52694ff14c2a80efaf7248ea045280cf904060fdc34e1ad18454948a7af\": rpc error: code = NotFound desc = could not find container \"7fcac52694ff14c2a80efaf7248ea045280cf904060fdc34e1ad18454948a7af\": container with ID starting with 7fcac52694ff14c2a80efaf7248ea045280cf904060fdc34e1ad18454948a7af not found: ID does not exist" Oct 31 13:03:14.013177 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:14.013150 6841 scope.go:122] "RemoveContainer" containerID="646855c7cb04a308807921ad8a07dff98776c8bd427c21a9b7cbaf234565d69a" Oct 31 13:03:14.026016 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:14.025178 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-c4njb"] Oct 31 13:03:14.026217 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:14.026090 6841 scope.go:122] "RemoveContainer" containerID="646855c7cb04a308807921ad8a07dff98776c8bd427c21a9b7cbaf234565d69a" Oct 31 13:03:14.026413 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:03:14.026389 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"646855c7cb04a308807921ad8a07dff98776c8bd427c21a9b7cbaf234565d69a\": container with ID starting with 646855c7cb04a308807921ad8a07dff98776c8bd427c21a9b7cbaf234565d69a not found: ID does not exist" containerID="646855c7cb04a308807921ad8a07dff98776c8bd427c21a9b7cbaf234565d69a" Oct 31 13:03:14.026469 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:14.026426 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"646855c7cb04a308807921ad8a07dff98776c8bd427c21a9b7cbaf234565d69a"} err="failed to get container status \"646855c7cb04a308807921ad8a07dff98776c8bd427c21a9b7cbaf234565d69a\": rpc error: code = NotFound desc = could not find container \"646855c7cb04a308807921ad8a07dff98776c8bd427c21a9b7cbaf234565d69a\": container with ID starting with 646855c7cb04a308807921ad8a07dff98776c8bd427c21a9b7cbaf234565d69a not found: ID does not exist" Oct 31 13:03:14.081849 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:14.081806 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58"] Oct 31 13:03:14.093318 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:14.084696 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5tk58"] Oct 31 13:03:14.125688 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:14.116970 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:14.639425 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:14.638791 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d"] Oct 31 13:03:14.836752 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:14.836669 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:14.985019 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:14.980698 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d" event={"ID":"2aed2f8d-a0b0-4416-b07c-e73f310bfb0f","Type":"ContainerStarted","Data":"ed9b23e49294ec4ab3b28610d0c072face4cb6ab35a22507bc457e9519ccea7d"} Oct 31 13:03:15.487699 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:15.485211 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws"] Oct 31 13:03:15.497030 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:15.491816 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws"] Oct 31 13:03:15.497030 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:15.491925 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws" Oct 31 13:03:15.497290 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:15.497233 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-28e4df-serving-cert\"" Oct 31 13:03:15.535467 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:15.535440 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmfgx\" (UniqueName: \"kubernetes.io/projected/f913d7e0-3523-4694-bbf3-294fcb0af9e3-kube-api-access-nmfgx\") pod \"maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws\" (UID: \"f913d7e0-3523-4694-bbf3-294fcb0af9e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws" Oct 31 13:03:15.535617 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:15.535482 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f913d7e0-3523-4694-bbf3-294fcb0af9e3-metrics-certs\") pod \"maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws\" (UID: \"f913d7e0-3523-4694-bbf3-294fcb0af9e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws" Oct 31 13:03:15.535682 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:15.535615 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f913d7e0-3523-4694-bbf3-294fcb0af9e3-maas-api-tls\") pod \"maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws\" (UID: \"f913d7e0-3523-4694-bbf3-294fcb0af9e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws" Oct 31 13:03:15.535729 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:15.535716 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f913d7e0-3523-4694-bbf3-294fcb0af9e3-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws\" (UID: \"f913d7e0-3523-4694-bbf3-294fcb0af9e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws" Oct 31 13:03:15.606879 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:15.606847 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:15.636525 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:15.636496 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f913d7e0-3523-4694-bbf3-294fcb0af9e3-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws\" (UID: \"f913d7e0-3523-4694-bbf3-294fcb0af9e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws" Oct 31 13:03:15.636689 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:15.636609 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmfgx\" (UniqueName: \"kubernetes.io/projected/f913d7e0-3523-4694-bbf3-294fcb0af9e3-kube-api-access-nmfgx\") pod \"maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws\" (UID: \"f913d7e0-3523-4694-bbf3-294fcb0af9e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws" Oct 31 13:03:15.636689 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:15.636659 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f913d7e0-3523-4694-bbf3-294fcb0af9e3-metrics-certs\") pod \"maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws\" (UID: \"f913d7e0-3523-4694-bbf3-294fcb0af9e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws" Oct 31 13:03:15.636844 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:15.636765 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f913d7e0-3523-4694-bbf3-294fcb0af9e3-maas-api-tls\") pod \"maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws\" (UID: \"f913d7e0-3523-4694-bbf3-294fcb0af9e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws" Oct 31 13:03:15.637378 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:15.637354 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f913d7e0-3523-4694-bbf3-294fcb0af9e3-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws\" (UID: \"f913d7e0-3523-4694-bbf3-294fcb0af9e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws" Oct 31 13:03:15.639775 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:15.639710 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f913d7e0-3523-4694-bbf3-294fcb0af9e3-maas-api-tls\") pod \"maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws\" (UID: \"f913d7e0-3523-4694-bbf3-294fcb0af9e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws" Oct 31 13:03:15.686007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:15.658570 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f913d7e0-3523-4694-bbf3-294fcb0af9e3-metrics-certs\") pod \"maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws\" (UID: \"f913d7e0-3523-4694-bbf3-294fcb0af9e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws" Oct 31 13:03:15.686007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:15.681119 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmfgx\" (UniqueName: \"kubernetes.io/projected/f913d7e0-3523-4694-bbf3-294fcb0af9e3-kube-api-access-nmfgx\") pod \"maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws\" (UID: \"f913d7e0-3523-4694-bbf3-294fcb0af9e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws" Oct 31 13:03:15.757629 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:15.723040 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="189ebc59-a986-4e40-a9b1-25a350c5d397" path="/var/lib/kubelet/pods/189ebc59-a986-4e40-a9b1-25a350c5d397/volumes" Oct 31 13:03:15.757629 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:15.723909 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91727fdb-0e14-4b48-b74f-05bfd90c0e17" path="/var/lib/kubelet/pods/91727fdb-0e14-4b48-b74f-05bfd90c0e17/volumes" Oct 31 13:03:15.814042 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:15.809296 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws" Oct 31 13:03:15.819222 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:15.819189 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:15.892147 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:15.890586 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg" Oct 31 13:03:16.011745 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:16.011612 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d" event={"ID":"2aed2f8d-a0b0-4416-b07c-e73f310bfb0f","Type":"ContainerStarted","Data":"5af71d0cfa59e2e6b80d8ca0aa8233710a3b66d1d6c97af3b7634662170515f6"} Oct 31 13:03:16.012523 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:16.012469 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d" Oct 31 13:03:16.035431 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:16.035375 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d" podStartSLOduration=2.593695975 podStartE2EDuration="3.035356949s" totalImagesPullingTime="441.660974ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:16.033718899 +0000 UTC m=+1842.516841801" watchObservedRunningTime="2026-10-31 13:03:16.035356949 +0000 UTC m=+1842.518479851" Oct 31 13:03:16.186849 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:16.186816 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws"] Oct 31 13:03:16.187086 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:03:16.187063 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf913d7e0_3523_4694_bbf3_294fcb0af9e3.slice/crio-a0803de29bd793034fb4cf7763aa475f801e8206362defc7d7bf1cfe1dd583c1 WatchSource:0}: Error finding container a0803de29bd793034fb4cf7763aa475f801e8206362defc7d7bf1cfe1dd583c1: Status 404 returned error can't find the container with id a0803de29bd793034fb4cf7763aa475f801e8206362defc7d7bf1cfe1dd583c1 Oct 31 13:03:16.613517 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:16.613479 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:17.022401 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:17.022368 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws" event={"ID":"f913d7e0-3523-4694-bbf3-294fcb0af9e3","Type":"ContainerStarted","Data":"5c44c243fddbd8c50c18f5a4bc5f8755c63b453c502c75a5250844a0fb50fe97"} Oct 31 13:03:17.023042 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:17.022963 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws" event={"ID":"f913d7e0-3523-4694-bbf3-294fcb0af9e3","Type":"ContainerStarted","Data":"a0803de29bd793034fb4cf7763aa475f801e8206362defc7d7bf1cfe1dd583c1"} Oct 31 13:03:17.023230 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:17.023044 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws" Oct 31 13:03:17.047311 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:17.047257 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws" podStartSLOduration=1.5109522819999999 podStartE2EDuration="2.047239357s" totalImagesPullingTime="536.287075ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:17.044729032 +0000 UTC m=+1843.527851929" watchObservedRunningTime="2026-10-31 13:03:17.047239357 +0000 UTC m=+1843.530362259" Oct 31 13:03:17.273685 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:17.273602 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0db792-54bf4947dd-q7k6g"] Oct 31 13:03:17.273853 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:17.273786 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db792-54bf4947dd-q7k6g" podUID="e238ba43-f409-4888-aff1-b47b000c0211" containerName="payload-processing" containerID="cri-o://817e7c84c84bab3d00079c648ce3648d8785be2f26f7a232f54ec20288cc1f71" gracePeriod=30 Oct 31 13:03:17.290294 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:17.290249 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-5d98778b55-snvlr"] Oct 31 13:03:17.290515 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:17.290434 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-5d98778b55-snvlr" podUID="93391f16-8d6c-45a1-9bb6-838d6b65289d" containerName="payload-pre-processing" containerID="cri-o://c9c2e355f6a1c2236bc2d61462ca6f60915c524e8fd960f95c7f4ed999440064" gracePeriod=30 Oct 31 13:03:17.323918 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:17.323705 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:17.613054 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:17.611495 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:17.782995 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:17.781685 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-28e4df-76bf65bc65-72zf6" Oct 31 13:03:17.862040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:17.859437 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db792-54bf4947dd-q7k6g" Oct 31 13:03:17.935122 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:17.932508 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e238ba43-f409-4888-aff1-b47b000c0211-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e238ba43-f409-4888-aff1-b47b000c0211-plugins-config-volume\") pod \"e238ba43-f409-4888-aff1-b47b000c0211\" (UID: \"e238ba43-f409-4888-aff1-b47b000c0211\") " Oct 31 13:03:17.935122 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:17.932575 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e238ba43-f409-4888-aff1-b47b000c0211-kube-api-access-sjcfd\" (UniqueName: \"kubernetes.io/projected/e238ba43-f409-4888-aff1-b47b000c0211-kube-api-access-sjcfd\") pod \"e238ba43-f409-4888-aff1-b47b000c0211\" (UID: \"e238ba43-f409-4888-aff1-b47b000c0211\") " Oct 31 13:03:17.935122 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:17.933209 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e238ba43-f409-4888-aff1-b47b000c0211-plugins-config-volume" pod "e238ba43-f409-4888-aff1-b47b000c0211" (UID: "e238ba43-f409-4888-aff1-b47b000c0211"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:03:17.938702 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:17.936884 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:17.946756 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:17.946720 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e238ba43-f409-4888-aff1-b47b000c0211-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:17.976505 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:17.975338 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e238ba43-f409-4888-aff1-b47b000c0211-kube-api-access-sjcfd" pod "e238ba43-f409-4888-aff1-b47b000c0211" (UID: "e238ba43-f409-4888-aff1-b47b000c0211"). InnerVolumeSpecName "kube-api-access-sjcfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:03:18.032581 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:18.032546 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-5d98778b55-snvlr" Oct 31 13:03:18.033020 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:18.032633 6841 generic.go:356] "Generic (PLEG): container finished" podID="93391f16-8d6c-45a1-9bb6-838d6b65289d" containerID="c9c2e355f6a1c2236bc2d61462ca6f60915c524e8fd960f95c7f4ed999440064" exitCode=0 Oct 31 13:03:18.033020 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:18.032735 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-5d98778b55-snvlr" event={"ID":"93391f16-8d6c-45a1-9bb6-838d6b65289d","Type":"ContainerDied","Data":"c9c2e355f6a1c2236bc2d61462ca6f60915c524e8fd960f95c7f4ed999440064"} Oct 31 13:03:18.033020 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:18.032789 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-5d98778b55-snvlr" event={"ID":"93391f16-8d6c-45a1-9bb6-838d6b65289d","Type":"ContainerDied","Data":"637b0fd96472785eefda8d2e32af1c1924dc302b2030515cbe3c69a8b2e8cce3"} Oct 31 13:03:18.033020 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:18.032817 6841 scope.go:122] "RemoveContainer" containerID="c9c2e355f6a1c2236bc2d61462ca6f60915c524e8fd960f95c7f4ed999440064" Oct 31 13:03:18.034484 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:18.034456 6841 generic.go:356] "Generic (PLEG): container finished" podID="e238ba43-f409-4888-aff1-b47b000c0211" containerID="817e7c84c84bab3d00079c648ce3648d8785be2f26f7a232f54ec20288cc1f71" exitCode=0 Oct 31 13:03:18.034617 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:18.034500 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db792-54bf4947dd-q7k6g" Oct 31 13:03:18.034753 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:18.034725 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db792-54bf4947dd-q7k6g" event={"ID":"e238ba43-f409-4888-aff1-b47b000c0211","Type":"ContainerDied","Data":"817e7c84c84bab3d00079c648ce3648d8785be2f26f7a232f54ec20288cc1f71"} Oct 31 13:03:18.034875 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:18.034786 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db792-54bf4947dd-q7k6g" event={"ID":"e238ba43-f409-4888-aff1-b47b000c0211","Type":"ContainerDied","Data":"ec6c9180125ec865b7bbdd5b7ddf513b2318f3f69a28fe3020db4c77dc4cbfc2"} Oct 31 13:03:18.044545 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:18.044521 6841 scope.go:122] "RemoveContainer" containerID="c9c2e355f6a1c2236bc2d61462ca6f60915c524e8fd960f95c7f4ed999440064" Oct 31 13:03:18.044843 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:03:18.044826 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9c2e355f6a1c2236bc2d61462ca6f60915c524e8fd960f95c7f4ed999440064\": container with ID starting with c9c2e355f6a1c2236bc2d61462ca6f60915c524e8fd960f95c7f4ed999440064 not found: ID does not exist" containerID="c9c2e355f6a1c2236bc2d61462ca6f60915c524e8fd960f95c7f4ed999440064" Oct 31 13:03:18.044929 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:18.044855 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9c2e355f6a1c2236bc2d61462ca6f60915c524e8fd960f95c7f4ed999440064"} err="failed to get container status \"c9c2e355f6a1c2236bc2d61462ca6f60915c524e8fd960f95c7f4ed999440064\": rpc error: code = NotFound desc = could not find container \"c9c2e355f6a1c2236bc2d61462ca6f60915c524e8fd960f95c7f4ed999440064\": container with ID starting with c9c2e355f6a1c2236bc2d61462ca6f60915c524e8fd960f95c7f4ed999440064 not found: ID does not exist" Oct 31 13:03:18.044929 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:18.044882 6841 scope.go:122] "RemoveContainer" containerID="817e7c84c84bab3d00079c648ce3648d8785be2f26f7a232f54ec20288cc1f71" Oct 31 13:03:18.047579 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:18.047558 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjcfd\" (UniqueName: \"kubernetes.io/projected/e238ba43-f409-4888-aff1-b47b000c0211-kube-api-access-sjcfd\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:18.054279 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:18.054259 6841 scope.go:122] "RemoveContainer" containerID="817e7c84c84bab3d00079c648ce3648d8785be2f26f7a232f54ec20288cc1f71" Oct 31 13:03:18.054581 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:03:18.054561 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"817e7c84c84bab3d00079c648ce3648d8785be2f26f7a232f54ec20288cc1f71\": container with ID starting with 817e7c84c84bab3d00079c648ce3648d8785be2f26f7a232f54ec20288cc1f71 not found: ID does not exist" containerID="817e7c84c84bab3d00079c648ce3648d8785be2f26f7a232f54ec20288cc1f71" Oct 31 13:03:18.054647 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:18.054587 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"817e7c84c84bab3d00079c648ce3648d8785be2f26f7a232f54ec20288cc1f71"} err="failed to get container status \"817e7c84c84bab3d00079c648ce3648d8785be2f26f7a232f54ec20288cc1f71\": rpc error: code = NotFound desc = could not find container \"817e7c84c84bab3d00079c648ce3648d8785be2f26f7a232f54ec20288cc1f71\": container with ID starting with 817e7c84c84bab3d00079c648ce3648d8785be2f26f7a232f54ec20288cc1f71 not found: ID does not exist" Oct 31 13:03:18.096657 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:18.096592 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0db792-54bf4947dd-q7k6g"] Oct 31 13:03:18.100201 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:18.100172 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0db792-54bf4947dd-q7k6g"] Oct 31 13:03:18.148913 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:18.148870 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93391f16-8d6c-45a1-9bb6-838d6b65289d-kube-api-access-gtp2s\" (UniqueName: \"kubernetes.io/projected/93391f16-8d6c-45a1-9bb6-838d6b65289d-kube-api-access-gtp2s\") pod \"93391f16-8d6c-45a1-9bb6-838d6b65289d\" (UID: \"93391f16-8d6c-45a1-9bb6-838d6b65289d\") " Oct 31 13:03:18.149176 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:18.149157 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93391f16-8d6c-45a1-9bb6-838d6b65289d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93391f16-8d6c-45a1-9bb6-838d6b65289d-plugins-config-volume\") pod \"93391f16-8d6c-45a1-9bb6-838d6b65289d\" (UID: \"93391f16-8d6c-45a1-9bb6-838d6b65289d\") " Oct 31 13:03:18.149515 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:18.149492 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93391f16-8d6c-45a1-9bb6-838d6b65289d-plugins-config-volume" pod "93391f16-8d6c-45a1-9bb6-838d6b65289d" (UID: "93391f16-8d6c-45a1-9bb6-838d6b65289d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:03:18.150011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:18.149966 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93391f16-8d6c-45a1-9bb6-838d6b65289d-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:18.151085 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:18.151058 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93391f16-8d6c-45a1-9bb6-838d6b65289d-kube-api-access-gtp2s" pod "93391f16-8d6c-45a1-9bb6-838d6b65289d" (UID: "93391f16-8d6c-45a1-9bb6-838d6b65289d"). InnerVolumeSpecName "kube-api-access-gtp2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:03:18.260195 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:18.260152 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtp2s\" (UniqueName: \"kubernetes.io/projected/93391f16-8d6c-45a1-9bb6-838d6b65289d-kube-api-access-gtp2s\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:18.817031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:18.809808 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-28e4df-9ff5f7b6c-fvgw4" Oct 31 13:03:18.951716 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:18.951677 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:19.045780 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:19.045747 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-5d98778b55-snvlr" Oct 31 13:03:19.094013 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:19.093943 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-5d98778b55-snvlr"] Oct 31 13:03:19.100081 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:19.100048 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-5d98778b55-snvlr"] Oct 31 13:03:19.710742 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:19.710699 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:19.714230 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:19.714192 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93391f16-8d6c-45a1-9bb6-838d6b65289d" path="/var/lib/kubelet/pods/93391f16-8d6c-45a1-9bb6-838d6b65289d/volumes" Oct 31 13:03:19.715001 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:19.714829 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e238ba43-f409-4888-aff1-b47b000c0211" path="/var/lib/kubelet/pods/e238ba43-f409-4888-aff1-b47b000c0211/volumes" Oct 31 13:03:19.932162 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:19.932121 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:20.036576 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:03:20.036494 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:03:20.217062 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:20.217023 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:20.652008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:20.651541 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:21.054621 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:21.054588 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:21.245233 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:21.244036 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": read tcp 10.134.0.2:42424->10.134.0.121:15021: read: connection reset by peer" start-of-body= Oct 31 13:03:21.245233 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:21.244102 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" podUID="13c0b6da-6006-44b6-8677-d16ffd433df2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": read tcp 10.134.0.2:42424->10.134.0.121:15021: read: connection reset by peer" Oct 31 13:03:22.057845 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:22.057797 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d" Oct 31 13:03:22.069840 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:22.069807 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:22.075187 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:22.075161 6841 generic.go:356] "Generic (PLEG): container finished" podID="13c0b6da-6006-44b6-8677-d16ffd433df2" containerID="4973ccb4472adf76b2d03bb79267b4580ef815a6276c6e778ad6376100b09d2d" exitCode=0 Oct 31 13:03:22.075321 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:22.075235 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" event={"ID":"13c0b6da-6006-44b6-8677-d16ffd433df2","Type":"ContainerDied","Data":"4973ccb4472adf76b2d03bb79267b4580ef815a6276c6e778ad6376100b09d2d"} Oct 31 13:03:22.075702 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:22.075687 6841 scope.go:122] "RemoveContainer" containerID="4973ccb4472adf76b2d03bb79267b4580ef815a6276c6e778ad6376100b09d2d" Oct 31 13:03:22.077346 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:22.077319 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:03:22.077411 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:22.077385 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:03:22.077448 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:22.077420 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:03:23.045007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:23.041048 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws" Oct 31 13:03:23.068062 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:23.067535 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:23.094266 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:23.094216 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" event={"ID":"13c0b6da-6006-44b6-8677-d16ffd433df2","Type":"ContainerStarted","Data":"86763a7e5a59fbcc1a8a4cb605298a0690bcae89ea0b98a50adbd1909be7b847"} Oct 31 13:03:23.487267 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:23.487238 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:23.514486 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:23.514451 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:03:23.514656 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:23.514616 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 31 13:03:23.514706 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:23.514655 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" podUID="13c0b6da-6006-44b6-8677-d16ffd433df2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 31 13:03:23.809763 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:23.809685 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds"] Oct 31 13:03:23.810772 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:23.810749 6841 cpu_manager.go:414] "containerMap: removing container" podUID="e238ba43-f409-4888-aff1-b47b000c0211" containerName="payload-processing" Oct 31 13:03:23.810825 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:23.810777 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e238ba43-f409-4888-aff1-b47b000c0211" containerName="payload-processing" Oct 31 13:03:23.810861 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:23.810829 6841 cpu_manager.go:414] "containerMap: removing container" podUID="93391f16-8d6c-45a1-9bb6-838d6b65289d" containerName="payload-pre-processing" Oct 31 13:03:23.810861 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:23.810837 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93391f16-8d6c-45a1-9bb6-838d6b65289d" containerName="payload-pre-processing" Oct 31 13:03:23.811068 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:23.811053 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="93391f16-8d6c-45a1-9bb6-838d6b65289d" containerName="payload-pre-processing" Oct 31 13:03:23.811109 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:23.811085 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="e238ba43-f409-4888-aff1-b47b000c0211" containerName="payload-processing" Oct 31 13:03:23.815772 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:23.815751 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:03:23.856058 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:23.838417 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-758f48-openshift-default-dockercfg-9d6f8\"" Oct 31 13:03:23.856058 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:23.841115 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds"] Oct 31 13:03:23.887833 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:23.887782 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ae8ae0-da38-4530-856c-c9e70d8484b7-istiod-ca-cert\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-9crds\" (UID: \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:03:23.888041 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:23.887840 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ae8ae0-da38-4530-856c-c9e70d8484b7-credential-socket\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-9crds\" (UID: \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:03:23.888041 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:23.887916 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ae8ae0-da38-4530-856c-c9e70d8484b7-istio-envoy\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-9crds\" (UID: \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:03:23.888199 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:23.888090 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmvnz\" (UniqueName: \"kubernetes.io/projected/e5ae8ae0-da38-4530-856c-c9e70d8484b7-kube-api-access-bmvnz\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-9crds\" (UID: \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:03:23.888199 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:23.888145 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ae8ae0-da38-4530-856c-c9e70d8484b7-istio-data\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-9crds\" (UID: \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:03:23.888314 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:23.888202 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ae8ae0-da38-4530-856c-c9e70d8484b7-istio-podinfo\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-9crds\" (UID: \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:03:23.888314 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:23.888237 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ae8ae0-da38-4530-856c-c9e70d8484b7-workload-certs\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-9crds\" (UID: \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:03:23.888314 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:23.888309 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ae8ae0-da38-4530-856c-c9e70d8484b7-workload-socket\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-9crds\" (UID: \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:03:23.888473 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:23.888340 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ae8ae0-da38-4530-856c-c9e70d8484b7-istio-token\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-9crds\" (UID: \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:03:23.994008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:23.989943 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmvnz\" (UniqueName: \"kubernetes.io/projected/e5ae8ae0-da38-4530-856c-c9e70d8484b7-kube-api-access-bmvnz\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-9crds\" (UID: \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:03:23.994008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:23.990044 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ae8ae0-da38-4530-856c-c9e70d8484b7-istio-data\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-9crds\" (UID: \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:03:23.994008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:23.990104 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ae8ae0-da38-4530-856c-c9e70d8484b7-istio-podinfo\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-9crds\" (UID: \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:03:23.994008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:23.990140 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ae8ae0-da38-4530-856c-c9e70d8484b7-workload-certs\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-9crds\" (UID: \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:03:23.994008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:23.990212 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ae8ae0-da38-4530-856c-c9e70d8484b7-workload-socket\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-9crds\" (UID: \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:03:23.994008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:23.990246 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ae8ae0-da38-4530-856c-c9e70d8484b7-istio-token\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-9crds\" (UID: \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:03:23.994008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:23.990362 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ae8ae0-da38-4530-856c-c9e70d8484b7-istiod-ca-cert\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-9crds\" (UID: \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:03:23.994008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:23.990401 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ae8ae0-da38-4530-856c-c9e70d8484b7-credential-socket\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-9crds\" (UID: \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:03:23.994008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:23.990480 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ae8ae0-da38-4530-856c-c9e70d8484b7-istio-envoy\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-9crds\" (UID: \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:03:24.032013 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:24.004419 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ae8ae0-da38-4530-856c-c9e70d8484b7-workload-certs\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-9crds\" (UID: \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:03:24.032013 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:24.004942 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ae8ae0-da38-4530-856c-c9e70d8484b7-workload-socket\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-9crds\" (UID: \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:03:24.032013 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:24.004994 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ae8ae0-da38-4530-856c-c9e70d8484b7-istio-data\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-9crds\" (UID: \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:03:24.032013 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:24.005575 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ae8ae0-da38-4530-856c-c9e70d8484b7-istiod-ca-cert\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-9crds\" (UID: \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:03:24.044241 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:24.034193 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:24.070045 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:24.066604 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ae8ae0-da38-4530-856c-c9e70d8484b7-credential-socket\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-9crds\" (UID: \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:03:24.167044 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:24.166935 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ae8ae0-da38-4530-856c-c9e70d8484b7-istio-token\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-9crds\" (UID: \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:03:24.176072 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:24.169589 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ae8ae0-da38-4530-856c-c9e70d8484b7-istio-envoy\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-9crds\" (UID: \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:03:24.176365 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:24.174669 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmvnz\" (UniqueName: \"kubernetes.io/projected/e5ae8ae0-da38-4530-856c-c9e70d8484b7-kube-api-access-bmvnz\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-9crds\" (UID: \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:03:24.194005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:24.189917 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ae8ae0-da38-4530-856c-c9e70d8484b7-istio-podinfo\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-9crds\" (UID: \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:03:24.460005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:24.459799 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:24.460223 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:24.460193 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:03:24.514758 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:24.514728 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 31 13:03:24.514911 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:24.514787 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" podUID="13c0b6da-6006-44b6-8677-d16ffd433df2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 31 13:03:24.768704 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:24.768657 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:24.770512 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:03:24.770482 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ae8ae0_da38_4530_856c_c9e70d8484b7.slice/crio-5604d38fb7f3fd427e42f4c2ffc005f7b73fe709e12af1b88f0438fa39b78c29 WatchSource:0}: Error finding container 5604d38fb7f3fd427e42f4c2ffc005f7b73fe709e12af1b88f0438fa39b78c29: Status 404 returned error can't find the container with id 5604d38fb7f3fd427e42f4c2ffc005f7b73fe709e12af1b88f0438fa39b78c29 Oct 31 13:03:24.771452 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:24.771432 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds"] Oct 31 13:03:24.773212 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:24.773181 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:03:24.773316 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:24.773243 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:03:24.773316 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:24.773272 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:03:25.188184 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:25.188070 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-94c96c5b7-54j27"] Oct 31 13:03:25.195054 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:25.195025 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-688b5d7bc-vkv7k"] Oct 31 13:03:25.287480 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:25.286371 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" event={"ID":"e5ae8ae0-da38-4530-856c-c9e70d8484b7","Type":"ContainerStarted","Data":"26a00c479d203d15b2e834160249c8440bc0518335332f369fc2b2b03393cfd7"} Oct 31 13:03:25.287480 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:25.286441 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" event={"ID":"e5ae8ae0-da38-4530-856c-c9e70d8484b7","Type":"ContainerStarted","Data":"5604d38fb7f3fd427e42f4c2ffc005f7b73fe709e12af1b88f0438fa39b78c29"} Oct 31 13:03:25.336652 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:25.336599 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" podStartSLOduration=2.336583748 podStartE2EDuration="2.336583748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:25.333239231 +0000 UTC m=+1851.816362151" watchObservedRunningTime="2026-10-31 13:03:25.336583748 +0000 UTC m=+1851.819706645" Oct 31 13:03:25.366648 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:25.366611 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:25.466041 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:25.465171 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:03:25.471711 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:25.471687 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:03:25.514726 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:25.514693 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 31 13:03:25.514929 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:25.514767 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" podUID="13c0b6da-6006-44b6-8677-d16ffd433df2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 31 13:03:25.701476 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:25.701437 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:25.985700 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:25.985665 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:26.014432 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:26.014400 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-12056b-54cd8754db-r8p4d" Oct 31 13:03:26.014840 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:26.014820 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-6bbff8b447-7827t" Oct 31 13:03:26.416112 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:26.406424 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:03:26.416112 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:26.408222 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:03:26.514088 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:26.514054 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 31 13:03:26.514261 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:26.514108 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" podUID="13c0b6da-6006-44b6-8677-d16ffd433df2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 31 13:03:26.578008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:26.577368 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:27.444008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:27.442878 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:27.514054 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:27.514015 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 31 13:03:27.514242 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:27.514085 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" podUID="13c0b6da-6006-44b6-8677-d16ffd433df2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 31 13:03:27.690113 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:27.690078 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:27.924526 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:27.924441 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:28.166700 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:28.166664 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:28.513845 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:28.513808 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 31 13:03:28.514398 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:28.513871 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" podUID="13c0b6da-6006-44b6-8677-d16ffd433df2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 31 13:03:28.677922 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:28.677741 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:28.907403 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:28.907321 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:29.229788 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:29.229751 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:29.606361 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:29.606281 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:03:29.608055 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:29.607888 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:03:29.610270 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:29.610225 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:03:29.628063 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:29.628029 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:29.896025 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:29.895919 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:30.131303 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.131232 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:30.189701 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.189616 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-94c96c5b7-54j27" podUID="6c35a13a-2806-4a74-bd75-af12f5cca5ad" containerName="payload-processing" containerID="cri-o://dc3558a9f3674917b2a6fa6c594a65213dec50ec99d721426db92034c782e85c" gracePeriod=25 Oct 31 13:03:30.196096 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.196066 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-688b5d7bc-vkv7k" podUID="5a44908b-d8ff-45cd-b505-d19ce41fbba9" containerName="payload-pre-processing" containerID="cri-o://8aba772457023feb68bbb65d9d4353febc5a56af8c3043124e76fdbeab9e86d3" gracePeriod=25 Oct 31 13:03:30.482475 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.482440 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-94c96c5b7-54j27" Oct 31 13:03:30.509041 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.509012 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-688b5d7bc-vkv7k" Oct 31 13:03:30.528542 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.528511 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:30.538731 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.538704 6841 generic.go:356] "Generic (PLEG): container finished" podID="6c35a13a-2806-4a74-bd75-af12f5cca5ad" containerID="dc3558a9f3674917b2a6fa6c594a65213dec50ec99d721426db92034c782e85c" exitCode=0 Oct 31 13:03:30.538899 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.538739 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-94c96c5b7-54j27" event={"ID":"6c35a13a-2806-4a74-bd75-af12f5cca5ad","Type":"ContainerDied","Data":"dc3558a9f3674917b2a6fa6c594a65213dec50ec99d721426db92034c782e85c"} Oct 31 13:03:30.538899 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.538790 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-94c96c5b7-54j27" event={"ID":"6c35a13a-2806-4a74-bd75-af12f5cca5ad","Type":"ContainerDied","Data":"79ef69f9dda2468e734fb336d8264ec76afcf555a6fdd8d966afaeb76cb2f5cb"} Oct 31 13:03:30.538899 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.538746 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-94c96c5b7-54j27" Oct 31 13:03:30.538899 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.538821 6841 scope.go:122] "RemoveContainer" containerID="dc3558a9f3674917b2a6fa6c594a65213dec50ec99d721426db92034c782e85c" Oct 31 13:03:30.540447 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.540425 6841 generic.go:356] "Generic (PLEG): container finished" podID="5a44908b-d8ff-45cd-b505-d19ce41fbba9" containerID="8aba772457023feb68bbb65d9d4353febc5a56af8c3043124e76fdbeab9e86d3" exitCode=0 Oct 31 13:03:30.540571 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.540455 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-688b5d7bc-vkv7k" event={"ID":"5a44908b-d8ff-45cd-b505-d19ce41fbba9","Type":"ContainerDied","Data":"8aba772457023feb68bbb65d9d4353febc5a56af8c3043124e76fdbeab9e86d3"} Oct 31 13:03:30.540571 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.540533 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-688b5d7bc-vkv7k" event={"ID":"5a44908b-d8ff-45cd-b505-d19ce41fbba9","Type":"ContainerDied","Data":"a7d02f7fcd5282a1490e4f1387ccea944102113a12a64747a8e84a12000d2230"} Oct 31 13:03:30.540571 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.540472 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-688b5d7bc-vkv7k" Oct 31 13:03:30.549503 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.549486 6841 scope.go:122] "RemoveContainer" containerID="dc3558a9f3674917b2a6fa6c594a65213dec50ec99d721426db92034c782e85c" Oct 31 13:03:30.549792 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:03:30.549774 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc3558a9f3674917b2a6fa6c594a65213dec50ec99d721426db92034c782e85c\": container with ID starting with dc3558a9f3674917b2a6fa6c594a65213dec50ec99d721426db92034c782e85c not found: ID does not exist" containerID="dc3558a9f3674917b2a6fa6c594a65213dec50ec99d721426db92034c782e85c" Oct 31 13:03:30.549869 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.549802 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc3558a9f3674917b2a6fa6c594a65213dec50ec99d721426db92034c782e85c"} err="failed to get container status \"dc3558a9f3674917b2a6fa6c594a65213dec50ec99d721426db92034c782e85c\": rpc error: code = NotFound desc = could not find container \"dc3558a9f3674917b2a6fa6c594a65213dec50ec99d721426db92034c782e85c\": container with ID starting with dc3558a9f3674917b2a6fa6c594a65213dec50ec99d721426db92034c782e85c not found: ID does not exist" Oct 31 13:03:30.549869 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.549824 6841 scope.go:122] "RemoveContainer" containerID="8aba772457023feb68bbb65d9d4353febc5a56af8c3043124e76fdbeab9e86d3" Oct 31 13:03:30.562078 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.561968 6841 scope.go:122] "RemoveContainer" containerID="8aba772457023feb68bbb65d9d4353febc5a56af8c3043124e76fdbeab9e86d3" Oct 31 13:03:30.562407 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:03:30.562383 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aba772457023feb68bbb65d9d4353febc5a56af8c3043124e76fdbeab9e86d3\": container with ID starting with 8aba772457023feb68bbb65d9d4353febc5a56af8c3043124e76fdbeab9e86d3 not found: ID does not exist" containerID="8aba772457023feb68bbb65d9d4353febc5a56af8c3043124e76fdbeab9e86d3" Oct 31 13:03:30.562474 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.562416 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aba772457023feb68bbb65d9d4353febc5a56af8c3043124e76fdbeab9e86d3"} err="failed to get container status \"8aba772457023feb68bbb65d9d4353febc5a56af8c3043124e76fdbeab9e86d3\": rpc error: code = NotFound desc = could not find container \"8aba772457023feb68bbb65d9d4353febc5a56af8c3043124e76fdbeab9e86d3\": container with ID starting with 8aba772457023feb68bbb65d9d4353febc5a56af8c3043124e76fdbeab9e86d3 not found: ID does not exist" Oct 31 13:03:30.582607 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.582577 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a44908b-d8ff-45cd-b505-d19ce41fbba9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a44908b-d8ff-45cd-b505-d19ce41fbba9-plugins-config-volume\") pod \"5a44908b-d8ff-45cd-b505-d19ce41fbba9\" (UID: \"5a44908b-d8ff-45cd-b505-d19ce41fbba9\") " Oct 31 13:03:30.582743 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.582698 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a44908b-d8ff-45cd-b505-d19ce41fbba9-kube-api-access-mkdnn\" (UniqueName: \"kubernetes.io/projected/5a44908b-d8ff-45cd-b505-d19ce41fbba9-kube-api-access-mkdnn\") pod \"5a44908b-d8ff-45cd-b505-d19ce41fbba9\" (UID: \"5a44908b-d8ff-45cd-b505-d19ce41fbba9\") " Oct 31 13:03:30.582802 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.582771 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c35a13a-2806-4a74-bd75-af12f5cca5ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c35a13a-2806-4a74-bd75-af12f5cca5ad-plugins-config-volume\") pod \"6c35a13a-2806-4a74-bd75-af12f5cca5ad\" (UID: \"6c35a13a-2806-4a74-bd75-af12f5cca5ad\") " Oct 31 13:03:30.582876 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.582860 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c35a13a-2806-4a74-bd75-af12f5cca5ad-kube-api-access-xdsfx\" (UniqueName: \"kubernetes.io/projected/6c35a13a-2806-4a74-bd75-af12f5cca5ad-kube-api-access-xdsfx\") pod \"6c35a13a-2806-4a74-bd75-af12f5cca5ad\" (UID: \"6c35a13a-2806-4a74-bd75-af12f5cca5ad\") " Oct 31 13:03:30.583025 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.582965 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a44908b-d8ff-45cd-b505-d19ce41fbba9-plugins-config-volume" pod "5a44908b-d8ff-45cd-b505-d19ce41fbba9" (UID: "5a44908b-d8ff-45cd-b505-d19ce41fbba9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:03:30.583782 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.583446 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c35a13a-2806-4a74-bd75-af12f5cca5ad-plugins-config-volume" pod "6c35a13a-2806-4a74-bd75-af12f5cca5ad" (UID: "6c35a13a-2806-4a74-bd75-af12f5cca5ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:03:30.583782 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.583678 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a44908b-d8ff-45cd-b505-d19ce41fbba9-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:30.583782 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.583700 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c35a13a-2806-4a74-bd75-af12f5cca5ad-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:30.585094 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.585074 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a44908b-d8ff-45cd-b505-d19ce41fbba9-kube-api-access-mkdnn" pod "5a44908b-d8ff-45cd-b505-d19ce41fbba9" (UID: "5a44908b-d8ff-45cd-b505-d19ce41fbba9"). InnerVolumeSpecName "kube-api-access-mkdnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:03:30.585710 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.585688 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c35a13a-2806-4a74-bd75-af12f5cca5ad-kube-api-access-xdsfx" pod "6c35a13a-2806-4a74-bd75-af12f5cca5ad" (UID: "6c35a13a-2806-4a74-bd75-af12f5cca5ad"). InnerVolumeSpecName "kube-api-access-xdsfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:03:30.672198 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.672162 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx"] Oct 31 13:03:30.673088 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.673065 6841 cpu_manager.go:414] "containerMap: removing container" podUID="5a44908b-d8ff-45cd-b505-d19ce41fbba9" containerName="payload-pre-processing" Oct 31 13:03:30.673088 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.673087 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a44908b-d8ff-45cd-b505-d19ce41fbba9" containerName="payload-pre-processing" Oct 31 13:03:30.673294 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.673106 6841 cpu_manager.go:414] "containerMap: removing container" podUID="6c35a13a-2806-4a74-bd75-af12f5cca5ad" containerName="payload-processing" Oct 31 13:03:30.673294 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.673114 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c35a13a-2806-4a74-bd75-af12f5cca5ad" containerName="payload-processing" Oct 31 13:03:30.673404 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.673335 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a44908b-d8ff-45cd-b505-d19ce41fbba9" containerName="payload-pre-processing" Oct 31 13:03:30.673404 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.673352 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c35a13a-2806-4a74-bd75-af12f5cca5ad" containerName="payload-processing" Oct 31 13:03:30.677335 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.677316 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:03:30.684716 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.684691 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkdnn\" (UniqueName: \"kubernetes.io/projected/5a44908b-d8ff-45cd-b505-d19ce41fbba9-kube-api-access-mkdnn\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:30.684832 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.684715 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdsfx\" (UniqueName: \"kubernetes.io/projected/6c35a13a-2806-4a74-bd75-af12f5cca5ad-kube-api-access-xdsfx\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:30.689806 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.689786 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-c347ff-openshift-default-dockercfg-f4pkl\"" Oct 31 13:03:30.707808 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.707744 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx"] Oct 31 13:03:30.785946 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.785913 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-workload-socket\") pod \"e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx\" (UID: \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\") " pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:03:30.786136 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.785965 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-credential-socket\") pod \"e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx\" (UID: \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\") " pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:03:30.786136 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.786124 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-workload-certs\") pod \"e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx\" (UID: \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\") " pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:03:30.786256 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.786231 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-istio-podinfo\") pod \"e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx\" (UID: \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\") " pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:03:30.786304 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.786289 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-istio-token\") pod \"e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx\" (UID: \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\") " pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:03:30.786352 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.786329 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-istio-envoy\") pod \"e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx\" (UID: \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\") " pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:03:30.786352 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.786346 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-istio-data\") pod \"e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx\" (UID: \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\") " pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:03:30.786489 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.786464 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt5mc\" (UniqueName: \"kubernetes.io/projected/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-kube-api-access-lt5mc\") pod \"e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx\" (UID: \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\") " pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:03:30.786553 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.786537 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-istiod-ca-cert\") pod \"e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx\" (UID: \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\") " pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:03:30.846645 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.846471 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:30.881555 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.881521 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-94c96c5b7-54j27"] Oct 31 13:03:30.887528 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.887468 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-94c96c5b7-54j27"] Oct 31 13:03:30.888170 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.888146 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-workload-socket\") pod \"e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx\" (UID: \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\") " pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:03:30.888251 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.888195 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-credential-socket\") pod \"e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx\" (UID: \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\") " pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:03:30.888251 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.888249 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-workload-certs\") pod \"e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx\" (UID: \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\") " pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:03:30.888364 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.888329 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-istio-podinfo\") pod \"e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx\" (UID: \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\") " pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:03:30.888420 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.888381 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-istio-token\") pod \"e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx\" (UID: \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\") " pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:03:30.888487 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.888429 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-istio-envoy\") pod \"e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx\" (UID: \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\") " pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:03:30.888569 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.888551 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-workload-socket\") pod \"e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx\" (UID: \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\") " pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:03:30.888627 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.888551 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-istio-data\") pod \"e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx\" (UID: \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\") " pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:03:30.888682 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.888625 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-credential-socket\") pod \"e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx\" (UID: \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\") " pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:03:30.888682 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.888653 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt5mc\" (UniqueName: \"kubernetes.io/projected/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-kube-api-access-lt5mc\") pod \"e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx\" (UID: \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\") " pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:03:30.888790 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.888706 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-istiod-ca-cert\") pod \"e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx\" (UID: \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\") " pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:03:30.888790 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.888745 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-workload-certs\") pod \"e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx\" (UID: \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\") " pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:03:30.888902 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.888881 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-istio-data\") pod \"e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx\" (UID: \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\") " pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:03:30.889508 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.889476 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-istiod-ca-cert\") pod \"e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx\" (UID: \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\") " pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:03:30.890786 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.890761 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-istio-envoy\") pod \"e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx\" (UID: \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\") " pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:03:30.891167 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.891150 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-istio-podinfo\") pod \"e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx\" (UID: \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\") " pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:03:30.906232 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.906210 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-istio-token\") pod \"e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx\" (UID: \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\") " pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:03:30.908903 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.908871 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt5mc\" (UniqueName: \"kubernetes.io/projected/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-kube-api-access-lt5mc\") pod \"e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx\" (UID: \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\") " pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:03:30.909674 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.909655 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-688b5d7bc-vkv7k"] Oct 31 13:03:30.931257 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.921721 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-688b5d7bc-vkv7k"] Oct 31 13:03:31.006094 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:30.991966 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:03:31.111386 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:31.111353 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:31.611421 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:31.611238 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:31.800067 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:31.797145 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a44908b-d8ff-45cd-b505-d19ce41fbba9" path="/var/lib/kubelet/pods/5a44908b-d8ff-45cd-b505-d19ce41fbba9/volumes" Oct 31 13:03:31.800067 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:31.797734 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c35a13a-2806-4a74-bd75-af12f5cca5ad" path="/var/lib/kubelet/pods/6c35a13a-2806-4a74-bd75-af12f5cca5ad/volumes" Oct 31 13:03:31.856156 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:03:31.856117 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0e2ba69_d8b6_4534_8113_5d4c0bfea735.slice/crio-938ef084b8d9e10e20022fe0210e06541cb04bbf44a567bd2bb95b21cee6b92a WatchSource:0}: Error finding container 938ef084b8d9e10e20022fe0210e06541cb04bbf44a567bd2bb95b21cee6b92a: Status 404 returned error can't find the container with id 938ef084b8d9e10e20022fe0210e06541cb04bbf44a567bd2bb95b21cee6b92a Oct 31 13:03:31.857479 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:31.857432 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx"] Oct 31 13:03:31.859061 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:31.859025 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:03:31.859176 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:31.859107 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:03:31.859176 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:31.859152 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:03:32.617568 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:32.617533 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" event={"ID":"c0e2ba69-d8b6-4534-8113-5d4c0bfea735","Type":"ContainerStarted","Data":"12a87e9dff5234ec8a142efacca456a19992230e502dc895e65c3cdbbf2f16b9"} Oct 31 13:03:32.617751 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:32.617587 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" event={"ID":"c0e2ba69-d8b6-4534-8113-5d4c0bfea735","Type":"ContainerStarted","Data":"938ef084b8d9e10e20022fe0210e06541cb04bbf44a567bd2bb95b21cee6b92a"} Oct 31 13:03:32.683005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:32.682035 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" podStartSLOduration=2.661962821 podStartE2EDuration="2.661962821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:32.661346757 +0000 UTC m=+1859.144469659" watchObservedRunningTime="2026-10-31 13:03:32.661962821 +0000 UTC m=+1859.145085724" Oct 31 13:03:32.914059 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:32.910810 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:33.002997 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:33.002950 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:03:33.010296 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:33.010267 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:03:33.423668 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:33.422107 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:33.837336 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:33.837261 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:33.839049 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:33.839022 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-b64d65f5b-rcxdp"] Oct 31 13:03:34.253872 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:34.253837 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-b64d65f5b-rcxdp"] Oct 31 13:03:34.254464 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:34.254444 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-5d54db4b4c-qxx8v"] Oct 31 13:03:34.271144 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:34.271118 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-b64d65f5b-rcxdp" Oct 31 13:03:34.274860 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:34.274838 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-5d54db4b4c-qxx8v" Oct 31 13:03:34.309003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:34.279803 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f4cdd9-dockercfg-g5xdw\"" Oct 31 13:03:34.309003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:34.304540 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:03:34.309003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:34.305491 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f4cdd9\"" Oct 31 13:03:34.311877 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:34.311847 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-5d54db4b4c-qxx8v"] Oct 31 13:03:34.316747 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:34.316727 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:03:34.373697 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:34.370516 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:34.410175 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:34.407271 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8fbq\" (UniqueName: \"kubernetes.io/projected/9574b8ac-9faf-47af-a648-8ebf24027496-kube-api-access-d8fbq\") pod \"payload-processing-e2e-ipp-b-f4cdd9-b64d65f5b-rcxdp\" (UID: \"9574b8ac-9faf-47af-a648-8ebf24027496\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-b64d65f5b-rcxdp" Oct 31 13:03:34.410175 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:34.407395 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9574b8ac-9faf-47af-a648-8ebf24027496-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f4cdd9-b64d65f5b-rcxdp\" (UID: \"9574b8ac-9faf-47af-a648-8ebf24027496\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-b64d65f5b-rcxdp" Oct 31 13:03:34.410175 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:34.407439 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl764\" (UniqueName: \"kubernetes.io/projected/59760ecf-ed86-48bc-92ab-de8527d1fff7-kube-api-access-cl764\") pod \"payload-pre-processing-e2e-ipp-b-f4cdd9-5d54db4b4c-qxx8v\" (UID: \"59760ecf-ed86-48bc-92ab-de8527d1fff7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-5d54db4b4c-qxx8v" Oct 31 13:03:34.410175 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:34.407525 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59760ecf-ed86-48bc-92ab-de8527d1fff7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f4cdd9-5d54db4b4c-qxx8v\" (UID: \"59760ecf-ed86-48bc-92ab-de8527d1fff7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-5d54db4b4c-qxx8v" Oct 31 13:03:34.509238 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:34.509152 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8fbq\" (UniqueName: \"kubernetes.io/projected/9574b8ac-9faf-47af-a648-8ebf24027496-kube-api-access-d8fbq\") pod \"payload-processing-e2e-ipp-b-f4cdd9-b64d65f5b-rcxdp\" (UID: \"9574b8ac-9faf-47af-a648-8ebf24027496\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-b64d65f5b-rcxdp" Oct 31 13:03:34.509377 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:34.509271 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9574b8ac-9faf-47af-a648-8ebf24027496-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f4cdd9-b64d65f5b-rcxdp\" (UID: \"9574b8ac-9faf-47af-a648-8ebf24027496\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-b64d65f5b-rcxdp" Oct 31 13:03:34.509377 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:34.509311 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl764\" (UniqueName: \"kubernetes.io/projected/59760ecf-ed86-48bc-92ab-de8527d1fff7-kube-api-access-cl764\") pod \"payload-pre-processing-e2e-ipp-b-f4cdd9-5d54db4b4c-qxx8v\" (UID: \"59760ecf-ed86-48bc-92ab-de8527d1fff7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-5d54db4b4c-qxx8v" Oct 31 13:03:34.509466 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:34.509390 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59760ecf-ed86-48bc-92ab-de8527d1fff7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f4cdd9-5d54db4b4c-qxx8v\" (UID: \"59760ecf-ed86-48bc-92ab-de8527d1fff7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-5d54db4b4c-qxx8v" Oct 31 13:03:34.510262 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:34.510238 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9574b8ac-9faf-47af-a648-8ebf24027496-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f4cdd9-b64d65f5b-rcxdp\" (UID: \"9574b8ac-9faf-47af-a648-8ebf24027496\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-b64d65f5b-rcxdp" Oct 31 13:03:34.510518 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:34.510493 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59760ecf-ed86-48bc-92ab-de8527d1fff7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f4cdd9-5d54db4b4c-qxx8v\" (UID: \"59760ecf-ed86-48bc-92ab-de8527d1fff7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-5d54db4b4c-qxx8v" Oct 31 13:03:34.531826 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:34.531804 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl764\" (UniqueName: \"kubernetes.io/projected/59760ecf-ed86-48bc-92ab-de8527d1fff7-kube-api-access-cl764\") pod \"payload-pre-processing-e2e-ipp-b-f4cdd9-5d54db4b4c-qxx8v\" (UID: \"59760ecf-ed86-48bc-92ab-de8527d1fff7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-5d54db4b4c-qxx8v" Oct 31 13:03:34.537356 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:34.537338 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8fbq\" (UniqueName: \"kubernetes.io/projected/9574b8ac-9faf-47af-a648-8ebf24027496-kube-api-access-d8fbq\") pod \"payload-processing-e2e-ipp-b-f4cdd9-b64d65f5b-rcxdp\" (UID: \"9574b8ac-9faf-47af-a648-8ebf24027496\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-b64d65f5b-rcxdp" Oct 31 13:03:34.634623 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:34.633435 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-b64d65f5b-rcxdp" Oct 31 13:03:34.710050 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:34.710015 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-5d54db4b4c-qxx8v" Oct 31 13:03:34.830466 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:34.830342 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:35.109199 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:35.109165 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:35.567244 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:35.565825 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:35.939934 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:35.939835 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:36.216202 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:03:36.216149 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9574b8ac_9faf_47af_a648_8ebf24027496.slice/crio-ae790791b462febe561a396cb636674c131642b592386015688a92e36623f1ec WatchSource:0}: Error finding container ae790791b462febe561a396cb636674c131642b592386015688a92e36623f1ec: Status 404 returned error can't find the container with id ae790791b462febe561a396cb636674c131642b592386015688a92e36623f1ec Oct 31 13:03:36.224884 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:36.224856 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq"] Oct 31 13:03:36.340014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:36.334609 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-b64d65f5b-rcxdp"] Oct 31 13:03:36.340014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:36.334740 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:03:36.340014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:36.336972 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq"] Oct 31 13:03:36.340014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:36.339465 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:36.425621 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:36.384090 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-fa2079-openshift-default-dockercfg-rb75b\"" Oct 31 13:03:36.431052 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:36.428826 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64dbdf9c-9faf-48af-be69-c820c44aac9f-istio-envoy\") pod \"e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq\" (UID: \"64dbdf9c-9faf-48af-be69-c820c44aac9f\") " pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:03:36.431052 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:36.428912 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64dbdf9c-9faf-48af-be69-c820c44aac9f-istio-data\") pod \"e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq\" (UID: \"64dbdf9c-9faf-48af-be69-c820c44aac9f\") " pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:03:36.431052 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:36.428964 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64dbdf9c-9faf-48af-be69-c820c44aac9f-istiod-ca-cert\") pod \"e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq\" (UID: \"64dbdf9c-9faf-48af-be69-c820c44aac9f\") " pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:03:36.474499 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:36.473127 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64dbdf9c-9faf-48af-be69-c820c44aac9f-credential-socket\") pod \"e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq\" (UID: \"64dbdf9c-9faf-48af-be69-c820c44aac9f\") " pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:03:36.474499 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:36.473173 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64dbdf9c-9faf-48af-be69-c820c44aac9f-workload-certs\") pod \"e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq\" (UID: \"64dbdf9c-9faf-48af-be69-c820c44aac9f\") " pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:03:36.474499 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:36.473211 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64dbdf9c-9faf-48af-be69-c820c44aac9f-istio-token\") pod \"e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq\" (UID: \"64dbdf9c-9faf-48af-be69-c820c44aac9f\") " pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:03:36.474499 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:36.473344 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64dbdf9c-9faf-48af-be69-c820c44aac9f-istio-podinfo\") pod \"e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq\" (UID: \"64dbdf9c-9faf-48af-be69-c820c44aac9f\") " pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:03:36.474499 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:36.473396 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64dbdf9c-9faf-48af-be69-c820c44aac9f-workload-socket\") pod \"e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq\" (UID: \"64dbdf9c-9faf-48af-be69-c820c44aac9f\") " pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:03:36.474499 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:36.473422 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c5zn\" (UniqueName: \"kubernetes.io/projected/64dbdf9c-9faf-48af-be69-c820c44aac9f-kube-api-access-8c5zn\") pod \"e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq\" (UID: \"64dbdf9c-9faf-48af-be69-c820c44aac9f\") " pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:03:36.637131 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:36.596057 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-5d54db4b4c-qxx8v"] Oct 31 13:03:36.637131 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:36.600265 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64dbdf9c-9faf-48af-be69-c820c44aac9f-istio-podinfo\") pod \"e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq\" (UID: \"64dbdf9c-9faf-48af-be69-c820c44aac9f\") " pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:03:36.637131 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:36.600337 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64dbdf9c-9faf-48af-be69-c820c44aac9f-workload-socket\") pod \"e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq\" (UID: \"64dbdf9c-9faf-48af-be69-c820c44aac9f\") " pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:03:36.637131 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:36.600366 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c5zn\" (UniqueName: \"kubernetes.io/projected/64dbdf9c-9faf-48af-be69-c820c44aac9f-kube-api-access-8c5zn\") pod \"e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq\" (UID: \"64dbdf9c-9faf-48af-be69-c820c44aac9f\") " pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:03:36.637131 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:36.600543 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64dbdf9c-9faf-48af-be69-c820c44aac9f-istio-envoy\") pod \"e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq\" (UID: \"64dbdf9c-9faf-48af-be69-c820c44aac9f\") " pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:03:36.637131 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:36.600608 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64dbdf9c-9faf-48af-be69-c820c44aac9f-istio-data\") pod \"e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq\" (UID: \"64dbdf9c-9faf-48af-be69-c820c44aac9f\") " pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:03:36.637131 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:36.600668 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64dbdf9c-9faf-48af-be69-c820c44aac9f-istiod-ca-cert\") pod \"e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq\" (UID: \"64dbdf9c-9faf-48af-be69-c820c44aac9f\") " pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:03:36.637131 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:36.600740 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64dbdf9c-9faf-48af-be69-c820c44aac9f-credential-socket\") pod \"e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq\" (UID: \"64dbdf9c-9faf-48af-be69-c820c44aac9f\") " pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:03:36.637131 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:36.600763 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64dbdf9c-9faf-48af-be69-c820c44aac9f-workload-certs\") pod \"e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq\" (UID: \"64dbdf9c-9faf-48af-be69-c820c44aac9f\") " pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:03:36.637131 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:36.600796 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64dbdf9c-9faf-48af-be69-c820c44aac9f-istio-token\") pod \"e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq\" (UID: \"64dbdf9c-9faf-48af-be69-c820c44aac9f\") " pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:03:36.637131 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:36.621549 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64dbdf9c-9faf-48af-be69-c820c44aac9f-istio-data\") pod \"e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq\" (UID: \"64dbdf9c-9faf-48af-be69-c820c44aac9f\") " pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:03:36.637131 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:36.621866 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64dbdf9c-9faf-48af-be69-c820c44aac9f-workload-socket\") pod \"e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq\" (UID: \"64dbdf9c-9faf-48af-be69-c820c44aac9f\") " pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:03:36.703661 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:36.647887 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64dbdf9c-9faf-48af-be69-c820c44aac9f-credential-socket\") pod \"e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq\" (UID: \"64dbdf9c-9faf-48af-be69-c820c44aac9f\") " pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:03:36.703661 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:36.648564 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64dbdf9c-9faf-48af-be69-c820c44aac9f-istiod-ca-cert\") pod \"e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq\" (UID: \"64dbdf9c-9faf-48af-be69-c820c44aac9f\") " pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:03:36.703661 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:36.648846 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64dbdf9c-9faf-48af-be69-c820c44aac9f-workload-certs\") pod \"e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq\" (UID: \"64dbdf9c-9faf-48af-be69-c820c44aac9f\") " pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:03:36.718942 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:36.714063 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64dbdf9c-9faf-48af-be69-c820c44aac9f-istio-envoy\") pod \"e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq\" (UID: \"64dbdf9c-9faf-48af-be69-c820c44aac9f\") " pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:03:36.759811 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:36.738656 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:37.157060 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:37.150287 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-b64d65f5b-rcxdp" event={"ID":"9574b8ac-9faf-47af-a648-8ebf24027496","Type":"ContainerStarted","Data":"ae790791b462febe561a396cb636674c131642b592386015688a92e36623f1ec"} Oct 31 13:03:37.174002 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:37.164663 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:37.339814 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:37.339351 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64dbdf9c-9faf-48af-be69-c820c44aac9f-istio-token\") pod \"e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq\" (UID: \"64dbdf9c-9faf-48af-be69-c820c44aac9f\") " pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:03:37.340140 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:37.340112 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c5zn\" (UniqueName: \"kubernetes.io/projected/64dbdf9c-9faf-48af-be69-c820c44aac9f-kube-api-access-8c5zn\") pod \"e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq\" (UID: \"64dbdf9c-9faf-48af-be69-c820c44aac9f\") " pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:03:37.432131 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:37.428346 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64dbdf9c-9faf-48af-be69-c820c44aac9f-istio-podinfo\") pod \"e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq\" (UID: \"64dbdf9c-9faf-48af-be69-c820c44aac9f\") " pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:03:37.693023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:37.680756 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:03:37.891126 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:37.875032 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:38.271933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:38.271895 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-b64d65f5b-rcxdp" event={"ID":"9574b8ac-9faf-47af-a648-8ebf24027496","Type":"ContainerStarted","Data":"b60859163da08a33fe6f623b8431143302aa1f7dcc72ca9053d27bc00218bc77"} Oct 31 13:03:38.273265 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:38.273241 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-b64d65f5b-rcxdp" Oct 31 13:03:38.276160 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:38.276144 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-5d54db4b4c-qxx8v" Oct 31 13:03:38.276307 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:38.276292 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-5d54db4b4c-qxx8v" event={"ID":"59760ecf-ed86-48bc-92ab-de8527d1fff7","Type":"ContainerStarted","Data":"71ccff3c9184af8f42f8548b7a5dc7a47d26df224fd68e99c0e833032276be26"} Oct 31 13:03:38.276413 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:38.276398 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-5d54db4b4c-qxx8v" event={"ID":"59760ecf-ed86-48bc-92ab-de8527d1fff7","Type":"ContainerStarted","Data":"65af52e935ecb81824670ece334dca2143e36774b463a97d4eff4e2ceccc3594"} Oct 31 13:03:38.332868 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:38.332803 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-b64d65f5b-rcxdp" podStartSLOduration=5.332781846 podStartE2EDuration="5.332781846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:38.307168024 +0000 UTC m=+1864.790290926" watchObservedRunningTime="2026-10-31 13:03:38.332781846 +0000 UTC m=+1864.815904750" Oct 31 13:03:38.354003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:38.346261 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-5d54db4b4c-qxx8v" podStartSLOduration=5.346240085 podStartE2EDuration="5.346240085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:38.344372623 +0000 UTC m=+1864.827495526" watchObservedRunningTime="2026-10-31 13:03:38.346240085 +0000 UTC m=+1864.829362987" Oct 31 13:03:39.094835 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:39.094729 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq"] Oct 31 13:03:39.112274 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:39.112237 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:39.131478 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:39.131428 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:03:39.136335 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:39.136300 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:03:39.136558 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:39.136531 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:03:39.318131 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:39.318089 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" event={"ID":"64dbdf9c-9faf-48af-be69-c820c44aac9f","Type":"ContainerStarted","Data":"f30f7193880df430c8e0c8f0507cae8f51167b18bad7293c35c012d19041a949"} Oct 31 13:03:39.563004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:39.562894 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:40.330273 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:40.330174 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" event={"ID":"64dbdf9c-9faf-48af-be69-c820c44aac9f","Type":"ContainerStarted","Data":"b1523f218f8367168b69655ff93caea63b9eab3413d6a6a37850a3e8f329c552"} Oct 31 13:03:40.416004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:40.409075 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" podStartSLOduration=5.40905159 podStartE2EDuration="5.40905159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:40.393115893 +0000 UTC m=+1866.876238796" watchObservedRunningTime="2026-10-31 13:03:40.40905159 +0000 UTC m=+1866.892174492" Oct 31 13:03:40.422003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:40.419204 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:40.684306 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:40.682129 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:03:40.699079 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:40.694070 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:03:41.292350 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:41.267945 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:41.381067 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:41.380165 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:03:41.382739 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:41.382524 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:03:41.479803 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:41.479731 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:42.391312 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:42.389868 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:42.921900 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:42.921859 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:43.310123 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:43.310085 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:43.666524 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:43.666427 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:44.345122 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:44.345082 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:44.642075 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:44.638237 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:45.287717 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:45.287683 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:46.245004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:46.241721 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:46.403169 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:46.403135 6841 patch_prober.go:29] interesting pod/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": read tcp 10.134.0.2:49066->10.134.0.130:15021: read: connection reset by peer" start-of-body= Oct 31 13:03:46.403662 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:46.403206 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" podUID="eaeed36d-9d97-46f0-93f7-955981f85ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": read tcp 10.134.0.2:49066->10.134.0.130:15021: read: connection reset by peer" Oct 31 13:03:46.658432 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:46.658395 6841 patch_prober.go:29] interesting pod/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 31 13:03:46.658593 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:46.658446 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" podUID="07b82d57-b897-421f-b06a-85d23a3d3f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 31 13:03:46.960054 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:46.960014 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg"] Oct 31 13:03:46.978652 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:46.978629 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg" Oct 31 13:03:46.988885 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:46.988863 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-204e6c-serving-cert\"" Oct 31 13:03:46.989943 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:46.989875 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg"] Oct 31 13:03:47.094844 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:47.094804 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c17825-4128-47f6-9fe6-b9cdda220ecc-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg\" (UID: \"c9c17825-4128-47f6-9fe6-b9cdda220ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg" Oct 31 13:03:47.095056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:47.094937 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c17825-4128-47f6-9fe6-b9cdda220ecc-maas-api-tls\") pod \"maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg\" (UID: \"c9c17825-4128-47f6-9fe6-b9cdda220ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg" Oct 31 13:03:47.095056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:47.094967 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q54b\" (UniqueName: \"kubernetes.io/projected/c9c17825-4128-47f6-9fe6-b9cdda220ecc-kube-api-access-5q54b\") pod \"maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg\" (UID: \"c9c17825-4128-47f6-9fe6-b9cdda220ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg" Oct 31 13:03:47.095273 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:47.095251 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9c17825-4128-47f6-9fe6-b9cdda220ecc-metrics-certs\") pod \"maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg\" (UID: \"c9c17825-4128-47f6-9fe6-b9cdda220ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg" Oct 31 13:03:47.198005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:47.197805 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c17825-4128-47f6-9fe6-b9cdda220ecc-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg\" (UID: \"c9c17825-4128-47f6-9fe6-b9cdda220ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg" Oct 31 13:03:47.202006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:47.198662 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:47.215057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:47.205903 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c17825-4128-47f6-9fe6-b9cdda220ecc-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg\" (UID: \"c9c17825-4128-47f6-9fe6-b9cdda220ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg" Oct 31 13:03:47.215057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:47.206348 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c17825-4128-47f6-9fe6-b9cdda220ecc-maas-api-tls\") pod \"maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg\" (UID: \"c9c17825-4128-47f6-9fe6-b9cdda220ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg" Oct 31 13:03:47.215057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:47.206427 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q54b\" (UniqueName: \"kubernetes.io/projected/c9c17825-4128-47f6-9fe6-b9cdda220ecc-kube-api-access-5q54b\") pod \"maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg\" (UID: \"c9c17825-4128-47f6-9fe6-b9cdda220ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg" Oct 31 13:03:47.215057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:47.212202 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9c17825-4128-47f6-9fe6-b9cdda220ecc-metrics-certs\") pod \"maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg\" (UID: \"c9c17825-4128-47f6-9fe6-b9cdda220ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg" Oct 31 13:03:47.234719 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:47.234673 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9c17825-4128-47f6-9fe6-b9cdda220ecc-metrics-certs\") pod \"maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg\" (UID: \"c9c17825-4128-47f6-9fe6-b9cdda220ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg" Oct 31 13:03:47.242543 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:47.242512 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q54b\" (UniqueName: \"kubernetes.io/projected/c9c17825-4128-47f6-9fe6-b9cdda220ecc-kube-api-access-5q54b\") pod \"maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg\" (UID: \"c9c17825-4128-47f6-9fe6-b9cdda220ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg" Oct 31 13:03:47.243140 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:47.243089 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c17825-4128-47f6-9fe6-b9cdda220ecc-maas-api-tls\") pod \"maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg\" (UID: \"c9c17825-4128-47f6-9fe6-b9cdda220ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg" Oct 31 13:03:47.317041 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:47.317006 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg" Oct 31 13:03:47.433629 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:47.433600 6841 generic.go:356] "Generic (PLEG): container finished" podID="07b82d57-b897-421f-b06a-85d23a3d3f3f" containerID="d8485f67ccfb9b121df99422d918b740b14c9068fbc5336a6807e756fe74fa29" exitCode=0 Oct 31 13:03:47.434102 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:47.433675 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" event={"ID":"07b82d57-b897-421f-b06a-85d23a3d3f3f","Type":"ContainerDied","Data":"d8485f67ccfb9b121df99422d918b740b14c9068fbc5336a6807e756fe74fa29"} Oct 31 13:03:47.434257 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:47.434238 6841 scope.go:122] "RemoveContainer" containerID="d8485f67ccfb9b121df99422d918b740b14c9068fbc5336a6807e756fe74fa29" Oct 31 13:03:47.435478 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:47.435455 6841 generic.go:356] "Generic (PLEG): container finished" podID="916a1b9e-516f-4cde-955a-71a255d469a4" containerID="054b0b9b69e2378d9f4cb1886539b7e4433b732d211813368c9e24f47fb8874d" exitCode=0 Oct 31 13:03:47.435601 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:47.435520 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" event={"ID":"916a1b9e-516f-4cde-955a-71a255d469a4","Type":"ContainerDied","Data":"054b0b9b69e2378d9f4cb1886539b7e4433b732d211813368c9e24f47fb8874d"} Oct 31 13:03:47.435913 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:47.435893 6841 scope.go:122] "RemoveContainer" containerID="054b0b9b69e2378d9f4cb1886539b7e4433b732d211813368c9e24f47fb8874d" Oct 31 13:03:47.435913 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:47.435894 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:03:47.436426 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:47.435975 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:03:47.436426 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:47.436062 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:03:47.437488 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:47.437467 6841 generic.go:356] "Generic (PLEG): container finished" podID="eaeed36d-9d97-46f0-93f7-955981f85ede" containerID="6eb378320491bf178a5472201b386cb6791e8a9f90659ef35458d9cb320bf8eb" exitCode=0 Oct 31 13:03:47.437605 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:47.437525 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" event={"ID":"eaeed36d-9d97-46f0-93f7-955981f85ede","Type":"ContainerDied","Data":"6eb378320491bf178a5472201b386cb6791e8a9f90659ef35458d9cb320bf8eb"} Oct 31 13:03:47.437680 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:47.437605 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:03:47.437680 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:47.437674 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:03:47.437761 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:47.437741 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:03:47.437905 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:47.437889 6841 scope.go:122] "RemoveContainer" containerID="6eb378320491bf178a5472201b386cb6791e8a9f90659ef35458d9cb320bf8eb" Oct 31 13:03:47.440224 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:47.440191 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:03:47.440300 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:47.440280 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:03:47.440409 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:47.440336 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:03:47.443776 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:47.443690 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:47.648264 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:47.644220 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg"] Oct 31 13:03:48.302055 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:48.294340 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:48.446869 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:48.446842 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg" event={"ID":"c9c17825-4128-47f6-9fe6-b9cdda220ecc","Type":"ContainerStarted","Data":"0b83df1ebb3f469f084d444ea9585825d25f61db8247cc4bf32b75c8ccbd7f6a"} Oct 31 13:03:48.449195 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:48.449170 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" event={"ID":"07b82d57-b897-421f-b06a-85d23a3d3f3f","Type":"ContainerStarted","Data":"129835ec5db53c5c28cafcbf434a8eff878cfcec213646347ac243f5e8ced7bb"} Oct 31 13:03:48.451589 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:48.451565 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" event={"ID":"916a1b9e-516f-4cde-955a-71a255d469a4","Type":"ContainerStarted","Data":"70aab0320cc4375536158511d28afcf417caf5ee0d964aa84c322af635cfdbe2"} Oct 31 13:03:48.453719 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:48.453699 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" event={"ID":"eaeed36d-9d97-46f0-93f7-955981f85ede","Type":"ContainerStarted","Data":"531dd6f1c4d6ac3c71766b073972cd52a90903f4c6ed8f6eaf81a717fd78a55a"} Oct 31 13:03:48.460327 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:48.460306 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:03:48.461150 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:48.461128 6841 patch_prober.go:29] interesting pod/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 31 13:03:48.461243 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:48.461167 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" podUID="07b82d57-b897-421f-b06a-85d23a3d3f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 31 13:03:49.022120 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:49.018585 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:03:49.036006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:49.032184 6841 patch_prober.go:29] interesting pod/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 31 13:03:49.036006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:49.032257 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" podUID="916a1b9e-516f-4cde-955a-71a255d469a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 31 13:03:49.321207 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:49.321128 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-5d54db4b4c-qxx8v" Oct 31 13:03:49.433277 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:49.433172 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:03:49.435833 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:49.435731 6841 patch_prober.go:29] interesting pod/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 31 13:03:49.436014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:49.435818 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" podUID="eaeed36d-9d97-46f0-93f7-955981f85ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 31 13:03:49.469100 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:49.468878 6841 patch_prober.go:29] interesting pod/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 31 13:03:49.469100 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:49.468942 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" podUID="07b82d57-b897-421f-b06a-85d23a3d3f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 31 13:03:49.819004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:49.790872 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:50.040396 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:50.040352 6841 patch_prober.go:29] interesting pod/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 31 13:03:50.040590 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:50.040412 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" podUID="916a1b9e-516f-4cde-955a-71a255d469a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 31 13:03:50.148058 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:50.064543 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:50.255006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:50.228499 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655"] Oct 31 13:03:50.395008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:50.391161 6841 patch_prober.go:29] interesting pod/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 31 13:03:50.395008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:50.391273 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" podUID="eaeed36d-9d97-46f0-93f7-955981f85ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 31 13:03:50.459201 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:50.445286 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg" Oct 31 13:03:50.459201 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:50.445347 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655"] Oct 31 13:03:50.459201 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:50.445389 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg" event={"ID":"c9c17825-4128-47f6-9fe6-b9cdda220ecc","Type":"ContainerStarted","Data":"796b34ca925a402a4a6712721b3e867f5f42c970e78c0f7bf418ceffa3f77618"} Oct 31 13:03:50.459201 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:50.445515 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655" Oct 31 13:03:50.469006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:50.466209 6841 patch_prober.go:29] interesting pod/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 31 13:03:50.469006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:50.466268 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" podUID="07b82d57-b897-421f-b06a-85d23a3d3f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 31 13:03:50.515005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:50.513389 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-c347ff-serving-cert\"" Oct 31 13:03:50.550009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:50.537823 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-b64d65f5b-rcxdp" Oct 31 13:03:50.550009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:50.544578 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg" podStartSLOduration=3.899198331 podStartE2EDuration="4.544556507s" totalImagesPullingTime="645.358176ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:50.519112435 +0000 UTC m=+1877.002235336" watchObservedRunningTime="2026-10-31 13:03:50.544556507 +0000 UTC m=+1877.027679410" Oct 31 13:03:50.626601 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:50.626542 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8lx9\" (UniqueName: \"kubernetes.io/projected/0934f64d-5734-4600-b7af-15025a56955e-kube-api-access-r8lx9\") pod \"maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655\" (UID: \"0934f64d-5734-4600-b7af-15025a56955e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655" Oct 31 13:03:50.637317 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:50.627111 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0934f64d-5734-4600-b7af-15025a56955e-metrics-certs\") pod \"maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655\" (UID: \"0934f64d-5734-4600-b7af-15025a56955e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655" Oct 31 13:03:50.637317 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:50.627274 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0934f64d-5734-4600-b7af-15025a56955e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655\" (UID: \"0934f64d-5734-4600-b7af-15025a56955e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655" Oct 31 13:03:50.637317 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:50.627535 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0934f64d-5734-4600-b7af-15025a56955e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655\" (UID: \"0934f64d-5734-4600-b7af-15025a56955e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655" Oct 31 13:03:50.729934 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:03:50.729851 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:03:50.738670 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:50.738628 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0934f64d-5734-4600-b7af-15025a56955e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655\" (UID: \"0934f64d-5734-4600-b7af-15025a56955e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655" Oct 31 13:03:50.738955 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:50.738929 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0934f64d-5734-4600-b7af-15025a56955e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655\" (UID: \"0934f64d-5734-4600-b7af-15025a56955e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655" Oct 31 13:03:50.739118 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:50.739098 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8lx9\" (UniqueName: \"kubernetes.io/projected/0934f64d-5734-4600-b7af-15025a56955e-kube-api-access-r8lx9\") pod \"maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655\" (UID: \"0934f64d-5734-4600-b7af-15025a56955e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655" Oct 31 13:03:50.739197 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:50.739138 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0934f64d-5734-4600-b7af-15025a56955e-metrics-certs\") pod \"maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655\" (UID: \"0934f64d-5734-4600-b7af-15025a56955e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655" Oct 31 13:03:50.740662 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:50.740631 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0934f64d-5734-4600-b7af-15025a56955e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655\" (UID: \"0934f64d-5734-4600-b7af-15025a56955e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655" Oct 31 13:03:50.854021 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:50.843330 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:50.856112 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:50.856067 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0934f64d-5734-4600-b7af-15025a56955e-metrics-certs\") pod \"maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655\" (UID: \"0934f64d-5734-4600-b7af-15025a56955e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655" Oct 31 13:03:50.857247 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:50.857211 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0934f64d-5734-4600-b7af-15025a56955e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655\" (UID: \"0934f64d-5734-4600-b7af-15025a56955e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655" Oct 31 13:03:51.023175 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:51.004716 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8lx9\" (UniqueName: \"kubernetes.io/projected/0934f64d-5734-4600-b7af-15025a56955e-kube-api-access-r8lx9\") pod \"maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655\" (UID: \"0934f64d-5734-4600-b7af-15025a56955e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655" Oct 31 13:03:51.047011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:51.034529 6841 patch_prober.go:29] interesting pod/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 31 13:03:51.047011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:51.034596 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" podUID="916a1b9e-516f-4cde-955a-71a255d469a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 31 13:03:51.225448 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:51.225413 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655" Oct 31 13:03:51.379066 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:51.373442 6841 patch_prober.go:29] interesting pod/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 31 13:03:51.379066 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:51.373522 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" podUID="eaeed36d-9d97-46f0-93f7-955981f85ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 31 13:03:51.463008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:51.462431 6841 patch_prober.go:29] interesting pod/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 31 13:03:51.463008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:51.462495 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" podUID="07b82d57-b897-421f-b06a-85d23a3d3f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 31 13:03:51.857427 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:51.857384 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:52.034117 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:52.033544 6841 patch_prober.go:29] interesting pod/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 31 13:03:52.034117 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:52.033594 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" podUID="916a1b9e-516f-4cde-955a-71a255d469a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 31 13:03:52.127212 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:52.127171 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655"] Oct 31 13:03:52.375609 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:52.375514 6841 patch_prober.go:29] interesting pod/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 31 13:03:52.375892 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:52.375848 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" podUID="eaeed36d-9d97-46f0-93f7-955981f85ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 31 13:03:52.423062 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:52.418467 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655" event={"ID":"0934f64d-5734-4600-b7af-15025a56955e","Type":"ContainerStarted","Data":"ff4539823f71b234776f33830f83b6ad073d29dd06b1a6c7ee8338b8874a5716"} Oct 31 13:03:52.464039 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:52.463373 6841 patch_prober.go:29] interesting pod/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 31 13:03:52.464039 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:52.463436 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" podUID="07b82d57-b897-421f-b06a-85d23a3d3f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 31 13:03:52.639057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:52.629260 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:53.028289 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:53.024664 6841 patch_prober.go:29] interesting pod/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 31 13:03:53.028289 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:53.024721 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" podUID="916a1b9e-516f-4cde-955a-71a255d469a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 31 13:03:53.436464 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:53.436261 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:03:53.445450 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:53.436390 6841 patch_prober.go:29] interesting pod/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 31 13:03:53.445966 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:53.445936 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" podUID="eaeed36d-9d97-46f0-93f7-955981f85ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 31 13:03:53.488044 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:53.479741 6841 patch_prober.go:29] interesting pod/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 31 13:03:53.488788 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:53.488734 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" podUID="07b82d57-b897-421f-b06a-85d23a3d3f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 31 13:03:53.659009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:53.578180 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:53.863013 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:53.858246 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:53.936923 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:53.936848 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655" podStartSLOduration=4.301882827 podStartE2EDuration="4.93682821s" totalImagesPullingTime="634.945383ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:53.878413465 +0000 UTC m=+1880.361536369" watchObservedRunningTime="2026-10-31 13:03:53.93682821 +0000 UTC m=+1880.419951112" Oct 31 13:03:53.966214 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:53.966175 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655" event={"ID":"0934f64d-5734-4600-b7af-15025a56955e","Type":"ContainerStarted","Data":"ca3ab5142234a3883affad6c506eef660b156657b538fb7a93bcefc266957a90"} Oct 31 13:03:54.017008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:54.009945 6841 patch_prober.go:29] interesting pod/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 31 13:03:54.017008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:54.010025 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" podUID="916a1b9e-516f-4cde-955a-71a255d469a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 31 13:03:54.376736 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:54.376698 6841 patch_prober.go:29] interesting pod/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 31 13:03:54.377360 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:54.377326 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" podUID="eaeed36d-9d97-46f0-93f7-955981f85ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 31 13:03:54.480425 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:54.480379 6841 patch_prober.go:29] interesting pod/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 31 13:03:54.480747 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:54.480697 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" podUID="07b82d57-b897-421f-b06a-85d23a3d3f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 31 13:03:54.670144 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:54.668548 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:54.789629 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:54.789594 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655" Oct 31 13:03:55.021128 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:55.021076 6841 patch_prober.go:29] interesting pod/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 31 13:03:55.021345 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:55.021138 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" podUID="916a1b9e-516f-4cde-955a-71a255d469a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 31 13:03:55.402060 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:55.401141 6841 patch_prober.go:29] interesting pod/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 31 13:03:55.402060 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:55.401224 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" podUID="eaeed36d-9d97-46f0-93f7-955981f85ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 31 13:03:55.461007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:55.460349 6841 patch_prober.go:29] interesting pod/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 31 13:03:55.461007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:55.460412 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" podUID="07b82d57-b897-421f-b06a-85d23a3d3f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 31 13:03:55.532498 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:55.532458 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:56.008227 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:56.008190 6841 patch_prober.go:29] interesting pod/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 31 13:03:56.008428 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:56.008245 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" podUID="916a1b9e-516f-4cde-955a-71a255d469a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 31 13:03:56.396734 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:56.396631 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg" Oct 31 13:03:56.419351 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:56.419319 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:03:56.420679 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:56.420642 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:03:56.422394 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:56.422366 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:03:56.430272 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:56.430243 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:56.466443 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:56.466409 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:03:56.467319 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:56.467296 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:03:56.468098 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:56.468073 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:03:57.225660 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:57.225627 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:03:57.229025 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:57.229002 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:03:57.462542 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:57.462039 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:57.710969 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:57.710928 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:58.326881 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:58.326846 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:58.973994 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:58.973949 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:59.621125 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:59.621089 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:03:59.971663 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:03:59.971368 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:00.257519 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:00.257432 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:00.502148 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:00.502098 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:00.867459 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:00.867427 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655" Oct 31 13:04:01.053007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:01.050532 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:01.820215 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:01.820170 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:02.622012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:02.620917 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:03.839838 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:03.839802 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:04.126453 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:04.126360 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:04.350963 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:04.350921 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:05.411174 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:05.411069 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:05.652014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:05.650957 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:06.575807 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:06.575767 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:07.616799 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:07.616764 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:08.330005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:08.324058 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:08.777744 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:08.777704 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:08.960711 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:08.960634 6841 generic.go:356] "Generic (PLEG): container finished" podID="e5ae8ae0-da38-4530-856c-c9e70d8484b7" containerID="26a00c479d203d15b2e834160249c8440bc0518335332f369fc2b2b03393cfd7" exitCode=0 Oct 31 13:04:08.960711 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:08.960686 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" event={"ID":"e5ae8ae0-da38-4530-856c-c9e70d8484b7","Type":"ContainerDied","Data":"26a00c479d203d15b2e834160249c8440bc0518335332f369fc2b2b03393cfd7"} Oct 31 13:04:08.961173 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:08.961151 6841 scope.go:122] "RemoveContainer" containerID="26a00c479d203d15b2e834160249c8440bc0518335332f369fc2b2b03393cfd7" Oct 31 13:04:08.963012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:08.962927 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:04:08.963106 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:08.963058 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:04:08.963106 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:08.963094 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:04:09.316662 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:09.316555 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:09.970270 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:09.970236 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" event={"ID":"e5ae8ae0-da38-4530-856c-c9e70d8484b7","Type":"ContainerStarted","Data":"d273b24c4de937a2a9366357c5348b01a69d2ac573d0d580a93217e0260c1118"} Oct 31 13:04:10.202517 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:10.202484 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:10.465008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:10.464328 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:04:10.467834 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:10.467802 6841 patch_prober.go:29] interesting pod/e2e-praxis-a-758f48-openshift-default-685758985c-9crds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 31 13:04:10.468024 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:10.467865 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" podUID="e5ae8ae0-da38-4530-856c-c9e70d8484b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 31 13:04:10.726359 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:10.726259 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:10.939057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:10.939016 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:11.202915 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:11.202879 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:11.406250 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:11.406220 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:04:11.466505 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:11.466406 6841 patch_prober.go:29] interesting pod/e2e-praxis-a-758f48-openshift-default-685758985c-9crds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 31 13:04:11.466505 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:11.466475 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" podUID="e5ae8ae0-da38-4530-856c-c9e70d8484b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 31 13:04:11.877706 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:11.877623 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:12.466481 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:12.466439 6841 patch_prober.go:29] interesting pod/e2e-praxis-a-758f48-openshift-default-685758985c-9crds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 31 13:04:12.466917 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:12.466524 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" podUID="e5ae8ae0-da38-4530-856c-c9e70d8484b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 31 13:04:12.632073 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:12.625710 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:13.424047 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:13.424013 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:13.464654 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:13.464614 6841 patch_prober.go:29] interesting pod/e2e-praxis-a-758f48-openshift-default-685758985c-9crds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 31 13:04:13.465077 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:13.465045 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" podUID="e5ae8ae0-da38-4530-856c-c9e70d8484b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 31 13:04:13.667717 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:13.667389 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:14.212007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:14.211121 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:14.462059 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:14.461431 6841 patch_prober.go:29] interesting pod/e2e-praxis-a-758f48-openshift-default-685758985c-9crds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 31 13:04:14.462059 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:14.461501 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" podUID="e5ae8ae0-da38-4530-856c-c9e70d8484b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 31 13:04:15.186006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:15.185565 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:15.466010 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:15.463695 6841 patch_prober.go:29] interesting pod/e2e-praxis-a-758f48-openshift-default-685758985c-9crds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 31 13:04:15.466010 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:15.463775 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" podUID="e5ae8ae0-da38-4530-856c-c9e70d8484b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 31 13:04:16.284798 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:16.284759 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:16.657066 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:16.641170 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:04:16.682006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:16.679297 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:04:17.129648 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:17.129610 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:17.823259 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:17.823225 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x"] Oct 31 13:04:17.831722 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:17.831693 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x"] Oct 31 13:04:17.831884 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:17.831804 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x" Oct 31 13:04:17.882011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:17.858273 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-e4420f-serving-cert\"" Oct 31 13:04:17.930675 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:17.930588 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8670d06e-4d46-4e8f-b4f3-532608114855-metrics-certs\") pod \"maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x\" (UID: \"8670d06e-4d46-4e8f-b4f3-532608114855\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x" Oct 31 13:04:17.931541 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:17.931508 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8670d06e-4d46-4e8f-b4f3-532608114855-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x\" (UID: \"8670d06e-4d46-4e8f-b4f3-532608114855\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x" Oct 31 13:04:17.931942 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:17.931772 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8670d06e-4d46-4e8f-b4f3-532608114855-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x\" (UID: \"8670d06e-4d46-4e8f-b4f3-532608114855\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x" Oct 31 13:04:17.932224 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:17.932207 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzj77\" (UniqueName: \"kubernetes.io/projected/8670d06e-4d46-4e8f-b4f3-532608114855-kube-api-access-hzj77\") pod \"maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x\" (UID: \"8670d06e-4d46-4e8f-b4f3-532608114855\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x" Oct 31 13:04:18.109934 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:18.109847 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:18.111484 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:18.110959 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8670d06e-4d46-4e8f-b4f3-532608114855-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x\" (UID: \"8670d06e-4d46-4e8f-b4f3-532608114855\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x" Oct 31 13:04:18.111821 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:18.111804 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzj77\" (UniqueName: \"kubernetes.io/projected/8670d06e-4d46-4e8f-b4f3-532608114855-kube-api-access-hzj77\") pod \"maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x\" (UID: \"8670d06e-4d46-4e8f-b4f3-532608114855\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x" Oct 31 13:04:18.116205 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:18.116176 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8670d06e-4d46-4e8f-b4f3-532608114855-metrics-certs\") pod \"maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x\" (UID: \"8670d06e-4d46-4e8f-b4f3-532608114855\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x" Oct 31 13:04:18.120072 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:18.120042 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8670d06e-4d46-4e8f-b4f3-532608114855-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x\" (UID: \"8670d06e-4d46-4e8f-b4f3-532608114855\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x" Oct 31 13:04:18.131053 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:18.121071 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8670d06e-4d46-4e8f-b4f3-532608114855-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x\" (UID: \"8670d06e-4d46-4e8f-b4f3-532608114855\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x" Oct 31 13:04:18.134348 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:18.125523 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8670d06e-4d46-4e8f-b4f3-532608114855-metrics-certs\") pod \"maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x\" (UID: \"8670d06e-4d46-4e8f-b4f3-532608114855\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x" Oct 31 13:04:18.134543 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:18.125967 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8670d06e-4d46-4e8f-b4f3-532608114855-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x\" (UID: \"8670d06e-4d46-4e8f-b4f3-532608114855\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x" Oct 31 13:04:18.134685 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:18.131367 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzj77\" (UniqueName: \"kubernetes.io/projected/8670d06e-4d46-4e8f-b4f3-532608114855-kube-api-access-hzj77\") pod \"maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x\" (UID: \"8670d06e-4d46-4e8f-b4f3-532608114855\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x" Oct 31 13:04:18.201649 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:18.200638 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x" Oct 31 13:04:18.319756 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:18.319629 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:18.653908 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:18.653870 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:18.836403 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:18.836361 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x"] Oct 31 13:04:19.006007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:18.998482 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:19.400009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:19.399369 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x" event={"ID":"8670d06e-4d46-4e8f-b4f3-532608114855","Type":"ContainerStarted","Data":"5fd0f2fce74b13bb59f264b984d3d5155e448d94d107cba817846c464765c25e"} Oct 31 13:04:20.039194 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:04:20.039160 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:04:20.234010 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:20.233385 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:20.421895 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:20.421818 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x" event={"ID":"8670d06e-4d46-4e8f-b4f3-532608114855","Type":"ContainerStarted","Data":"f1fb0cea8a622a3a3e8502f4ff53d0b82c874ab7fe3d5b6ed798c5c5d776dec7"} Oct 31 13:04:20.422995 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:20.422960 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x" Oct 31 13:04:20.483002 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:20.482776 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x" podStartSLOduration=3.025916536 podStartE2EDuration="3.482753916s" totalImagesPullingTime="456.83738ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:04:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:04:20.481762151 +0000 UTC m=+1906.964885052" watchObservedRunningTime="2026-10-31 13:04:20.482753916 +0000 UTC m=+1906.965876818" Oct 31 13:04:20.492742 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:20.490317 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:20.708003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:20.707752 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:21.202583 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:21.202547 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:21.451007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:21.437427 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:22.009030 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:21.997190 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:22.965382 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:22.965343 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:23.853655 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:23.853559 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:24.810933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:24.810893 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:25.236118 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:25.236084 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d"] Oct 31 13:04:25.242355 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:25.242315 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d" podUID="2aed2f8d-a0b0-4416-b07c-e73f310bfb0f" containerName="maas-api" containerID="cri-o://5af71d0cfa59e2e6b80d8ca0aa8233710a3b66d1d6c97af3b7634662170515f6" gracePeriod=30 Oct 31 13:04:25.680877 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:25.665967 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:25.724004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:25.723939 6841 generic.go:356] "Generic (PLEG): container finished" podID="2aed2f8d-a0b0-4416-b07c-e73f310bfb0f" containerID="5af71d0cfa59e2e6b80d8ca0aa8233710a3b66d1d6c97af3b7634662170515f6" exitCode=0 Oct 31 13:04:25.763032 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:25.762953 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d" event={"ID":"2aed2f8d-a0b0-4416-b07c-e73f310bfb0f","Type":"ContainerDied","Data":"5af71d0cfa59e2e6b80d8ca0aa8233710a3b66d1d6c97af3b7634662170515f6"} Oct 31 13:04:25.875722 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:25.875686 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:25.891066 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:25.891040 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d" Oct 31 13:04:25.990077 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:25.990045 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aed2f8d-a0b0-4416-b07c-e73f310bfb0f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aed2f8d-a0b0-4416-b07c-e73f310bfb0f-openshift-service-ca\") pod \"2aed2f8d-a0b0-4416-b07c-e73f310bfb0f\" (UID: \"2aed2f8d-a0b0-4416-b07c-e73f310bfb0f\") " Oct 31 13:04:25.990304 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:25.990167 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aed2f8d-a0b0-4416-b07c-e73f310bfb0f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aed2f8d-a0b0-4416-b07c-e73f310bfb0f-maas-api-tls\") pod \"2aed2f8d-a0b0-4416-b07c-e73f310bfb0f\" (UID: \"2aed2f8d-a0b0-4416-b07c-e73f310bfb0f\") " Oct 31 13:04:25.990470 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:25.990446 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aed2f8d-a0b0-4416-b07c-e73f310bfb0f-kube-api-access-n6z86\" (UniqueName: \"kubernetes.io/projected/2aed2f8d-a0b0-4416-b07c-e73f310bfb0f-kube-api-access-n6z86\") pod \"2aed2f8d-a0b0-4416-b07c-e73f310bfb0f\" (UID: \"2aed2f8d-a0b0-4416-b07c-e73f310bfb0f\") " Oct 31 13:04:25.990550 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:25.990518 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aed2f8d-a0b0-4416-b07c-e73f310bfb0f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aed2f8d-a0b0-4416-b07c-e73f310bfb0f-metrics-certs\") pod \"2aed2f8d-a0b0-4416-b07c-e73f310bfb0f\" (UID: \"2aed2f8d-a0b0-4416-b07c-e73f310bfb0f\") " Oct 31 13:04:25.991902 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:25.991641 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aed2f8d-a0b0-4416-b07c-e73f310bfb0f-openshift-service-ca" pod "2aed2f8d-a0b0-4416-b07c-e73f310bfb0f" (UID: "2aed2f8d-a0b0-4416-b07c-e73f310bfb0f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:04:25.996307 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:25.996280 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aed2f8d-a0b0-4416-b07c-e73f310bfb0f-maas-api-tls" pod "2aed2f8d-a0b0-4416-b07c-e73f310bfb0f" (UID: "2aed2f8d-a0b0-4416-b07c-e73f310bfb0f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:04:25.996691 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:25.996663 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aed2f8d-a0b0-4416-b07c-e73f310bfb0f-kube-api-access-n6z86" pod "2aed2f8d-a0b0-4416-b07c-e73f310bfb0f" (UID: "2aed2f8d-a0b0-4416-b07c-e73f310bfb0f"). InnerVolumeSpecName "kube-api-access-n6z86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:04:25.998436 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:25.998410 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aed2f8d-a0b0-4416-b07c-e73f310bfb0f-metrics-certs" pod "2aed2f8d-a0b0-4416-b07c-e73f310bfb0f" (UID: "2aed2f8d-a0b0-4416-b07c-e73f310bfb0f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:04:26.093031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:26.092995 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6z86\" (UniqueName: \"kubernetes.io/projected/2aed2f8d-a0b0-4416-b07c-e73f310bfb0f-kube-api-access-n6z86\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:26.093031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:26.093026 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aed2f8d-a0b0-4416-b07c-e73f310bfb0f-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:26.093031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:26.093036 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aed2f8d-a0b0-4416-b07c-e73f310bfb0f-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:26.093318 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:26.093045 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aed2f8d-a0b0-4416-b07c-e73f310bfb0f-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:26.763554 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:26.763078 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:26.970538 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:26.764466 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d" event={"ID":"2aed2f8d-a0b0-4416-b07c-e73f310bfb0f","Type":"ContainerDied","Data":"ed9b23e49294ec4ab3b28610d0c072face4cb6ab35a22507bc457e9519ccea7d"} Oct 31 13:04:26.970538 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:26.764536 6841 scope.go:122] "RemoveContainer" containerID="5af71d0cfa59e2e6b80d8ca0aa8233710a3b66d1d6c97af3b7634662170515f6" Oct 31 13:04:26.970538 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:26.764699 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d" Oct 31 13:04:26.970538 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:26.812375 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d"] Oct 31 13:04:26.970538 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:26.819153 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-49bffb-56c45bc885-4fk2d"] Oct 31 13:04:27.225394 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:27.225363 6841 patch_prober.go:29] interesting pod/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": read tcp 10.134.0.2:37752->10.134.0.143:15021: read: connection reset by peer" start-of-body= Oct 31 13:04:27.225594 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:27.225425 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" podUID="64dbdf9c-9faf-48af-be69-c820c44aac9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": read tcp 10.134.0.2:37752->10.134.0.143:15021: read: connection reset by peer" Oct 31 13:04:27.512077 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:27.512046 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x" Oct 31 13:04:27.604268 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:27.603904 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:27.645578 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:27.645534 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aed2f8d-a0b0-4416-b07c-e73f310bfb0f" path="/var/lib/kubelet/pods/2aed2f8d-a0b0-4416-b07c-e73f310bfb0f/volumes" Oct 31 13:04:27.773611 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:27.773530 6841 generic.go:356] "Generic (PLEG): container finished" podID="64dbdf9c-9faf-48af-be69-c820c44aac9f" containerID="b1523f218f8367168b69655ff93caea63b9eab3413d6a6a37850a3e8f329c552" exitCode=0 Oct 31 13:04:27.773788 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:27.773604 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" event={"ID":"64dbdf9c-9faf-48af-be69-c820c44aac9f","Type":"ContainerDied","Data":"b1523f218f8367168b69655ff93caea63b9eab3413d6a6a37850a3e8f329c552"} Oct 31 13:04:27.774294 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:27.774270 6841 scope.go:122] "RemoveContainer" containerID="b1523f218f8367168b69655ff93caea63b9eab3413d6a6a37850a3e8f329c552" Oct 31 13:04:27.776197 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:27.776168 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:04:27.776283 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:27.776239 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:04:27.776330 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:27.776293 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:04:27.776559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:27.776540 6841 generic.go:356] "Generic (PLEG): container finished" podID="c0e2ba69-d8b6-4534-8113-5d4c0bfea735" containerID="12a87e9dff5234ec8a142efacca456a19992230e502dc895e65c3cdbbf2f16b9" exitCode=0 Oct 31 13:04:27.776642 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:27.776616 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" event={"ID":"c0e2ba69-d8b6-4534-8113-5d4c0bfea735","Type":"ContainerDied","Data":"12a87e9dff5234ec8a142efacca456a19992230e502dc895e65c3cdbbf2f16b9"} Oct 31 13:04:27.777019 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:27.776977 6841 scope.go:122] "RemoveContainer" containerID="12a87e9dff5234ec8a142efacca456a19992230e502dc895e65c3cdbbf2f16b9" Oct 31 13:04:27.778534 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:27.778507 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:04:27.778615 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:27.778583 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:04:27.778656 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:27.778616 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:04:27.802275 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:27.802251 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-28e4df-76bf65bc65-72zf6"] Oct 31 13:04:27.839030 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:27.832005 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-28e4df-9ff5f7b6c-fvgw4"] Oct 31 13:04:28.012014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:28.007456 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:28.212009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:28.207249 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws"] Oct 31 13:04:28.212009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:28.208750 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws" podUID="f913d7e0-3523-4694-bbf3-294fcb0af9e3" containerName="maas-api" containerID="cri-o://5c44c243fddbd8c50c18f5a4bc5f8755c63b453c502c75a5250844a0fb50fe97" gracePeriod=30 Oct 31 13:04:28.289008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:28.279007 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:28.515240 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:28.515094 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:28.772058 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:28.771775 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:28.830777 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:28.830549 6841 generic.go:356] "Generic (PLEG): container finished" podID="f913d7e0-3523-4694-bbf3-294fcb0af9e3" containerID="5c44c243fddbd8c50c18f5a4bc5f8755c63b453c502c75a5250844a0fb50fe97" exitCode=0 Oct 31 13:04:28.830777 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:28.830650 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws" event={"ID":"f913d7e0-3523-4694-bbf3-294fcb0af9e3","Type":"ContainerDied","Data":"5c44c243fddbd8c50c18f5a4bc5f8755c63b453c502c75a5250844a0fb50fe97"} Oct 31 13:04:28.835176 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:28.833839 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" event={"ID":"64dbdf9c-9faf-48af-be69-c820c44aac9f","Type":"ContainerStarted","Data":"fc464ae15786892df57839c9378da40b0680a222196893ac50dfa544394b9a56"} Oct 31 13:04:28.842720 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:28.841964 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" event={"ID":"c0e2ba69-d8b6-4534-8113-5d4c0bfea735","Type":"ContainerStarted","Data":"c0950330646dbb70d32459735c2a3025ef02ace87c66094db6b17013c2f15941"} Oct 31 13:04:28.922642 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:28.922617 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws" Oct 31 13:04:28.994126 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:28.994094 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:04:28.994813 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:28.994788 6841 patch_prober.go:29] interesting pod/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 31 13:04:28.994924 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:28.994837 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" podUID="c0e2ba69-d8b6-4534-8113-5d4c0bfea735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 31 13:04:29.019143 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:29.019120 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f913d7e0-3523-4694-bbf3-294fcb0af9e3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f913d7e0-3523-4694-bbf3-294fcb0af9e3-openshift-service-ca\") pod \"f913d7e0-3523-4694-bbf3-294fcb0af9e3\" (UID: \"f913d7e0-3523-4694-bbf3-294fcb0af9e3\") " Oct 31 13:04:29.019461 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:29.019317 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f913d7e0-3523-4694-bbf3-294fcb0af9e3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f913d7e0-3523-4694-bbf3-294fcb0af9e3-maas-api-tls\") pod \"f913d7e0-3523-4694-bbf3-294fcb0af9e3\" (UID: \"f913d7e0-3523-4694-bbf3-294fcb0af9e3\") " Oct 31 13:04:29.019507 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:29.019476 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f913d7e0-3523-4694-bbf3-294fcb0af9e3-kube-api-access-nmfgx\" (UniqueName: \"kubernetes.io/projected/f913d7e0-3523-4694-bbf3-294fcb0af9e3-kube-api-access-nmfgx\") pod \"f913d7e0-3523-4694-bbf3-294fcb0af9e3\" (UID: \"f913d7e0-3523-4694-bbf3-294fcb0af9e3\") " Oct 31 13:04:29.019546 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:29.019507 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f913d7e0-3523-4694-bbf3-294fcb0af9e3-openshift-service-ca" pod "f913d7e0-3523-4694-bbf3-294fcb0af9e3" (UID: "f913d7e0-3523-4694-bbf3-294fcb0af9e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:04:29.019581 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:29.019559 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f913d7e0-3523-4694-bbf3-294fcb0af9e3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f913d7e0-3523-4694-bbf3-294fcb0af9e3-metrics-certs\") pod \"f913d7e0-3523-4694-bbf3-294fcb0af9e3\" (UID: \"f913d7e0-3523-4694-bbf3-294fcb0af9e3\") " Oct 31 13:04:29.020024 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:29.019978 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f913d7e0-3523-4694-bbf3-294fcb0af9e3-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:29.021441 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:29.021414 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f913d7e0-3523-4694-bbf3-294fcb0af9e3-maas-api-tls" pod "f913d7e0-3523-4694-bbf3-294fcb0af9e3" (UID: "f913d7e0-3523-4694-bbf3-294fcb0af9e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:04:29.021532 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:29.021491 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f913d7e0-3523-4694-bbf3-294fcb0af9e3-kube-api-access-nmfgx" pod "f913d7e0-3523-4694-bbf3-294fcb0af9e3" (UID: "f913d7e0-3523-4694-bbf3-294fcb0af9e3"). InnerVolumeSpecName "kube-api-access-nmfgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:04:29.021582 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:29.021566 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f913d7e0-3523-4694-bbf3-294fcb0af9e3-metrics-certs" pod "f913d7e0-3523-4694-bbf3-294fcb0af9e3" (UID: "f913d7e0-3523-4694-bbf3-294fcb0af9e3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:04:29.125006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:29.122377 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f913d7e0-3523-4694-bbf3-294fcb0af9e3-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:29.125006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:29.122423 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f913d7e0-3523-4694-bbf3-294fcb0af9e3-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:29.125006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:29.122438 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmfgx\" (UniqueName: \"kubernetes.io/projected/f913d7e0-3523-4694-bbf3-294fcb0af9e3-kube-api-access-nmfgx\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:29.145003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:29.143669 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:29.279009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:29.274900 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf"] Oct 31 13:04:29.279009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:29.275180 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" podUID="eaeed36d-9d97-46f0-93f7-955981f85ede" containerName="istio-proxy" containerID="cri-o://531dd6f1c4d6ac3c71766b073972cd52a90903f4c6ed8f6eaf81a717fd78a55a" gracePeriod=30 Oct 31 13:04:29.364060 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:29.362908 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:29.580048 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:29.574487 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:29.689189 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:29.689084 6841 patch_prober.go:29] interesting pod/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 31 13:04:29.689189 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:29.689138 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" podUID="64dbdf9c-9faf-48af-be69-c820c44aac9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 31 13:04:29.692010 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:29.690233 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:04:30.043004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:30.007261 6841 patch_prober.go:29] interesting pod/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 31 13:04:30.043004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:30.007512 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" podUID="c0e2ba69-d8b6-4534-8113-5d4c0bfea735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 31 13:04:30.068044 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:30.058532 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:30.106021 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:30.105605 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws" event={"ID":"f913d7e0-3523-4694-bbf3-294fcb0af9e3","Type":"ContainerDied","Data":"a0803de29bd793034fb4cf7763aa475f801e8206362defc7d7bf1cfe1dd583c1"} Oct 31 13:04:30.106021 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:30.105740 6841 scope.go:122] "RemoveContainer" containerID="5c44c243fddbd8c50c18f5a4bc5f8755c63b453c502c75a5250844a0fb50fe97" Oct 31 13:04:30.106298 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:30.106050 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws" Oct 31 13:04:30.133740 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:30.133709 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws"] Oct 31 13:04:30.140829 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:30.140792 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-28e4df-7bfdf7bfb7-l4wws"] Oct 31 13:04:30.691004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:30.684468 6841 patch_prober.go:29] interesting pod/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 31 13:04:30.691004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:30.684544 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" podUID="64dbdf9c-9faf-48af-be69-c820c44aac9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 31 13:04:30.748011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:30.746581 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:30.993840 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:30.993795 6841 patch_prober.go:29] interesting pod/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 31 13:04:30.994164 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:30.994116 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" podUID="c0e2ba69-d8b6-4534-8113-5d4c0bfea735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 31 13:04:31.684007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:31.683484 6841 patch_prober.go:29] interesting pod/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 31 13:04:31.684007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:31.683547 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" podUID="64dbdf9c-9faf-48af-be69-c820c44aac9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 31 13:04:31.698022 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:31.689050 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f913d7e0-3523-4694-bbf3-294fcb0af9e3" path="/var/lib/kubelet/pods/f913d7e0-3523-4694-bbf3-294fcb0af9e3/volumes" Oct 31 13:04:31.741006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:31.740629 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:31.998856 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:31.998824 6841 patch_prober.go:29] interesting pod/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 31 13:04:31.999171 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:31.999140 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" podUID="c0e2ba69-d8b6-4534-8113-5d4c0bfea735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 31 13:04:32.684826 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:32.684376 6841 patch_prober.go:29] interesting pod/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 31 13:04:32.684826 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:32.684444 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" podUID="64dbdf9c-9faf-48af-be69-c820c44aac9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 31 13:04:32.808008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:32.806050 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-28e4df-76bf65bc65-72zf6" podUID="30f31282-bb5f-426c-a167-af1e4bb39696" containerName="payload-processing" containerID="cri-o://10498e593c5142d90a58c68583185bca54c87509c2b007c2972768016b17e763" gracePeriod=25 Oct 31 13:04:32.840102 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:32.838582 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-28e4df-9ff5f7b6c-fvgw4" podUID="db09c0fc-2974-4e9a-8c0e-f9edd5528c49" containerName="payload-pre-processing" containerID="cri-o://5d1c7b6d321f4883fc295165bbfbd786f36a3c915d2442bda284a89cb2fd1bd4" gracePeriod=25 Oct 31 13:04:32.840102 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:32.839964 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:32.999538 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:32.999375 6841 patch_prober.go:29] interesting pod/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 31 13:04:32.999538 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:32.999430 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" podUID="c0e2ba69-d8b6-4534-8113-5d4c0bfea735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 31 13:04:33.201668 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:33.201538 6841 generic.go:356] "Generic (PLEG): container finished" podID="db09c0fc-2974-4e9a-8c0e-f9edd5528c49" containerID="5d1c7b6d321f4883fc295165bbfbd786f36a3c915d2442bda284a89cb2fd1bd4" exitCode=0 Oct 31 13:04:33.201668 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:33.201639 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-28e4df-9ff5f7b6c-fvgw4" event={"ID":"db09c0fc-2974-4e9a-8c0e-f9edd5528c49","Type":"ContainerDied","Data":"5d1c7b6d321f4883fc295165bbfbd786f36a3c915d2442bda284a89cb2fd1bd4"} Oct 31 13:04:33.204145 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:33.204117 6841 generic.go:356] "Generic (PLEG): container finished" podID="30f31282-bb5f-426c-a167-af1e4bb39696" containerID="10498e593c5142d90a58c68583185bca54c87509c2b007c2972768016b17e763" exitCode=0 Oct 31 13:04:33.204395 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:33.204375 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-28e4df-76bf65bc65-72zf6" event={"ID":"30f31282-bb5f-426c-a167-af1e4bb39696","Type":"ContainerDied","Data":"10498e593c5142d90a58c68583185bca54c87509c2b007c2972768016b17e763"} Oct 31 13:04:33.408952 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:33.408765 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-28e4df-9ff5f7b6c-fvgw4" Oct 31 13:04:33.414615 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:33.413777 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-28e4df-76bf65bc65-72zf6" Oct 31 13:04:33.561449 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:33.547595 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db09c0fc-2974-4e9a-8c0e-f9edd5528c49-kube-api-access-qhrxc\" (UniqueName: \"kubernetes.io/projected/db09c0fc-2974-4e9a-8c0e-f9edd5528c49-kube-api-access-qhrxc\") pod \"db09c0fc-2974-4e9a-8c0e-f9edd5528c49\" (UID: \"db09c0fc-2974-4e9a-8c0e-f9edd5528c49\") " Oct 31 13:04:33.561449 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:33.547756 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30f31282-bb5f-426c-a167-af1e4bb39696-kube-api-access-2nd4r\" (UniqueName: \"kubernetes.io/projected/30f31282-bb5f-426c-a167-af1e4bb39696-kube-api-access-2nd4r\") pod \"30f31282-bb5f-426c-a167-af1e4bb39696\" (UID: \"30f31282-bb5f-426c-a167-af1e4bb39696\") " Oct 31 13:04:33.561449 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:33.547903 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db09c0fc-2974-4e9a-8c0e-f9edd5528c49-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db09c0fc-2974-4e9a-8c0e-f9edd5528c49-plugins-config-volume\") pod \"db09c0fc-2974-4e9a-8c0e-f9edd5528c49\" (UID: \"db09c0fc-2974-4e9a-8c0e-f9edd5528c49\") " Oct 31 13:04:33.561449 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:33.548082 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30f31282-bb5f-426c-a167-af1e4bb39696-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30f31282-bb5f-426c-a167-af1e4bb39696-plugins-config-volume\") pod \"30f31282-bb5f-426c-a167-af1e4bb39696\" (UID: \"30f31282-bb5f-426c-a167-af1e4bb39696\") " Oct 31 13:04:33.562888 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:33.562829 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db09c0fc-2974-4e9a-8c0e-f9edd5528c49-plugins-config-volume" pod "db09c0fc-2974-4e9a-8c0e-f9edd5528c49" (UID: "db09c0fc-2974-4e9a-8c0e-f9edd5528c49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:04:33.563287 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:33.563248 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30f31282-bb5f-426c-a167-af1e4bb39696-plugins-config-volume" pod "30f31282-bb5f-426c-a167-af1e4bb39696" (UID: "30f31282-bb5f-426c-a167-af1e4bb39696"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:04:33.599240 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:33.599163 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db09c0fc-2974-4e9a-8c0e-f9edd5528c49-kube-api-access-qhrxc" pod "db09c0fc-2974-4e9a-8c0e-f9edd5528c49" (UID: "db09c0fc-2974-4e9a-8c0e-f9edd5528c49"). InnerVolumeSpecName "kube-api-access-qhrxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:04:33.704023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:33.689910 6841 patch_prober.go:29] interesting pod/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 31 13:04:33.704023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:33.690001 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" podUID="64dbdf9c-9faf-48af-be69-c820c44aac9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 31 13:04:33.779241 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:33.779193 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30f31282-bb5f-426c-a167-af1e4bb39696-kube-api-access-2nd4r" pod "30f31282-bb5f-426c-a167-af1e4bb39696" (UID: "30f31282-bb5f-426c-a167-af1e4bb39696"). InnerVolumeSpecName "kube-api-access-2nd4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:04:33.860437 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:33.860397 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30f31282-bb5f-426c-a167-af1e4bb39696-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:34.016559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:34.016188 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhrxc\" (UniqueName: \"kubernetes.io/projected/db09c0fc-2974-4e9a-8c0e-f9edd5528c49-kube-api-access-qhrxc\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:34.016559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:34.016224 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nd4r\" (UniqueName: \"kubernetes.io/projected/30f31282-bb5f-426c-a167-af1e4bb39696-kube-api-access-2nd4r\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:34.016559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:33.933099 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:34.016559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:34.016245 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db09c0fc-2974-4e9a-8c0e-f9edd5528c49-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:34.016559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:34.008049 6841 patch_prober.go:29] interesting pod/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 31 13:04:34.016559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:34.016313 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" podUID="c0e2ba69-d8b6-4534-8113-5d4c0bfea735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 31 13:04:34.215155 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:34.213997 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-28e4df-9ff5f7b6c-fvgw4" event={"ID":"db09c0fc-2974-4e9a-8c0e-f9edd5528c49","Type":"ContainerDied","Data":"213f0df112b21dce17e584231f0dff0b9fab9afad5f9126ba05b5fc42713b374"} Oct 31 13:04:34.215155 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:34.214450 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-28e4df-9ff5f7b6c-fvgw4" Oct 31 13:04:34.215155 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:34.214971 6841 scope.go:122] "RemoveContainer" containerID="5d1c7b6d321f4883fc295165bbfbd786f36a3c915d2442bda284a89cb2fd1bd4" Oct 31 13:04:34.217230 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:34.217203 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-28e4df-76bf65bc65-72zf6" event={"ID":"30f31282-bb5f-426c-a167-af1e4bb39696","Type":"ContainerDied","Data":"b305cb4830c27d48d2700a5b4965cd5222c5287ffe873e622dd5dfe559664f80"} Oct 31 13:04:34.217230 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:34.217212 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-28e4df-76bf65bc65-72zf6" Oct 31 13:04:34.229434 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:34.229419 6841 scope.go:122] "RemoveContainer" containerID="10498e593c5142d90a58c68583185bca54c87509c2b007c2972768016b17e763" Oct 31 13:04:34.305036 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:34.304101 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:04:34.305208 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:34.305068 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-28e4df-9ff5f7b6c-fvgw4"] Oct 31 13:04:34.312971 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:34.312937 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-28e4df-9ff5f7b6c-fvgw4"] Oct 31 13:04:34.323705 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:34.323678 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-28e4df-76bf65bc65-72zf6"] Oct 31 13:04:34.327527 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:34.327500 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-28e4df-76bf65bc65-72zf6"] Oct 31 13:04:34.467095 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:34.465638 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:34.702061 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:34.696150 6841 patch_prober.go:29] interesting pod/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 31 13:04:34.702061 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:34.696216 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" podUID="64dbdf9c-9faf-48af-be69-c820c44aac9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 31 13:04:34.907828 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:34.907606 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:35.000306 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:34.999682 6841 patch_prober.go:29] interesting pod/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 31 13:04:35.000306 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:34.999741 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" podUID="c0e2ba69-d8b6-4534-8113-5d4c0bfea735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 31 13:04:35.010995 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.010956 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:04:35.101703 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.100407 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaeed36d-9d97-46f0-93f7-955981f85ede-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaeed36d-9d97-46f0-93f7-955981f85ede-workload-socket\") pod \"eaeed36d-9d97-46f0-93f7-955981f85ede\" (UID: \"eaeed36d-9d97-46f0-93f7-955981f85ede\") " Oct 31 13:04:35.101703 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.100505 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaeed36d-9d97-46f0-93f7-955981f85ede-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaeed36d-9d97-46f0-93f7-955981f85ede-credential-socket\") pod \"eaeed36d-9d97-46f0-93f7-955981f85ede\" (UID: \"eaeed36d-9d97-46f0-93f7-955981f85ede\") " Oct 31 13:04:35.101703 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.100544 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaeed36d-9d97-46f0-93f7-955981f85ede-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaeed36d-9d97-46f0-93f7-955981f85ede-istiod-ca-cert\") pod \"eaeed36d-9d97-46f0-93f7-955981f85ede\" (UID: \"eaeed36d-9d97-46f0-93f7-955981f85ede\") " Oct 31 13:04:35.101703 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.100570 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaeed36d-9d97-46f0-93f7-955981f85ede-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaeed36d-9d97-46f0-93f7-955981f85ede-workload-certs\") pod \"eaeed36d-9d97-46f0-93f7-955981f85ede\" (UID: \"eaeed36d-9d97-46f0-93f7-955981f85ede\") " Oct 31 13:04:35.101703 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.100602 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaeed36d-9d97-46f0-93f7-955981f85ede-kube-api-access-vcgrh\" (UniqueName: \"kubernetes.io/projected/eaeed36d-9d97-46f0-93f7-955981f85ede-kube-api-access-vcgrh\") pod \"eaeed36d-9d97-46f0-93f7-955981f85ede\" (UID: \"eaeed36d-9d97-46f0-93f7-955981f85ede\") " Oct 31 13:04:35.101703 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.100696 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaeed36d-9d97-46f0-93f7-955981f85ede-istio-token\" (UniqueName: \"kubernetes.io/projected/eaeed36d-9d97-46f0-93f7-955981f85ede-istio-token\") pod \"eaeed36d-9d97-46f0-93f7-955981f85ede\" (UID: \"eaeed36d-9d97-46f0-93f7-955981f85ede\") " Oct 31 13:04:35.101703 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.100744 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eaeed36d-9d97-46f0-93f7-955981f85ede-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaeed36d-9d97-46f0-93f7-955981f85ede-istio-podinfo\") pod \"eaeed36d-9d97-46f0-93f7-955981f85ede\" (UID: \"eaeed36d-9d97-46f0-93f7-955981f85ede\") " Oct 31 13:04:35.101703 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.100771 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaeed36d-9d97-46f0-93f7-955981f85ede-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaeed36d-9d97-46f0-93f7-955981f85ede-istio-envoy\") pod \"eaeed36d-9d97-46f0-93f7-955981f85ede\" (UID: \"eaeed36d-9d97-46f0-93f7-955981f85ede\") " Oct 31 13:04:35.101703 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.100794 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaeed36d-9d97-46f0-93f7-955981f85ede-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaeed36d-9d97-46f0-93f7-955981f85ede-istio-data\") pod \"eaeed36d-9d97-46f0-93f7-955981f85ede\" (UID: \"eaeed36d-9d97-46f0-93f7-955981f85ede\") " Oct 31 13:04:35.101703 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.100846 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaeed36d-9d97-46f0-93f7-955981f85ede-credential-socket" pod "eaeed36d-9d97-46f0-93f7-955981f85ede" (UID: "eaeed36d-9d97-46f0-93f7-955981f85ede"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:35.101703 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.100956 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaeed36d-9d97-46f0-93f7-955981f85ede-workload-socket" pod "eaeed36d-9d97-46f0-93f7-955981f85ede" (UID: "eaeed36d-9d97-46f0-93f7-955981f85ede"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:35.101703 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.101304 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaeed36d-9d97-46f0-93f7-955981f85ede-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:35.101703 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.101337 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaeed36d-9d97-46f0-93f7-955981f85ede-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:35.102290 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.102060 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaeed36d-9d97-46f0-93f7-955981f85ede-workload-certs" pod "eaeed36d-9d97-46f0-93f7-955981f85ede" (UID: "eaeed36d-9d97-46f0-93f7-955981f85ede"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:35.102290 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.102273 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaeed36d-9d97-46f0-93f7-955981f85ede-istio-data" pod "eaeed36d-9d97-46f0-93f7-955981f85ede" (UID: "eaeed36d-9d97-46f0-93f7-955981f85ede"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:35.102942 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.102913 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaeed36d-9d97-46f0-93f7-955981f85ede-istiod-ca-cert" pod "eaeed36d-9d97-46f0-93f7-955981f85ede" (UID: "eaeed36d-9d97-46f0-93f7-955981f85ede"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:04:35.104074 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.104051 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaeed36d-9d97-46f0-93f7-955981f85ede-kube-api-access-vcgrh" pod "eaeed36d-9d97-46f0-93f7-955981f85ede" (UID: "eaeed36d-9d97-46f0-93f7-955981f85ede"). InnerVolumeSpecName "kube-api-access-vcgrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:04:35.104183 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.104133 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaeed36d-9d97-46f0-93f7-955981f85ede-istio-envoy" pod "eaeed36d-9d97-46f0-93f7-955981f85ede" (UID: "eaeed36d-9d97-46f0-93f7-955981f85ede"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:35.104922 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.104877 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eaeed36d-9d97-46f0-93f7-955981f85ede-istio-podinfo" pod "eaeed36d-9d97-46f0-93f7-955981f85ede" (UID: "eaeed36d-9d97-46f0-93f7-955981f85ede"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:04:35.105050 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.104917 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaeed36d-9d97-46f0-93f7-955981f85ede-istio-token" pod "eaeed36d-9d97-46f0-93f7-955981f85ede" (UID: "eaeed36d-9d97-46f0-93f7-955981f85ede"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:04:35.203490 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.203342 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaeed36d-9d97-46f0-93f7-955981f85ede-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:35.203490 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.203376 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaeed36d-9d97-46f0-93f7-955981f85ede-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:35.203490 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.203392 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcgrh\" (UniqueName: \"kubernetes.io/projected/eaeed36d-9d97-46f0-93f7-955981f85ede-kube-api-access-vcgrh\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:35.203490 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.203409 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaeed36d-9d97-46f0-93f7-955981f85ede-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:35.203490 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.203424 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaeed36d-9d97-46f0-93f7-955981f85ede-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:35.203490 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.203438 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaeed36d-9d97-46f0-93f7-955981f85ede-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:35.203490 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.203455 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaeed36d-9d97-46f0-93f7-955981f85ede-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:35.227665 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.227619 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:35.229050 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.229027 6841 generic.go:356] "Generic (PLEG): container finished" podID="eaeed36d-9d97-46f0-93f7-955981f85ede" containerID="531dd6f1c4d6ac3c71766b073972cd52a90903f4c6ed8f6eaf81a717fd78a55a" exitCode=0 Oct 31 13:04:35.229162 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.229095 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" Oct 31 13:04:35.229213 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.229094 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" event={"ID":"eaeed36d-9d97-46f0-93f7-955981f85ede","Type":"ContainerDied","Data":"531dd6f1c4d6ac3c71766b073972cd52a90903f4c6ed8f6eaf81a717fd78a55a"} Oct 31 13:04:35.229256 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.229239 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf" event={"ID":"eaeed36d-9d97-46f0-93f7-955981f85ede","Type":"ContainerDied","Data":"4c3687ac4242896937337119c76aec8fc4bacad1b73abae0e8deb3df7b626bab"} Oct 31 13:04:35.229291 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.229262 6841 scope.go:122] "RemoveContainer" containerID="531dd6f1c4d6ac3c71766b073972cd52a90903f4c6ed8f6eaf81a717fd78a55a" Oct 31 13:04:35.266016 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.265200 6841 scope.go:122] "RemoveContainer" containerID="6eb378320491bf178a5472201b386cb6791e8a9f90659ef35458d9cb320bf8eb" Oct 31 13:04:35.322012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.321692 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf"] Oct 31 13:04:35.330084 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.324391 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-fq8nf"] Oct 31 13:04:35.377592 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.377534 6841 scope.go:122] "RemoveContainer" containerID="531dd6f1c4d6ac3c71766b073972cd52a90903f4c6ed8f6eaf81a717fd78a55a" Oct 31 13:04:35.378634 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:04:35.378579 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"531dd6f1c4d6ac3c71766b073972cd52a90903f4c6ed8f6eaf81a717fd78a55a\": container with ID starting with 531dd6f1c4d6ac3c71766b073972cd52a90903f4c6ed8f6eaf81a717fd78a55a not found: ID does not exist" containerID="531dd6f1c4d6ac3c71766b073972cd52a90903f4c6ed8f6eaf81a717fd78a55a" Oct 31 13:04:35.378854 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.378787 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"531dd6f1c4d6ac3c71766b073972cd52a90903f4c6ed8f6eaf81a717fd78a55a"} err="failed to get container status \"531dd6f1c4d6ac3c71766b073972cd52a90903f4c6ed8f6eaf81a717fd78a55a\": rpc error: code = NotFound desc = could not find container \"531dd6f1c4d6ac3c71766b073972cd52a90903f4c6ed8f6eaf81a717fd78a55a\": container with ID starting with 531dd6f1c4d6ac3c71766b073972cd52a90903f4c6ed8f6eaf81a717fd78a55a not found: ID does not exist" Oct 31 13:04:35.378854 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.378820 6841 scope.go:122] "RemoveContainer" containerID="6eb378320491bf178a5472201b386cb6791e8a9f90659ef35458d9cb320bf8eb" Oct 31 13:04:35.379535 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:04:35.379415 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eb378320491bf178a5472201b386cb6791e8a9f90659ef35458d9cb320bf8eb\": container with ID starting with 6eb378320491bf178a5472201b386cb6791e8a9f90659ef35458d9cb320bf8eb not found: ID does not exist" containerID="6eb378320491bf178a5472201b386cb6791e8a9f90659ef35458d9cb320bf8eb" Oct 31 13:04:35.379535 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.379469 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eb378320491bf178a5472201b386cb6791e8a9f90659ef35458d9cb320bf8eb"} err="failed to get container status \"6eb378320491bf178a5472201b386cb6791e8a9f90659ef35458d9cb320bf8eb\": rpc error: code = NotFound desc = could not find container \"6eb378320491bf178a5472201b386cb6791e8a9f90659ef35458d9cb320bf8eb\": container with ID starting with 6eb378320491bf178a5472201b386cb6791e8a9f90659ef35458d9cb320bf8eb not found: ID does not exist" Oct 31 13:04:35.437315 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.437281 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:35.671486 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.671389 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30f31282-bb5f-426c-a167-af1e4bb39696" path="/var/lib/kubelet/pods/30f31282-bb5f-426c-a167-af1e4bb39696/volumes" Oct 31 13:04:35.672471 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.672447 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db09c0fc-2974-4e9a-8c0e-f9edd5528c49" path="/var/lib/kubelet/pods/db09c0fc-2974-4e9a-8c0e-f9edd5528c49/volumes" Oct 31 13:04:35.673429 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.673397 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaeed36d-9d97-46f0-93f7-955981f85ede" path="/var/lib/kubelet/pods/eaeed36d-9d97-46f0-93f7-955981f85ede/volumes" Oct 31 13:04:35.695030 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.688601 6841 patch_prober.go:29] interesting pod/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 31 13:04:35.695030 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.688677 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" podUID="64dbdf9c-9faf-48af-be69-c820c44aac9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 31 13:04:35.753135 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.752523 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:35.979688 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.979185 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:35.998971 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:35.998940 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:04:36.000454 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:36.000429 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:04:36.130876 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:36.130764 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css"] Oct 31 13:04:36.131074 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:36.131027 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" podUID="07b82d57-b897-421f-b06a-85d23a3d3f3f" containerName="istio-proxy" containerID="cri-o://129835ec5db53c5c28cafcbf434a8eff878cfcec213646347ac243f5e8ced7bb" gracePeriod=30 Oct 31 13:04:36.293561 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:36.293468 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:36.686548 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:36.686472 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:04:36.686972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:36.686949 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:04:36.687725 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:36.687705 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:04:36.804130 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:04:36.803344 6841 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f31282_bb5f_426c_a167_af1e4bb39696.slice/crio-b305cb4830c27d48d2700a5b4965cd5222c5287ffe873e622dd5dfe559664f80\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb09c0fc_2974_4e9a_8c0e_f9edd5528c49.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f31282_bb5f_426c_a167_af1e4bb39696.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb09c0fc_2974_4e9a_8c0e_f9edd5528c49.slice/crio-213f0df112b21dce17e584231f0dff0b9fab9afad5f9126ba05b5fc42713b374\": RecentStats: unable to find data in memory cache]" Oct 31 13:04:36.857199 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:36.857166 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:37.101007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:37.097544 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:37.305881 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:37.305841 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:37.443230 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:04:37.443199 6841 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f31282_bb5f_426c_a167_af1e4bb39696.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f31282_bb5f_426c_a167_af1e4bb39696.slice/crio-b305cb4830c27d48d2700a5b4965cd5222c5287ffe873e622dd5dfe559664f80\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb09c0fc_2974_4e9a_8c0e_f9edd5528c49.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb09c0fc_2974_4e9a_8c0e_f9edd5528c49.slice/crio-213f0df112b21dce17e584231f0dff0b9fab9afad5f9126ba05b5fc42713b374\": RecentStats: unable to find data in memory cache]" Oct 31 13:04:37.974713 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:37.974672 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:38.924737 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:38.923513 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:38.952953 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:38.952793 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh"] Oct 31 13:04:38.966000 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:38.953898 6841 cpu_manager.go:414] "containerMap: removing container" podUID="eaeed36d-9d97-46f0-93f7-955981f85ede" containerName="istio-proxy" Oct 31 13:04:38.966000 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:38.953928 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaeed36d-9d97-46f0-93f7-955981f85ede" containerName="istio-proxy" Oct 31 13:04:38.966000 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:38.953950 6841 cpu_manager.go:414] "containerMap: removing container" podUID="f913d7e0-3523-4694-bbf3-294fcb0af9e3" containerName="maas-api" Oct 31 13:04:38.966000 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:38.953959 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f913d7e0-3523-4694-bbf3-294fcb0af9e3" containerName="maas-api" Oct 31 13:04:38.966000 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:38.954040 6841 cpu_manager.go:414] "containerMap: removing container" podUID="eaeed36d-9d97-46f0-93f7-955981f85ede" containerName="istio-proxy" Oct 31 13:04:38.966000 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:38.954050 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaeed36d-9d97-46f0-93f7-955981f85ede" containerName="istio-proxy" Oct 31 13:04:38.966000 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:38.954064 6841 cpu_manager.go:414] "containerMap: removing container" podUID="2aed2f8d-a0b0-4416-b07c-e73f310bfb0f" containerName="maas-api" Oct 31 13:04:38.966000 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:38.954072 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aed2f8d-a0b0-4416-b07c-e73f310bfb0f" containerName="maas-api" Oct 31 13:04:38.966000 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:38.954086 6841 cpu_manager.go:414] "containerMap: removing container" podUID="30f31282-bb5f-426c-a167-af1e4bb39696" containerName="payload-processing" Oct 31 13:04:38.966000 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:38.954094 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30f31282-bb5f-426c-a167-af1e4bb39696" containerName="payload-processing" Oct 31 13:04:38.966000 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:38.954154 6841 cpu_manager.go:414] "containerMap: removing container" podUID="db09c0fc-2974-4e9a-8c0e-f9edd5528c49" containerName="payload-pre-processing" Oct 31 13:04:38.966000 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:38.954161 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db09c0fc-2974-4e9a-8c0e-f9edd5528c49" containerName="payload-pre-processing" Oct 31 13:04:38.966000 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:38.954315 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaeed36d-9d97-46f0-93f7-955981f85ede" containerName="istio-proxy" Oct 31 13:04:38.966000 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:38.954325 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aed2f8d-a0b0-4416-b07c-e73f310bfb0f" containerName="maas-api" Oct 31 13:04:38.966000 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:38.954336 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="30f31282-bb5f-426c-a167-af1e4bb39696" containerName="payload-processing" Oct 31 13:04:38.966000 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:38.954354 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="db09c0fc-2974-4e9a-8c0e-f9edd5528c49" containerName="payload-pre-processing" Oct 31 13:04:38.966000 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:38.954362 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaeed36d-9d97-46f0-93f7-955981f85ede" containerName="istio-proxy" Oct 31 13:04:38.966000 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:38.954374 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="f913d7e0-3523-4694-bbf3-294fcb0af9e3" containerName="maas-api" Oct 31 13:04:38.969177 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:38.966252 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" Oct 31 13:04:38.973510 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:38.973487 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh"] Oct 31 13:04:38.976168 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:38.976125 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-758f48\"/\"kube-root-ca.crt\"" Oct 31 13:04:38.977278 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:38.977255 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-758f48\"/\"openshift-service-ca.crt\"" Oct 31 13:04:38.977385 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:38.977362 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-758f48\"/\"praxis-route-758f48-kserve-self-signed-certs\"" Oct 31 13:04:38.977494 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:38.977477 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-758f48\"/\"default-dockercfg-9dk9h\"" Oct 31 13:04:39.001969 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:39.001929 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcs4f\" (UniqueName: \"kubernetes.io/projected/688d5b2f-0756-480e-83f2-66e05c48527b-kube-api-access-tcs4f\") pod \"praxis-route-758f48-kserve-7bc886646f-rsfdh\" (UID: \"688d5b2f-0756-480e-83f2-66e05c48527b\") " pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" Oct 31 13:04:39.002156 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:39.002023 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/688d5b2f-0756-480e-83f2-66e05c48527b-model-cache\") pod \"praxis-route-758f48-kserve-7bc886646f-rsfdh\" (UID: \"688d5b2f-0756-480e-83f2-66e05c48527b\") " pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" Oct 31 13:04:39.002156 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:39.002102 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/688d5b2f-0756-480e-83f2-66e05c48527b-tmp-dir\") pod \"praxis-route-758f48-kserve-7bc886646f-rsfdh\" (UID: \"688d5b2f-0756-480e-83f2-66e05c48527b\") " pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" Oct 31 13:04:39.002272 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:39.002172 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/688d5b2f-0756-480e-83f2-66e05c48527b-tls-certs\") pod \"praxis-route-758f48-kserve-7bc886646f-rsfdh\" (UID: \"688d5b2f-0756-480e-83f2-66e05c48527b\") " pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" Oct 31 13:04:39.002377 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:39.002352 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/688d5b2f-0756-480e-83f2-66e05c48527b-home\") pod \"praxis-route-758f48-kserve-7bc886646f-rsfdh\" (UID: \"688d5b2f-0756-480e-83f2-66e05c48527b\") " pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" Oct 31 13:04:39.002518 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:39.002499 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/688d5b2f-0756-480e-83f2-66e05c48527b-dshm\") pod \"praxis-route-758f48-kserve-7bc886646f-rsfdh\" (UID: \"688d5b2f-0756-480e-83f2-66e05c48527b\") " pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" Oct 31 13:04:39.103595 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:39.103563 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/688d5b2f-0756-480e-83f2-66e05c48527b-home\") pod \"praxis-route-758f48-kserve-7bc886646f-rsfdh\" (UID: \"688d5b2f-0756-480e-83f2-66e05c48527b\") " pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" Oct 31 13:04:39.103753 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:39.103704 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/688d5b2f-0756-480e-83f2-66e05c48527b-dshm\") pod \"praxis-route-758f48-kserve-7bc886646f-rsfdh\" (UID: \"688d5b2f-0756-480e-83f2-66e05c48527b\") " pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" Oct 31 13:04:39.103824 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:39.103810 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcs4f\" (UniqueName: \"kubernetes.io/projected/688d5b2f-0756-480e-83f2-66e05c48527b-kube-api-access-tcs4f\") pod \"praxis-route-758f48-kserve-7bc886646f-rsfdh\" (UID: \"688d5b2f-0756-480e-83f2-66e05c48527b\") " pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" Oct 31 13:04:39.103876 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:39.103835 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/688d5b2f-0756-480e-83f2-66e05c48527b-model-cache\") pod \"praxis-route-758f48-kserve-7bc886646f-rsfdh\" (UID: \"688d5b2f-0756-480e-83f2-66e05c48527b\") " pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" Oct 31 13:04:39.103921 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:39.103880 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/688d5b2f-0756-480e-83f2-66e05c48527b-tmp-dir\") pod \"praxis-route-758f48-kserve-7bc886646f-rsfdh\" (UID: \"688d5b2f-0756-480e-83f2-66e05c48527b\") " pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" Oct 31 13:04:39.103960 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:39.103941 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/688d5b2f-0756-480e-83f2-66e05c48527b-home\") pod \"praxis-route-758f48-kserve-7bc886646f-rsfdh\" (UID: \"688d5b2f-0756-480e-83f2-66e05c48527b\") " pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" Oct 31 13:04:39.104032 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:39.103946 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/688d5b2f-0756-480e-83f2-66e05c48527b-tls-certs\") pod \"praxis-route-758f48-kserve-7bc886646f-rsfdh\" (UID: \"688d5b2f-0756-480e-83f2-66e05c48527b\") " pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" Oct 31 13:04:39.104198 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:39.104175 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/688d5b2f-0756-480e-83f2-66e05c48527b-model-cache\") pod \"praxis-route-758f48-kserve-7bc886646f-rsfdh\" (UID: \"688d5b2f-0756-480e-83f2-66e05c48527b\") " pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" Oct 31 13:04:39.104273 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:39.104257 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/688d5b2f-0756-480e-83f2-66e05c48527b-tmp-dir\") pod \"praxis-route-758f48-kserve-7bc886646f-rsfdh\" (UID: \"688d5b2f-0756-480e-83f2-66e05c48527b\") " pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" Oct 31 13:04:39.106174 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:39.106151 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/688d5b2f-0756-480e-83f2-66e05c48527b-dshm\") pod \"praxis-route-758f48-kserve-7bc886646f-rsfdh\" (UID: \"688d5b2f-0756-480e-83f2-66e05c48527b\") " pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" Oct 31 13:04:39.106412 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:39.106394 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/688d5b2f-0756-480e-83f2-66e05c48527b-tls-certs\") pod \"praxis-route-758f48-kserve-7bc886646f-rsfdh\" (UID: \"688d5b2f-0756-480e-83f2-66e05c48527b\") " pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" Oct 31 13:04:39.115671 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:39.115648 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcs4f\" (UniqueName: \"kubernetes.io/projected/688d5b2f-0756-480e-83f2-66e05c48527b-kube-api-access-tcs4f\") pod \"praxis-route-758f48-kserve-7bc886646f-rsfdh\" (UID: \"688d5b2f-0756-480e-83f2-66e05c48527b\") " pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" Oct 31 13:04:39.156016 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:39.155911 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:39.296366 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:39.294363 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" Oct 31 13:04:39.580834 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:39.580801 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh"] Oct 31 13:04:39.581275 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:04:39.581238 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod688d5b2f_0756_480e_83f2_66e05c48527b.slice/crio-488ee03cbc25cc26aff7f1924ebfa573a6b174202d2e11644e23d4521f8ff323 WatchSource:0}: Error finding container 488ee03cbc25cc26aff7f1924ebfa573a6b174202d2e11644e23d4521f8ff323: Status 404 returned error can't find the container with id 488ee03cbc25cc26aff7f1924ebfa573a6b174202d2e11644e23d4521f8ff323 Oct 31 13:04:39.814974 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:39.812234 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:40.351838 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:40.351796 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" event={"ID":"688d5b2f-0756-480e-83f2-66e05c48527b","Type":"ContainerStarted","Data":"948fc3e8d748982dab2910d12204f0a9e5cf3a86fcdf7f653ce1e30973023fbd"} Oct 31 13:04:40.352347 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:40.351843 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" event={"ID":"688d5b2f-0756-480e-83f2-66e05c48527b","Type":"ContainerStarted","Data":"488ee03cbc25cc26aff7f1924ebfa573a6b174202d2e11644e23d4521f8ff323"} Oct 31 13:04:40.375497 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:40.375449 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" podStartSLOduration=2.375435694 podStartE2EDuration="2.375435694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:04:40.371572009 +0000 UTC m=+1926.854694912" watchObservedRunningTime="2026-10-31 13:04:40.375435694 +0000 UTC m=+1926.858558595" Oct 31 13:04:40.681706 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:40.681621 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:41.295270 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:41.295238 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" Oct 31 13:04:41.364519 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:41.364467 6841 generic.go:356] "Generic (PLEG): container finished" podID="07b82d57-b897-421f-b06a-85d23a3d3f3f" containerID="129835ec5db53c5c28cafcbf434a8eff878cfcec213646347ac243f5e8ced7bb" exitCode=0 Oct 31 13:04:41.365202 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:41.364532 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" event={"ID":"07b82d57-b897-421f-b06a-85d23a3d3f3f","Type":"ContainerDied","Data":"129835ec5db53c5c28cafcbf434a8eff878cfcec213646347ac243f5e8ced7bb"} Oct 31 13:04:41.365202 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:41.364593 6841 scope.go:122] "RemoveContainer" containerID="d8485f67ccfb9b121df99422d918b740b14c9068fbc5336a6807e756fe74fa29" Oct 31 13:04:41.435961 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:41.435921 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:04:41.544360 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:41.544323 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07b82d57-b897-421f-b06a-85d23a3d3f3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07b82d57-b897-421f-b06a-85d23a3d3f3f-istio-envoy\") pod \"07b82d57-b897-421f-b06a-85d23a3d3f3f\" (UID: \"07b82d57-b897-421f-b06a-85d23a3d3f3f\") " Oct 31 13:04:41.544568 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:41.544404 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07b82d57-b897-421f-b06a-85d23a3d3f3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07b82d57-b897-421f-b06a-85d23a3d3f3f-istio-podinfo\") pod \"07b82d57-b897-421f-b06a-85d23a3d3f3f\" (UID: \"07b82d57-b897-421f-b06a-85d23a3d3f3f\") " Oct 31 13:04:41.544568 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:41.544429 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07b82d57-b897-421f-b06a-85d23a3d3f3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07b82d57-b897-421f-b06a-85d23a3d3f3f-istio-data\") pod \"07b82d57-b897-421f-b06a-85d23a3d3f3f\" (UID: \"07b82d57-b897-421f-b06a-85d23a3d3f3f\") " Oct 31 13:04:41.544568 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:41.544470 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07b82d57-b897-421f-b06a-85d23a3d3f3f-istio-token\" (UniqueName: \"kubernetes.io/projected/07b82d57-b897-421f-b06a-85d23a3d3f3f-istio-token\") pod \"07b82d57-b897-421f-b06a-85d23a3d3f3f\" (UID: \"07b82d57-b897-421f-b06a-85d23a3d3f3f\") " Oct 31 13:04:41.544568 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:41.544500 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07b82d57-b897-421f-b06a-85d23a3d3f3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07b82d57-b897-421f-b06a-85d23a3d3f3f-istiod-ca-cert\") pod \"07b82d57-b897-421f-b06a-85d23a3d3f3f\" (UID: \"07b82d57-b897-421f-b06a-85d23a3d3f3f\") " Oct 31 13:04:41.544568 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:41.544532 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07b82d57-b897-421f-b06a-85d23a3d3f3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b82d57-b897-421f-b06a-85d23a3d3f3f-credential-socket\") pod \"07b82d57-b897-421f-b06a-85d23a3d3f3f\" (UID: \"07b82d57-b897-421f-b06a-85d23a3d3f3f\") " Oct 31 13:04:41.544826 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:41.544630 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07b82d57-b897-421f-b06a-85d23a3d3f3f-kube-api-access-dglzq\" (UniqueName: \"kubernetes.io/projected/07b82d57-b897-421f-b06a-85d23a3d3f3f-kube-api-access-dglzq\") pod \"07b82d57-b897-421f-b06a-85d23a3d3f3f\" (UID: \"07b82d57-b897-421f-b06a-85d23a3d3f3f\") " Oct 31 13:04:41.544826 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:41.544660 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07b82d57-b897-421f-b06a-85d23a3d3f3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b82d57-b897-421f-b06a-85d23a3d3f3f-workload-socket\") pod \"07b82d57-b897-421f-b06a-85d23a3d3f3f\" (UID: \"07b82d57-b897-421f-b06a-85d23a3d3f3f\") " Oct 31 13:04:41.544826 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:41.544747 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07b82d57-b897-421f-b06a-85d23a3d3f3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07b82d57-b897-421f-b06a-85d23a3d3f3f-workload-certs\") pod \"07b82d57-b897-421f-b06a-85d23a3d3f3f\" (UID: \"07b82d57-b897-421f-b06a-85d23a3d3f3f\") " Oct 31 13:04:41.544826 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:41.544806 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07b82d57-b897-421f-b06a-85d23a3d3f3f-istio-data" pod "07b82d57-b897-421f-b06a-85d23a3d3f3f" (UID: "07b82d57-b897-421f-b06a-85d23a3d3f3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:41.545661 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:41.545052 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07b82d57-b897-421f-b06a-85d23a3d3f3f-credential-socket" pod "07b82d57-b897-421f-b06a-85d23a3d3f3f" (UID: "07b82d57-b897-421f-b06a-85d23a3d3f3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:41.545661 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:41.545277 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07b82d57-b897-421f-b06a-85d23a3d3f3f-workload-certs" pod "07b82d57-b897-421f-b06a-85d23a3d3f3f" (UID: "07b82d57-b897-421f-b06a-85d23a3d3f3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:41.545661 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:41.545459 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07b82d57-b897-421f-b06a-85d23a3d3f3f-workload-socket" pod "07b82d57-b897-421f-b06a-85d23a3d3f3f" (UID: "07b82d57-b897-421f-b06a-85d23a3d3f3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:41.545661 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:41.545577 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07b82d57-b897-421f-b06a-85d23a3d3f3f-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:41.545661 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:41.545617 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07b82d57-b897-421f-b06a-85d23a3d3f3f-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:41.545661 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:41.545619 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07b82d57-b897-421f-b06a-85d23a3d3f3f-istiod-ca-cert" pod "07b82d57-b897-421f-b06a-85d23a3d3f3f" (UID: "07b82d57-b897-421f-b06a-85d23a3d3f3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:04:41.545661 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:41.545634 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b82d57-b897-421f-b06a-85d23a3d3f3f-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:41.547409 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:41.547377 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07b82d57-b897-421f-b06a-85d23a3d3f3f-istio-token" pod "07b82d57-b897-421f-b06a-85d23a3d3f3f" (UID: "07b82d57-b897-421f-b06a-85d23a3d3f3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:04:41.547838 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:41.547811 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07b82d57-b897-421f-b06a-85d23a3d3f3f-istio-podinfo" pod "07b82d57-b897-421f-b06a-85d23a3d3f3f" (UID: "07b82d57-b897-421f-b06a-85d23a3d3f3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:04:41.548491 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:41.548461 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07b82d57-b897-421f-b06a-85d23a3d3f3f-kube-api-access-dglzq" pod "07b82d57-b897-421f-b06a-85d23a3d3f3f" (UID: "07b82d57-b897-421f-b06a-85d23a3d3f3f"). InnerVolumeSpecName "kube-api-access-dglzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:04:41.549529 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:41.549506 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07b82d57-b897-421f-b06a-85d23a3d3f3f-istio-envoy" pod "07b82d57-b897-421f-b06a-85d23a3d3f3f" (UID: "07b82d57-b897-421f-b06a-85d23a3d3f3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:41.646632 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:41.646600 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07b82d57-b897-421f-b06a-85d23a3d3f3f-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:41.646632 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:41.646631 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07b82d57-b897-421f-b06a-85d23a3d3f3f-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:41.646816 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:41.646641 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07b82d57-b897-421f-b06a-85d23a3d3f3f-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:41.646816 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:41.646650 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07b82d57-b897-421f-b06a-85d23a3d3f3f-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:41.646816 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:41.646659 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dglzq\" (UniqueName: \"kubernetes.io/projected/07b82d57-b897-421f-b06a-85d23a3d3f3f-kube-api-access-dglzq\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:41.646816 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:41.646670 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b82d57-b897-421f-b06a-85d23a3d3f3f-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:41.655635 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:41.655607 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:41.889005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:41.878421 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:42.372946 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:42.372918 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" Oct 31 13:04:42.373572 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:42.372919 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css" event={"ID":"07b82d57-b897-421f-b06a-85d23a3d3f3f","Type":"ContainerDied","Data":"9649641d437dea0bc23ffb13a4890e958f53013c01905cfa74f9ebcc5e255dd0"} Oct 31 13:04:42.373572 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:42.373041 6841 scope.go:122] "RemoveContainer" containerID="129835ec5db53c5c28cafcbf434a8eff878cfcec213646347ac243f5e8ced7bb" Oct 31 13:04:42.465013 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:42.459659 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css"] Oct 31 13:04:42.479033 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:42.475212 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-28e4df-openshift-default-6ffd8b88f7-59css"] Oct 31 13:04:42.481501 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:42.480581 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:43.278273 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:43.278033 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:43.516407 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:43.516365 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:43.646270 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:43.646182 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07b82d57-b897-421f-b06a-85d23a3d3f3f" path="/var/lib/kubelet/pods/07b82d57-b897-421f-b06a-85d23a3d3f3f/volumes" Oct 31 13:04:43.747777 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:43.747738 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:43.961029 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:43.953433 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:44.414317 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:44.414057 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:44.695665 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:44.695628 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:44.933275 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:44.933242 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg"] Oct 31 13:04:44.933506 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:44.933482 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg" podUID="c9c17825-4128-47f6-9fe6-b9cdda220ecc" containerName="maas-api" containerID="cri-o://796b34ca925a402a4a6712721b3e867f5f42c970e78c0f7bf418ceffa3f77618" gracePeriod=30 Oct 31 13:04:45.086038 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:45.085972 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:45.317487 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:45.317396 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:45.462314 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:45.462282 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg" Oct 31 13:04:45.469090 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:45.469057 6841 generic.go:356] "Generic (PLEG): container finished" podID="c9c17825-4128-47f6-9fe6-b9cdda220ecc" containerID="796b34ca925a402a4a6712721b3e867f5f42c970e78c0f7bf418ceffa3f77618" exitCode=0 Oct 31 13:04:45.469613 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:45.469337 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg" event={"ID":"c9c17825-4128-47f6-9fe6-b9cdda220ecc","Type":"ContainerDied","Data":"796b34ca925a402a4a6712721b3e867f5f42c970e78c0f7bf418ceffa3f77618"} Oct 31 13:04:45.469835 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:45.469814 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg" event={"ID":"c9c17825-4128-47f6-9fe6-b9cdda220ecc","Type":"ContainerDied","Data":"0b83df1ebb3f469f084d444ea9585825d25f61db8247cc4bf32b75c8ccbd7f6a"} Oct 31 13:04:45.470063 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:45.469353 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg" Oct 31 13:04:45.470424 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:45.469950 6841 scope.go:122] "RemoveContainer" containerID="796b34ca925a402a4a6712721b3e867f5f42c970e78c0f7bf418ceffa3f77618" Oct 31 13:04:45.486650 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:45.486626 6841 scope.go:122] "RemoveContainer" containerID="796b34ca925a402a4a6712721b3e867f5f42c970e78c0f7bf418ceffa3f77618" Oct 31 13:04:45.487294 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:04:45.487263 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"796b34ca925a402a4a6712721b3e867f5f42c970e78c0f7bf418ceffa3f77618\": container with ID starting with 796b34ca925a402a4a6712721b3e867f5f42c970e78c0f7bf418ceffa3f77618 not found: ID does not exist" containerID="796b34ca925a402a4a6712721b3e867f5f42c970e78c0f7bf418ceffa3f77618" Oct 31 13:04:45.487416 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:45.487302 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"796b34ca925a402a4a6712721b3e867f5f42c970e78c0f7bf418ceffa3f77618"} err="failed to get container status \"796b34ca925a402a4a6712721b3e867f5f42c970e78c0f7bf418ceffa3f77618\": rpc error: code = NotFound desc = could not find container \"796b34ca925a402a4a6712721b3e867f5f42c970e78c0f7bf418ceffa3f77618\": container with ID starting with 796b34ca925a402a4a6712721b3e867f5f42c970e78c0f7bf418ceffa3f77618 not found: ID does not exist" Oct 31 13:04:45.558872 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:45.558842 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9c17825-4128-47f6-9fe6-b9cdda220ecc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c17825-4128-47f6-9fe6-b9cdda220ecc-maas-api-tls\") pod \"c9c17825-4128-47f6-9fe6-b9cdda220ecc\" (UID: \"c9c17825-4128-47f6-9fe6-b9cdda220ecc\") " Oct 31 13:04:45.559062 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:45.558894 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9c17825-4128-47f6-9fe6-b9cdda220ecc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9c17825-4128-47f6-9fe6-b9cdda220ecc-metrics-certs\") pod \"c9c17825-4128-47f6-9fe6-b9cdda220ecc\" (UID: \"c9c17825-4128-47f6-9fe6-b9cdda220ecc\") " Oct 31 13:04:45.559062 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:45.558935 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9c17825-4128-47f6-9fe6-b9cdda220ecc-kube-api-access-5q54b\" (UniqueName: \"kubernetes.io/projected/c9c17825-4128-47f6-9fe6-b9cdda220ecc-kube-api-access-5q54b\") pod \"c9c17825-4128-47f6-9fe6-b9cdda220ecc\" (UID: \"c9c17825-4128-47f6-9fe6-b9cdda220ecc\") " Oct 31 13:04:45.559062 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:45.559043 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9c17825-4128-47f6-9fe6-b9cdda220ecc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c17825-4128-47f6-9fe6-b9cdda220ecc-openshift-service-ca\") pod \"c9c17825-4128-47f6-9fe6-b9cdda220ecc\" (UID: \"c9c17825-4128-47f6-9fe6-b9cdda220ecc\") " Oct 31 13:04:45.559415 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:45.559387 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9c17825-4128-47f6-9fe6-b9cdda220ecc-openshift-service-ca" pod "c9c17825-4128-47f6-9fe6-b9cdda220ecc" (UID: "c9c17825-4128-47f6-9fe6-b9cdda220ecc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:04:45.559811 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:45.559792 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c17825-4128-47f6-9fe6-b9cdda220ecc-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:45.561163 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:45.561138 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9c17825-4128-47f6-9fe6-b9cdda220ecc-maas-api-tls" pod "c9c17825-4128-47f6-9fe6-b9cdda220ecc" (UID: "c9c17825-4128-47f6-9fe6-b9cdda220ecc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:04:45.561163 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:45.561154 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c17825-4128-47f6-9fe6-b9cdda220ecc-kube-api-access-5q54b" pod "c9c17825-4128-47f6-9fe6-b9cdda220ecc" (UID: "c9c17825-4128-47f6-9fe6-b9cdda220ecc"). InnerVolumeSpecName "kube-api-access-5q54b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:04:45.561309 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:45.561208 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9c17825-4128-47f6-9fe6-b9cdda220ecc-metrics-certs" pod "c9c17825-4128-47f6-9fe6-b9cdda220ecc" (UID: "c9c17825-4128-47f6-9fe6-b9cdda220ecc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:04:45.662355 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:45.662322 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c17825-4128-47f6-9fe6-b9cdda220ecc-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:45.662355 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:45.662350 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9c17825-4128-47f6-9fe6-b9cdda220ecc-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:45.662539 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:45.662364 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q54b\" (UniqueName: \"kubernetes.io/projected/c9c17825-4128-47f6-9fe6-b9cdda220ecc-kube-api-access-5q54b\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:45.791768 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:45.791737 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg"] Oct 31 13:04:45.795168 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:45.795147 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-204e6c-76df45f75-dsqwg"] Oct 31 13:04:46.204099 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:46.201356 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:47.208826 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:47.208790 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:47.529724 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:04:47.529637 6841 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb09c0fc_2974_4e9a_8c0e_f9edd5528c49.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb09c0fc_2974_4e9a_8c0e_f9edd5528c49.slice/crio-213f0df112b21dce17e584231f0dff0b9fab9afad5f9126ba05b5fc42713b374\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f31282_bb5f_426c_a167_af1e4bb39696.slice/crio-b305cb4830c27d48d2700a5b4965cd5222c5287ffe873e622dd5dfe559664f80\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f31282_bb5f_426c_a167_af1e4bb39696.slice\": RecentStats: unable to find data in memory cache]" Oct 31 13:04:47.643284 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:47.643245 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9c17825-4128-47f6-9fe6-b9cdda220ecc" path="/var/lib/kubelet/pods/c9c17825-4128-47f6-9fe6-b9cdda220ecc/volumes" Oct 31 13:04:47.885189 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:47.885108 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf"] Oct 31 13:04:47.885804 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:47.885790 6841 cpu_manager.go:414] "containerMap: removing container" podUID="07b82d57-b897-421f-b06a-85d23a3d3f3f" containerName="istio-proxy" Oct 31 13:04:47.885858 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:47.885806 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07b82d57-b897-421f-b06a-85d23a3d3f3f" containerName="istio-proxy" Oct 31 13:04:47.885858 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:47.885822 6841 cpu_manager.go:414] "containerMap: removing container" podUID="07b82d57-b897-421f-b06a-85d23a3d3f3f" containerName="istio-proxy" Oct 31 13:04:47.885858 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:47.885826 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07b82d57-b897-421f-b06a-85d23a3d3f3f" containerName="istio-proxy" Oct 31 13:04:47.885858 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:47.885850 6841 cpu_manager.go:414] "containerMap: removing container" podUID="c9c17825-4128-47f6-9fe6-b9cdda220ecc" containerName="maas-api" Oct 31 13:04:47.885858 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:47.885855 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9c17825-4128-47f6-9fe6-b9cdda220ecc" containerName="maas-api" Oct 31 13:04:47.886059 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:47.885935 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="07b82d57-b897-421f-b06a-85d23a3d3f3f" containerName="istio-proxy" Oct 31 13:04:47.886059 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:47.885943 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="07b82d57-b897-421f-b06a-85d23a3d3f3f" containerName="istio-proxy" Oct 31 13:04:47.886059 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:47.885951 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9c17825-4128-47f6-9fe6-b9cdda220ecc" containerName="maas-api" Oct 31 13:04:47.890868 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:47.890843 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" Oct 31 13:04:47.896566 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:47.896546 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-4989f9-openshift-default-dockercfg-8md7r\"" Oct 31 13:04:47.904481 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:47.904458 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf"] Oct 31 13:04:48.021691 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:48.021652 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a47faf-ed37-4499-abcb-cd5fec47dc13-istio-token\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf\" (UID: \"84a47faf-ed37-4499-abcb-cd5fec47dc13\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" Oct 31 13:04:48.021880 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:48.021798 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a47faf-ed37-4499-abcb-cd5fec47dc13-workload-socket\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf\" (UID: \"84a47faf-ed37-4499-abcb-cd5fec47dc13\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" Oct 31 13:04:48.021880 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:48.021864 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a47faf-ed37-4499-abcb-cd5fec47dc13-credential-socket\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf\" (UID: \"84a47faf-ed37-4499-abcb-cd5fec47dc13\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" Oct 31 13:04:48.022040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:48.021965 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a47faf-ed37-4499-abcb-cd5fec47dc13-istio-podinfo\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf\" (UID: \"84a47faf-ed37-4499-abcb-cd5fec47dc13\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" Oct 31 13:04:48.022106 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:48.022052 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a47faf-ed37-4499-abcb-cd5fec47dc13-workload-certs\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf\" (UID: \"84a47faf-ed37-4499-abcb-cd5fec47dc13\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" Oct 31 13:04:48.022172 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:48.022149 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a47faf-ed37-4499-abcb-cd5fec47dc13-istiod-ca-cert\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf\" (UID: \"84a47faf-ed37-4499-abcb-cd5fec47dc13\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" Oct 31 13:04:48.022239 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:48.022195 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvls5\" (UniqueName: \"kubernetes.io/projected/84a47faf-ed37-4499-abcb-cd5fec47dc13-kube-api-access-fvls5\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf\" (UID: \"84a47faf-ed37-4499-abcb-cd5fec47dc13\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" Oct 31 13:04:48.022313 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:48.022295 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a47faf-ed37-4499-abcb-cd5fec47dc13-istio-data\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf\" (UID: \"84a47faf-ed37-4499-abcb-cd5fec47dc13\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" Oct 31 13:04:48.022438 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:48.022417 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a47faf-ed37-4499-abcb-cd5fec47dc13-istio-envoy\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf\" (UID: \"84a47faf-ed37-4499-abcb-cd5fec47dc13\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" Oct 31 13:04:48.124374 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:48.124337 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a47faf-ed37-4499-abcb-cd5fec47dc13-istio-envoy\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf\" (UID: \"84a47faf-ed37-4499-abcb-cd5fec47dc13\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" Oct 31 13:04:48.124571 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:48.124383 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a47faf-ed37-4499-abcb-cd5fec47dc13-istio-token\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf\" (UID: \"84a47faf-ed37-4499-abcb-cd5fec47dc13\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" Oct 31 13:04:48.124571 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:48.124430 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a47faf-ed37-4499-abcb-cd5fec47dc13-workload-socket\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf\" (UID: \"84a47faf-ed37-4499-abcb-cd5fec47dc13\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" Oct 31 13:04:48.124571 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:48.124469 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a47faf-ed37-4499-abcb-cd5fec47dc13-credential-socket\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf\" (UID: \"84a47faf-ed37-4499-abcb-cd5fec47dc13\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" Oct 31 13:04:48.124571 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:48.124498 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a47faf-ed37-4499-abcb-cd5fec47dc13-istio-podinfo\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf\" (UID: \"84a47faf-ed37-4499-abcb-cd5fec47dc13\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" Oct 31 13:04:48.124571 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:48.124538 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a47faf-ed37-4499-abcb-cd5fec47dc13-workload-certs\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf\" (UID: \"84a47faf-ed37-4499-abcb-cd5fec47dc13\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" Oct 31 13:04:48.124850 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:48.124629 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a47faf-ed37-4499-abcb-cd5fec47dc13-istiod-ca-cert\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf\" (UID: \"84a47faf-ed37-4499-abcb-cd5fec47dc13\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" Oct 31 13:04:48.124850 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:48.124681 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvls5\" (UniqueName: \"kubernetes.io/projected/84a47faf-ed37-4499-abcb-cd5fec47dc13-kube-api-access-fvls5\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf\" (UID: \"84a47faf-ed37-4499-abcb-cd5fec47dc13\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" Oct 31 13:04:48.124969 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:48.124853 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a47faf-ed37-4499-abcb-cd5fec47dc13-istio-data\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf\" (UID: \"84a47faf-ed37-4499-abcb-cd5fec47dc13\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" Oct 31 13:04:48.124969 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:48.124863 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a47faf-ed37-4499-abcb-cd5fec47dc13-workload-socket\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf\" (UID: \"84a47faf-ed37-4499-abcb-cd5fec47dc13\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" Oct 31 13:04:48.124969 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:48.124904 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a47faf-ed37-4499-abcb-cd5fec47dc13-credential-socket\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf\" (UID: \"84a47faf-ed37-4499-abcb-cd5fec47dc13\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" Oct 31 13:04:48.125195 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:48.125074 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a47faf-ed37-4499-abcb-cd5fec47dc13-workload-certs\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf\" (UID: \"84a47faf-ed37-4499-abcb-cd5fec47dc13\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" Oct 31 13:04:48.125309 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:48.125287 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a47faf-ed37-4499-abcb-cd5fec47dc13-istio-data\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf\" (UID: \"84a47faf-ed37-4499-abcb-cd5fec47dc13\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" Oct 31 13:04:48.125483 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:48.125462 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a47faf-ed37-4499-abcb-cd5fec47dc13-istiod-ca-cert\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf\" (UID: \"84a47faf-ed37-4499-abcb-cd5fec47dc13\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" Oct 31 13:04:48.127546 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:48.127521 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a47faf-ed37-4499-abcb-cd5fec47dc13-istio-podinfo\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf\" (UID: \"84a47faf-ed37-4499-abcb-cd5fec47dc13\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" Oct 31 13:04:48.127730 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:48.127710 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a47faf-ed37-4499-abcb-cd5fec47dc13-istio-envoy\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf\" (UID: \"84a47faf-ed37-4499-abcb-cd5fec47dc13\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" Oct 31 13:04:48.138089 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:48.138036 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a47faf-ed37-4499-abcb-cd5fec47dc13-istio-token\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf\" (UID: \"84a47faf-ed37-4499-abcb-cd5fec47dc13\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" Oct 31 13:04:48.144419 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:48.144400 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvls5\" (UniqueName: \"kubernetes.io/projected/84a47faf-ed37-4499-abcb-cd5fec47dc13-kube-api-access-fvls5\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf\" (UID: \"84a47faf-ed37-4499-abcb-cd5fec47dc13\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" Oct 31 13:04:48.207418 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:48.207393 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" Oct 31 13:04:48.384364 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:48.384333 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:48.528493 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:48.528464 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf"] Oct 31 13:04:48.528790 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:04:48.528766 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a47faf_ed37_4499_abcb_cd5fec47dc13.slice/crio-2511901b3319532630b21d37c63947ed59b6db7e435a3d87f6c5e7aed4655588 WatchSource:0}: Error finding container 2511901b3319532630b21d37c63947ed59b6db7e435a3d87f6c5e7aed4655588: Status 404 returned error can't find the container with id 2511901b3319532630b21d37c63947ed59b6db7e435a3d87f6c5e7aed4655588 Oct 31 13:04:48.531502 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:48.531472 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:04:48.531580 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:48.531540 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:04:48.531580 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:48.531567 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:04:49.294667 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:49.294615 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" Oct 31 13:04:49.306868 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:49.306840 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" Oct 31 13:04:49.490005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:49.489478 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:49.529306 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:49.529268 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" event={"ID":"84a47faf-ed37-4499-abcb-cd5fec47dc13","Type":"ContainerStarted","Data":"162fd22a616ae574ea364983c82388208f3b2d79d12c7f32996881784a6bd0a3"} Oct 31 13:04:49.530025 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:49.529956 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" event={"ID":"84a47faf-ed37-4499-abcb-cd5fec47dc13","Type":"ContainerStarted","Data":"2511901b3319532630b21d37c63947ed59b6db7e435a3d87f6c5e7aed4655588"} Oct 31 13:04:49.566208 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:49.559345 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" Oct 31 13:04:49.583215 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:49.583142 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" podStartSLOduration=2.583124629 podStartE2EDuration="2.583124629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:04:49.55963893 +0000 UTC m=+1936.042761832" watchObservedRunningTime="2026-10-31 13:04:49.583124629 +0000 UTC m=+1936.066247530" Oct 31 13:04:49.701185 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:49.701151 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:50.078718 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:50.078665 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb"] Oct 31 13:04:50.079249 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:50.078905 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb" podUID="3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f" containerName="maas-api" containerID="cri-o://696785d1ecbccadc9acfaea4a4a25c125f9b24e033991b4804301a5c1b83c24d" gracePeriod=30 Oct 31 13:04:50.086709 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:04:50.086668 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:04:50.119054 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:50.119021 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-12056b-54cd8754db-r8p4d"] Oct 31 13:04:50.134736 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:50.134700 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-6bbff8b447-7827t"] Oct 31 13:04:50.134998 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:50.134952 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-6bbff8b447-7827t" podUID="faafb6b6-9a5a-4a60-a327-1629d2c49868" containerName="payload-pre-processing" containerID="cri-o://fd4e6c1e0813d908b1058c97ac39f25c9bf4c945dbeb2c0a938a4fd67c8bf36e" gracePeriod=30 Oct 31 13:04:50.194135 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:50.194096 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:50.207896 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:50.207863 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" Oct 31 13:04:50.209274 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:50.209233 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 31 13:04:50.209422 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:50.209316 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" podUID="84a47faf-ed37-4499-abcb-cd5fec47dc13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 31 13:04:50.398005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:50.397677 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:50.573006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:50.567955 6841 generic.go:356] "Generic (PLEG): container finished" podID="3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f" containerID="696785d1ecbccadc9acfaea4a4a25c125f9b24e033991b4804301a5c1b83c24d" exitCode=0 Oct 31 13:04:50.573006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:50.568088 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb" event={"ID":"3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f","Type":"ContainerDied","Data":"696785d1ecbccadc9acfaea4a4a25c125f9b24e033991b4804301a5c1b83c24d"} Oct 31 13:04:50.584919 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:50.584376 6841 generic.go:356] "Generic (PLEG): container finished" podID="faafb6b6-9a5a-4a60-a327-1629d2c49868" containerID="fd4e6c1e0813d908b1058c97ac39f25c9bf4c945dbeb2c0a938a4fd67c8bf36e" exitCode=0 Oct 31 13:04:50.584919 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:50.584442 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-6bbff8b447-7827t" event={"ID":"faafb6b6-9a5a-4a60-a327-1629d2c49868","Type":"ContainerDied","Data":"fd4e6c1e0813d908b1058c97ac39f25c9bf4c945dbeb2c0a938a4fd67c8bf36e"} Oct 31 13:04:50.586682 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:50.586059 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-12056b-54cd8754db-r8p4d" podUID="6a4859e8-5b79-4f2a-afe6-0b26a64cce08" containerName="payload-processing" containerID="cri-o://5413df0b729086d9d32bf9c59ae46670699436492c5be164e43513ccb3a02c60" gracePeriod=30 Oct 31 13:04:50.617055 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:50.616924 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:50.990486 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:50.989867 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb" Oct 31 13:04:50.991610 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:50.991537 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:50.995630 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:50.995554 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-6bbff8b447-7827t" Oct 31 13:04:51.098881 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.098724 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f-maas-api-tls\") pod \"3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f\" (UID: \"3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f\") " Oct 31 13:04:51.098881 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.098863 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faafb6b6-9a5a-4a60-a327-1629d2c49868-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faafb6b6-9a5a-4a60-a327-1629d2c49868-plugins-config-volume\") pod \"faafb6b6-9a5a-4a60-a327-1629d2c49868\" (UID: \"faafb6b6-9a5a-4a60-a327-1629d2c49868\") " Oct 31 13:04:51.099156 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.098932 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f-metrics-certs\") pod \"3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f\" (UID: \"3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f\") " Oct 31 13:04:51.099156 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.099018 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faafb6b6-9a5a-4a60-a327-1629d2c49868-kube-api-access-5hlrj\" (UniqueName: \"kubernetes.io/projected/faafb6b6-9a5a-4a60-a327-1629d2c49868-kube-api-access-5hlrj\") pod \"faafb6b6-9a5a-4a60-a327-1629d2c49868\" (UID: \"faafb6b6-9a5a-4a60-a327-1629d2c49868\") " Oct 31 13:04:51.099156 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.099078 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f-kube-api-access-stcvf\" (UniqueName: \"kubernetes.io/projected/3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f-kube-api-access-stcvf\") pod \"3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f\" (UID: \"3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f\") " Oct 31 13:04:51.099156 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.099141 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f-openshift-service-ca\") pod \"3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f\" (UID: \"3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f\") " Oct 31 13:04:51.099897 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.099690 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faafb6b6-9a5a-4a60-a327-1629d2c49868-plugins-config-volume" pod "faafb6b6-9a5a-4a60-a327-1629d2c49868" (UID: "faafb6b6-9a5a-4a60-a327-1629d2c49868"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:04:51.100187 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.100129 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f-openshift-service-ca" pod "3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f" (UID: "3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:04:51.102840 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.102810 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f-metrics-certs" pod "3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f" (UID: "3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:04:51.103702 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.103657 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f-maas-api-tls" pod "3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f" (UID: "3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:04:51.104141 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.104115 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f-kube-api-access-stcvf" pod "3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f" (UID: "3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f"). InnerVolumeSpecName "kube-api-access-stcvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:04:51.110507 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.110479 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faafb6b6-9a5a-4a60-a327-1629d2c49868-kube-api-access-5hlrj" pod "faafb6b6-9a5a-4a60-a327-1629d2c49868" (UID: "faafb6b6-9a5a-4a60-a327-1629d2c49868"). InnerVolumeSpecName "kube-api-access-5hlrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:04:51.133867 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.133845 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-12056b-54cd8754db-r8p4d" Oct 31 13:04:51.200544 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.200510 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a4859e8-5b79-4f2a-afe6-0b26a64cce08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a4859e8-5b79-4f2a-afe6-0b26a64cce08-plugins-config-volume\") pod \"6a4859e8-5b79-4f2a-afe6-0b26a64cce08\" (UID: \"6a4859e8-5b79-4f2a-afe6-0b26a64cce08\") " Oct 31 13:04:51.200720 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.200701 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a4859e8-5b79-4f2a-afe6-0b26a64cce08-kube-api-access-lwqvs\" (UniqueName: \"kubernetes.io/projected/6a4859e8-5b79-4f2a-afe6-0b26a64cce08-kube-api-access-lwqvs\") pod \"6a4859e8-5b79-4f2a-afe6-0b26a64cce08\" (UID: \"6a4859e8-5b79-4f2a-afe6-0b26a64cce08\") " Oct 31 13:04:51.201053 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.200977 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a4859e8-5b79-4f2a-afe6-0b26a64cce08-plugins-config-volume" pod "6a4859e8-5b79-4f2a-afe6-0b26a64cce08" (UID: "6a4859e8-5b79-4f2a-afe6-0b26a64cce08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:04:51.201499 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.201471 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faafb6b6-9a5a-4a60-a327-1629d2c49868-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:51.201603 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.201506 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:51.201603 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.201522 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hlrj\" (UniqueName: \"kubernetes.io/projected/faafb6b6-9a5a-4a60-a327-1629d2c49868-kube-api-access-5hlrj\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:51.201603 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.201536 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stcvf\" (UniqueName: \"kubernetes.io/projected/3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f-kube-api-access-stcvf\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:51.201603 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.201550 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:51.201603 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.201563 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a4859e8-5b79-4f2a-afe6-0b26a64cce08-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:51.201603 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.201576 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:51.203622 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.203591 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a4859e8-5b79-4f2a-afe6-0b26a64cce08-kube-api-access-lwqvs" pod "6a4859e8-5b79-4f2a-afe6-0b26a64cce08" (UID: "6a4859e8-5b79-4f2a-afe6-0b26a64cce08"). InnerVolumeSpecName "kube-api-access-lwqvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:04:51.208288 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.208260 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 31 13:04:51.208415 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.208336 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" podUID="84a47faf-ed37-4499-abcb-cd5fec47dc13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 31 13:04:51.318004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.317481 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwqvs\" (UniqueName: \"kubernetes.io/projected/6a4859e8-5b79-4f2a-afe6-0b26a64cce08-kube-api-access-lwqvs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:51.386476 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.386387 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:51.596000 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.595805 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb" event={"ID":"3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f","Type":"ContainerDied","Data":"aace8892791ab1b5063014a4f257400a05bcb03eadf01f49c9cce3ae2b9824ff"} Oct 31 13:04:51.596000 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.595863 6841 scope.go:122] "RemoveContainer" containerID="696785d1ecbccadc9acfaea4a4a25c125f9b24e033991b4804301a5c1b83c24d" Oct 31 13:04:51.596538 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.596018 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb" Oct 31 13:04:51.601205 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.601175 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-6bbff8b447-7827t" event={"ID":"faafb6b6-9a5a-4a60-a327-1629d2c49868","Type":"ContainerDied","Data":"dd0b58eb4ff6d4c5702fefc1c0458925464842f1ce97a5eca668f26c8b80814f"} Oct 31 13:04:51.601314 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.601277 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-6bbff8b447-7827t" Oct 31 13:04:51.608404 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.608379 6841 generic.go:356] "Generic (PLEG): container finished" podID="6a4859e8-5b79-4f2a-afe6-0b26a64cce08" containerID="5413df0b729086d9d32bf9c59ae46670699436492c5be164e43513ccb3a02c60" exitCode=0 Oct 31 13:04:51.609426 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.609405 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-12056b-54cd8754db-r8p4d" Oct 31 13:04:51.611181 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.611141 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-12056b-54cd8754db-r8p4d" event={"ID":"6a4859e8-5b79-4f2a-afe6-0b26a64cce08","Type":"ContainerDied","Data":"5413df0b729086d9d32bf9c59ae46670699436492c5be164e43513ccb3a02c60"} Oct 31 13:04:51.611272 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.611208 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-12056b-54cd8754db-r8p4d" event={"ID":"6a4859e8-5b79-4f2a-afe6-0b26a64cce08","Type":"ContainerDied","Data":"71682856657dadec5d5115bf7a6fff7091df7eb04bb54cb84541c04a1f4a3f0d"} Oct 31 13:04:51.614493 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.614467 6841 scope.go:122] "RemoveContainer" containerID="fd4e6c1e0813d908b1058c97ac39f25c9bf4c945dbeb2c0a938a4fd67c8bf36e" Oct 31 13:04:51.639147 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.639126 6841 scope.go:122] "RemoveContainer" containerID="5413df0b729086d9d32bf9c59ae46670699436492c5be164e43513ccb3a02c60" Oct 31 13:04:51.676122 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.676100 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb"] Oct 31 13:04:51.677423 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.677404 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-12056b-7df76d8f4d-j6drb"] Oct 31 13:04:51.685354 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.685330 6841 scope.go:122] "RemoveContainer" containerID="5413df0b729086d9d32bf9c59ae46670699436492c5be164e43513ccb3a02c60" Oct 31 13:04:51.685840 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:04:51.685813 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5413df0b729086d9d32bf9c59ae46670699436492c5be164e43513ccb3a02c60\": container with ID starting with 5413df0b729086d9d32bf9c59ae46670699436492c5be164e43513ccb3a02c60 not found: ID does not exist" containerID="5413df0b729086d9d32bf9c59ae46670699436492c5be164e43513ccb3a02c60" Oct 31 13:04:51.686037 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.685848 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5413df0b729086d9d32bf9c59ae46670699436492c5be164e43513ccb3a02c60"} err="failed to get container status \"5413df0b729086d9d32bf9c59ae46670699436492c5be164e43513ccb3a02c60\": rpc error: code = NotFound desc = could not find container \"5413df0b729086d9d32bf9c59ae46670699436492c5be164e43513ccb3a02c60\": container with ID starting with 5413df0b729086d9d32bf9c59ae46670699436492c5be164e43513ccb3a02c60 not found: ID does not exist" Oct 31 13:04:51.686921 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.686900 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-6bbff8b447-7827t"] Oct 31 13:04:51.690617 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.690596 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-12056b-6bbff8b447-7827t"] Oct 31 13:04:51.714023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.713415 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-12056b-54cd8754db-r8p4d"] Oct 31 13:04:51.754351 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:04:51.741938 6841 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f31282_bb5f_426c_a167_af1e4bb39696.slice/crio-b305cb4830c27d48d2700a5b4965cd5222c5287ffe873e622dd5dfe559664f80\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb09c0fc_2974_4e9a_8c0e_f9edd5528c49.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb09c0fc_2974_4e9a_8c0e_f9edd5528c49.slice/crio-213f0df112b21dce17e584231f0dff0b9fab9afad5f9126ba05b5fc42713b374\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f31282_bb5f_426c_a167_af1e4bb39696.slice\": RecentStats: unable to find data in memory cache]" Oct 31 13:04:51.765670 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.765624 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:51.848008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.830399 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-12056b-54cd8754db-r8p4d"] Oct 31 13:04:51.989440 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:51.989404 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:52.208755 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:52.208720 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 31 13:04:52.208941 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:52.208771 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" podUID="84a47faf-ed37-4499-abcb-cd5fec47dc13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 31 13:04:52.785098 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:52.785012 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:53.130306 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:53.130219 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:53.208386 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:53.208352 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 31 13:04:53.208540 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:53.208402 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" podUID="84a47faf-ed37-4499-abcb-cd5fec47dc13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 31 13:04:53.410289 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:53.410198 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:53.784126 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:53.784084 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f" path="/var/lib/kubelet/pods/3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f/volumes" Oct 31 13:04:53.784795 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:53.784767 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a4859e8-5b79-4f2a-afe6-0b26a64cce08" path="/var/lib/kubelet/pods/6a4859e8-5b79-4f2a-afe6-0b26a64cce08/volumes" Oct 31 13:04:53.785412 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:53.785386 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faafb6b6-9a5a-4a60-a327-1629d2c49868" path="/var/lib/kubelet/pods/faafb6b6-9a5a-4a60-a327-1629d2c49868/volumes" Oct 31 13:04:53.840805 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:53.840768 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:54.207901 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:54.207865 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 31 13:04:54.208107 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:54.207924 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" podUID="84a47faf-ed37-4499-abcb-cd5fec47dc13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 31 13:04:54.748743 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:54.748700 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:55.208159 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:55.208131 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 31 13:04:55.208537 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:55.208183 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" podUID="84a47faf-ed37-4499-abcb-cd5fec47dc13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 31 13:04:55.511871 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:55.507947 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:56.241524 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:56.239269 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" Oct 31 13:04:56.241524 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:56.241017 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" Oct 31 13:04:56.247371 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:56.247343 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" Oct 31 13:04:56.320671 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:56.320605 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:56.593240 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:56.593162 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:56.793436 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:56.793404 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:57.007218 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:57.007183 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6"] Oct 31 13:04:57.007464 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:57.007416 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" podUID="13c0b6da-6006-44b6-8677-d16ffd433df2" containerName="istio-proxy" containerID="cri-o://86763a7e5a59fbcc1a8a4cb605298a0690bcae89ea0b98a50adbd1909be7b847" gracePeriod=30 Oct 31 13:04:57.179109 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:57.179063 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:57.567873 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:57.567840 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:57.568558 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:04:57.568512 6841 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f31282_bb5f_426c_a167_af1e4bb39696.slice/crio-b305cb4830c27d48d2700a5b4965cd5222c5287ffe873e622dd5dfe559664f80\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb09c0fc_2974_4e9a_8c0e_f9edd5528c49.slice/crio-213f0df112b21dce17e584231f0dff0b9fab9afad5f9126ba05b5fc42713b374\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb09c0fc_2974_4e9a_8c0e_f9edd5528c49.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f31282_bb5f_426c_a167_af1e4bb39696.slice\": RecentStats: unable to find data in memory cache]" Oct 31 13:04:58.075766 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:58.075724 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:59.164550 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:59.164498 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:04:59.610461 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:59.610427 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:04:59.610630 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:04:59.610476 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" podUID="13c0b6da-6006-44b6-8677-d16ffd433df2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:05:00.242375 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:00.242338 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:00.476016 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:00.475942 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:00.717263 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:00.717225 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:01.018956 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:01.018867 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:01.231930 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:01.231892 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq"] Oct 31 13:05:01.232219 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:01.232191 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" podUID="64dbdf9c-9faf-48af-be69-c820c44aac9f" containerName="istio-proxy" containerID="cri-o://fc464ae15786892df57839c9378da40b0680a222196893ac50dfa544394b9a56" gracePeriod=30 Oct 31 13:05:01.371772 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:01.371680 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:01.609219 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:01.609170 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:02.294459 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.294433 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:05:02.377607 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.371445 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13c0b6da-6006-44b6-8677-d16ffd433df2-istio-token\" (UniqueName: \"kubernetes.io/projected/13c0b6da-6006-44b6-8677-d16ffd433df2-istio-token\") pod \"13c0b6da-6006-44b6-8677-d16ffd433df2\" (UID: \"13c0b6da-6006-44b6-8677-d16ffd433df2\") " Oct 31 13:05:02.391497 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.391460 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13c0b6da-6006-44b6-8677-d16ffd433df2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c0b6da-6006-44b6-8677-d16ffd433df2-workload-socket\") pod \"13c0b6da-6006-44b6-8677-d16ffd433df2\" (UID: \"13c0b6da-6006-44b6-8677-d16ffd433df2\") " Oct 31 13:05:02.395643 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.392118 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/13c0b6da-6006-44b6-8677-d16ffd433df2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13c0b6da-6006-44b6-8677-d16ffd433df2-istio-podinfo\") pod \"13c0b6da-6006-44b6-8677-d16ffd433df2\" (UID: \"13c0b6da-6006-44b6-8677-d16ffd433df2\") " Oct 31 13:05:02.395643 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.392217 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13c0b6da-6006-44b6-8677-d16ffd433df2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13c0b6da-6006-44b6-8677-d16ffd433df2-workload-certs\") pod \"13c0b6da-6006-44b6-8677-d16ffd433df2\" (UID: \"13c0b6da-6006-44b6-8677-d16ffd433df2\") " Oct 31 13:05:02.395643 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.392379 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13c0b6da-6006-44b6-8677-d16ffd433df2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13c0b6da-6006-44b6-8677-d16ffd433df2-istio-data\") pod \"13c0b6da-6006-44b6-8677-d16ffd433df2\" (UID: \"13c0b6da-6006-44b6-8677-d16ffd433df2\") " Oct 31 13:05:02.395643 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.392432 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13c0b6da-6006-44b6-8677-d16ffd433df2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c0b6da-6006-44b6-8677-d16ffd433df2-credential-socket\") pod \"13c0b6da-6006-44b6-8677-d16ffd433df2\" (UID: \"13c0b6da-6006-44b6-8677-d16ffd433df2\") " Oct 31 13:05:02.395643 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.392510 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13c0b6da-6006-44b6-8677-d16ffd433df2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13c0b6da-6006-44b6-8677-d16ffd433df2-istio-envoy\") pod \"13c0b6da-6006-44b6-8677-d16ffd433df2\" (UID: \"13c0b6da-6006-44b6-8677-d16ffd433df2\") " Oct 31 13:05:02.395643 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.392743 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13c0b6da-6006-44b6-8677-d16ffd433df2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13c0b6da-6006-44b6-8677-d16ffd433df2-istiod-ca-cert\") pod \"13c0b6da-6006-44b6-8677-d16ffd433df2\" (UID: \"13c0b6da-6006-44b6-8677-d16ffd433df2\") " Oct 31 13:05:02.395643 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.392826 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13c0b6da-6006-44b6-8677-d16ffd433df2-kube-api-access-7gqqg\" (UniqueName: \"kubernetes.io/projected/13c0b6da-6006-44b6-8677-d16ffd433df2-kube-api-access-7gqqg\") pod \"13c0b6da-6006-44b6-8677-d16ffd433df2\" (UID: \"13c0b6da-6006-44b6-8677-d16ffd433df2\") " Oct 31 13:05:02.416100 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.392033 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c0b6da-6006-44b6-8677-d16ffd433df2-workload-socket" pod "13c0b6da-6006-44b6-8677-d16ffd433df2" (UID: "13c0b6da-6006-44b6-8677-d16ffd433df2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:05:02.416100 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.415595 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c0b6da-6006-44b6-8677-d16ffd433df2-istio-data" pod "13c0b6da-6006-44b6-8677-d16ffd433df2" (UID: "13c0b6da-6006-44b6-8677-d16ffd433df2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:05:02.419859 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.419819 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c0b6da-6006-44b6-8677-d16ffd433df2-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:02.420835 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.420769 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c0b6da-6006-44b6-8677-d16ffd433df2-credential-socket" pod "13c0b6da-6006-44b6-8677-d16ffd433df2" (UID: "13c0b6da-6006-44b6-8677-d16ffd433df2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:05:02.420835 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.420755 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13c0b6da-6006-44b6-8677-d16ffd433df2-kube-api-access-7gqqg" pod "13c0b6da-6006-44b6-8677-d16ffd433df2" (UID: "13c0b6da-6006-44b6-8677-d16ffd433df2"). InnerVolumeSpecName "kube-api-access-7gqqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:05:02.423026 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.422834 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c0b6da-6006-44b6-8677-d16ffd433df2-workload-certs" pod "13c0b6da-6006-44b6-8677-d16ffd433df2" (UID: "13c0b6da-6006-44b6-8677-d16ffd433df2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:05:02.423155 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.423059 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:02.423657 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.423634 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c0b6da-6006-44b6-8677-d16ffd433df2-istio-envoy" pod "13c0b6da-6006-44b6-8677-d16ffd433df2" (UID: "13c0b6da-6006-44b6-8677-d16ffd433df2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:05:02.429900 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.429839 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13c0b6da-6006-44b6-8677-d16ffd433df2-istiod-ca-cert" pod "13c0b6da-6006-44b6-8677-d16ffd433df2" (UID: "13c0b6da-6006-44b6-8677-d16ffd433df2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:05:02.431207 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.431163 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13c0b6da-6006-44b6-8677-d16ffd433df2-istio-token" pod "13c0b6da-6006-44b6-8677-d16ffd433df2" (UID: "13c0b6da-6006-44b6-8677-d16ffd433df2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:05:02.437222 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.433226 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13c0b6da-6006-44b6-8677-d16ffd433df2-istio-podinfo" pod "13c0b6da-6006-44b6-8677-d16ffd433df2" (UID: "13c0b6da-6006-44b6-8677-d16ffd433df2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:05:02.444041 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.440228 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13c0b6da-6006-44b6-8677-d16ffd433df2-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:02.541734 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.541643 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13c0b6da-6006-44b6-8677-d16ffd433df2-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:02.541734 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.541680 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gqqg\" (UniqueName: \"kubernetes.io/projected/13c0b6da-6006-44b6-8677-d16ffd433df2-kube-api-access-7gqqg\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:02.541734 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.541693 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13c0b6da-6006-44b6-8677-d16ffd433df2-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:02.541734 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.541702 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13c0b6da-6006-44b6-8677-d16ffd433df2-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:02.541734 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.541711 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13c0b6da-6006-44b6-8677-d16ffd433df2-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:02.541734 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.541723 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c0b6da-6006-44b6-8677-d16ffd433df2-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:02.541734 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.541737 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13c0b6da-6006-44b6-8677-d16ffd433df2-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:02.820657 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.820574 6841 generic.go:356] "Generic (PLEG): container finished" podID="13c0b6da-6006-44b6-8677-d16ffd433df2" containerID="86763a7e5a59fbcc1a8a4cb605298a0690bcae89ea0b98a50adbd1909be7b847" exitCode=0 Oct 31 13:05:02.820657 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.820629 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" Oct 31 13:05:02.820657 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.820635 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" event={"ID":"13c0b6da-6006-44b6-8677-d16ffd433df2","Type":"ContainerDied","Data":"86763a7e5a59fbcc1a8a4cb605298a0690bcae89ea0b98a50adbd1909be7b847"} Oct 31 13:05:02.820875 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.820699 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6" event={"ID":"13c0b6da-6006-44b6-8677-d16ffd433df2","Type":"ContainerDied","Data":"a86ee5435d933dba7cbadcb64dd181a23ddd6affda06ff73154217205ded0c57"} Oct 31 13:05:02.820875 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.820725 6841 scope.go:122] "RemoveContainer" containerID="86763a7e5a59fbcc1a8a4cb605298a0690bcae89ea0b98a50adbd1909be7b847" Oct 31 13:05:02.832094 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.832076 6841 scope.go:122] "RemoveContainer" containerID="4973ccb4472adf76b2d03bb79267b4580ef815a6276c6e778ad6376100b09d2d" Oct 31 13:05:02.849376 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.849344 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6"] Oct 31 13:05:02.853298 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.853264 6841 scope.go:122] "RemoveContainer" containerID="86763a7e5a59fbcc1a8a4cb605298a0690bcae89ea0b98a50adbd1909be7b847" Oct 31 13:05:02.853633 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:05:02.853609 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86763a7e5a59fbcc1a8a4cb605298a0690bcae89ea0b98a50adbd1909be7b847\": container with ID starting with 86763a7e5a59fbcc1a8a4cb605298a0690bcae89ea0b98a50adbd1909be7b847 not found: ID does not exist" containerID="86763a7e5a59fbcc1a8a4cb605298a0690bcae89ea0b98a50adbd1909be7b847" Oct 31 13:05:02.853718 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.853641 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86763a7e5a59fbcc1a8a4cb605298a0690bcae89ea0b98a50adbd1909be7b847"} err="failed to get container status \"86763a7e5a59fbcc1a8a4cb605298a0690bcae89ea0b98a50adbd1909be7b847\": rpc error: code = NotFound desc = could not find container \"86763a7e5a59fbcc1a8a4cb605298a0690bcae89ea0b98a50adbd1909be7b847\": container with ID starting with 86763a7e5a59fbcc1a8a4cb605298a0690bcae89ea0b98a50adbd1909be7b847 not found: ID does not exist" Oct 31 13:05:02.853718 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.853661 6841 scope.go:122] "RemoveContainer" containerID="4973ccb4472adf76b2d03bb79267b4580ef815a6276c6e778ad6376100b09d2d" Oct 31 13:05:02.853907 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.853885 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-12056b-openshift-default-85877bbc-sxbf6"] Oct 31 13:05:02.853951 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:05:02.853885 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4973ccb4472adf76b2d03bb79267b4580ef815a6276c6e778ad6376100b09d2d\": container with ID starting with 4973ccb4472adf76b2d03bb79267b4580ef815a6276c6e778ad6376100b09d2d not found: ID does not exist" containerID="4973ccb4472adf76b2d03bb79267b4580ef815a6276c6e778ad6376100b09d2d" Oct 31 13:05:02.853999 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:02.853943 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4973ccb4472adf76b2d03bb79267b4580ef815a6276c6e778ad6376100b09d2d"} err="failed to get container status \"4973ccb4472adf76b2d03bb79267b4580ef815a6276c6e778ad6376100b09d2d\": rpc error: code = NotFound desc = could not find container \"4973ccb4472adf76b2d03bb79267b4580ef815a6276c6e778ad6376100b09d2d\": container with ID starting with 4973ccb4472adf76b2d03bb79267b4580ef815a6276c6e778ad6376100b09d2d not found: ID does not exist" Oct 31 13:05:03.358569 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:03.358528 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:03.644078 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:03.643968 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13c0b6da-6006-44b6-8677-d16ffd433df2" path="/var/lib/kubelet/pods/13c0b6da-6006-44b6-8677-d16ffd433df2/volumes" Oct 31 13:05:04.109740 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:04.109689 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:04.933079 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:04.930661 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:05.412036 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:05.411997 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:05.715280 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:05.715238 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:06.417858 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.417820 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:06.579779 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.579753 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:05:06.615091 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.615063 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64dbdf9c-9faf-48af-be69-c820c44aac9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64dbdf9c-9faf-48af-be69-c820c44aac9f-istio-podinfo\") pod \"64dbdf9c-9faf-48af-be69-c820c44aac9f\" (UID: \"64dbdf9c-9faf-48af-be69-c820c44aac9f\") " Oct 31 13:05:06.615268 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.615111 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64dbdf9c-9faf-48af-be69-c820c44aac9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64dbdf9c-9faf-48af-be69-c820c44aac9f-credential-socket\") pod \"64dbdf9c-9faf-48af-be69-c820c44aac9f\" (UID: \"64dbdf9c-9faf-48af-be69-c820c44aac9f\") " Oct 31 13:05:06.615268 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.615156 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64dbdf9c-9faf-48af-be69-c820c44aac9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64dbdf9c-9faf-48af-be69-c820c44aac9f-istio-envoy\") pod \"64dbdf9c-9faf-48af-be69-c820c44aac9f\" (UID: \"64dbdf9c-9faf-48af-be69-c820c44aac9f\") " Oct 31 13:05:06.615268 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.615212 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64dbdf9c-9faf-48af-be69-c820c44aac9f-kube-api-access-8c5zn\" (UniqueName: \"kubernetes.io/projected/64dbdf9c-9faf-48af-be69-c820c44aac9f-kube-api-access-8c5zn\") pod \"64dbdf9c-9faf-48af-be69-c820c44aac9f\" (UID: \"64dbdf9c-9faf-48af-be69-c820c44aac9f\") " Oct 31 13:05:06.615407 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.615332 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64dbdf9c-9faf-48af-be69-c820c44aac9f-istio-token\" (UniqueName: \"kubernetes.io/projected/64dbdf9c-9faf-48af-be69-c820c44aac9f-istio-token\") pod \"64dbdf9c-9faf-48af-be69-c820c44aac9f\" (UID: \"64dbdf9c-9faf-48af-be69-c820c44aac9f\") " Oct 31 13:05:06.615407 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.615391 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64dbdf9c-9faf-48af-be69-c820c44aac9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64dbdf9c-9faf-48af-be69-c820c44aac9f-istio-data\") pod \"64dbdf9c-9faf-48af-be69-c820c44aac9f\" (UID: \"64dbdf9c-9faf-48af-be69-c820c44aac9f\") " Oct 31 13:05:06.615525 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.615441 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64dbdf9c-9faf-48af-be69-c820c44aac9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64dbdf9c-9faf-48af-be69-c820c44aac9f-workload-certs\") pod \"64dbdf9c-9faf-48af-be69-c820c44aac9f\" (UID: \"64dbdf9c-9faf-48af-be69-c820c44aac9f\") " Oct 31 13:05:06.615525 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.615465 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64dbdf9c-9faf-48af-be69-c820c44aac9f-credential-socket" pod "64dbdf9c-9faf-48af-be69-c820c44aac9f" (UID: "64dbdf9c-9faf-48af-be69-c820c44aac9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:05:06.615525 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.615496 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64dbdf9c-9faf-48af-be69-c820c44aac9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64dbdf9c-9faf-48af-be69-c820c44aac9f-workload-socket\") pod \"64dbdf9c-9faf-48af-be69-c820c44aac9f\" (UID: \"64dbdf9c-9faf-48af-be69-c820c44aac9f\") " Oct 31 13:05:06.615678 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.615534 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64dbdf9c-9faf-48af-be69-c820c44aac9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64dbdf9c-9faf-48af-be69-c820c44aac9f-istiod-ca-cert\") pod \"64dbdf9c-9faf-48af-be69-c820c44aac9f\" (UID: \"64dbdf9c-9faf-48af-be69-c820c44aac9f\") " Oct 31 13:05:06.615718 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.615685 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64dbdf9c-9faf-48af-be69-c820c44aac9f-istio-data" pod "64dbdf9c-9faf-48af-be69-c820c44aac9f" (UID: "64dbdf9c-9faf-48af-be69-c820c44aac9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:05:06.615881 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.615843 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64dbdf9c-9faf-48af-be69-c820c44aac9f-workload-certs" pod "64dbdf9c-9faf-48af-be69-c820c44aac9f" (UID: "64dbdf9c-9faf-48af-be69-c820c44aac9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:05:06.615942 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.615864 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64dbdf9c-9faf-48af-be69-c820c44aac9f-workload-socket" pod "64dbdf9c-9faf-48af-be69-c820c44aac9f" (UID: "64dbdf9c-9faf-48af-be69-c820c44aac9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:05:06.616205 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.616178 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64dbdf9c-9faf-48af-be69-c820c44aac9f-istiod-ca-cert" pod "64dbdf9c-9faf-48af-be69-c820c44aac9f" (UID: "64dbdf9c-9faf-48af-be69-c820c44aac9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:05:06.616326 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.616307 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64dbdf9c-9faf-48af-be69-c820c44aac9f-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:06.616326 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.616322 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64dbdf9c-9faf-48af-be69-c820c44aac9f-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:06.616442 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.616332 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64dbdf9c-9faf-48af-be69-c820c44aac9f-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:06.616442 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.616341 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64dbdf9c-9faf-48af-be69-c820c44aac9f-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:06.616442 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.616353 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64dbdf9c-9faf-48af-be69-c820c44aac9f-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:06.617603 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.617576 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64dbdf9c-9faf-48af-be69-c820c44aac9f-istio-podinfo" pod "64dbdf9c-9faf-48af-be69-c820c44aac9f" (UID: "64dbdf9c-9faf-48af-be69-c820c44aac9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:05:06.617761 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.617743 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64dbdf9c-9faf-48af-be69-c820c44aac9f-kube-api-access-8c5zn" pod "64dbdf9c-9faf-48af-be69-c820c44aac9f" (UID: "64dbdf9c-9faf-48af-be69-c820c44aac9f"). InnerVolumeSpecName "kube-api-access-8c5zn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:05:06.618164 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.618144 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64dbdf9c-9faf-48af-be69-c820c44aac9f-istio-token" pod "64dbdf9c-9faf-48af-be69-c820c44aac9f" (UID: "64dbdf9c-9faf-48af-be69-c820c44aac9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:05:06.618470 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.618444 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64dbdf9c-9faf-48af-be69-c820c44aac9f-istio-envoy" pod "64dbdf9c-9faf-48af-be69-c820c44aac9f" (UID: "64dbdf9c-9faf-48af-be69-c820c44aac9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:05:06.717728 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.717686 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64dbdf9c-9faf-48af-be69-c820c44aac9f-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:06.717728 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.717726 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64dbdf9c-9faf-48af-be69-c820c44aac9f-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:06.717971 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.717742 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64dbdf9c-9faf-48af-be69-c820c44aac9f-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:06.717971 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.717755 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c5zn\" (UniqueName: \"kubernetes.io/projected/64dbdf9c-9faf-48af-be69-c820c44aac9f-kube-api-access-8c5zn\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:06.786121 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:05:06.786074 6841 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb09c0fc_2974_4e9a_8c0e_f9edd5528c49.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f31282_bb5f_426c_a167_af1e4bb39696.slice/crio-b305cb4830c27d48d2700a5b4965cd5222c5287ffe873e622dd5dfe559664f80\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb09c0fc_2974_4e9a_8c0e_f9edd5528c49.slice/crio-213f0df112b21dce17e584231f0dff0b9fab9afad5f9126ba05b5fc42713b374\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f31282_bb5f_426c_a167_af1e4bb39696.slice\": RecentStats: unable to find data in memory cache]" Oct 31 13:05:06.853949 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.853872 6841 generic.go:356] "Generic (PLEG): container finished" podID="64dbdf9c-9faf-48af-be69-c820c44aac9f" containerID="fc464ae15786892df57839c9378da40b0680a222196893ac50dfa544394b9a56" exitCode=0 Oct 31 13:05:06.853949 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.853924 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" event={"ID":"64dbdf9c-9faf-48af-be69-c820c44aac9f","Type":"ContainerDied","Data":"fc464ae15786892df57839c9378da40b0680a222196893ac50dfa544394b9a56"} Oct 31 13:05:06.853949 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.853942 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" Oct 31 13:05:06.854291 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.854002 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq" event={"ID":"64dbdf9c-9faf-48af-be69-c820c44aac9f","Type":"ContainerDied","Data":"f30f7193880df430c8e0c8f0507cae8f51167b18bad7293c35c012d19041a949"} Oct 31 13:05:06.854291 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.854025 6841 scope.go:122] "RemoveContainer" containerID="fc464ae15786892df57839c9378da40b0680a222196893ac50dfa544394b9a56" Oct 31 13:05:06.865250 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.865233 6841 scope.go:122] "RemoveContainer" containerID="b1523f218f8367168b69655ff93caea63b9eab3413d6a6a37850a3e8f329c552" Oct 31 13:05:06.874437 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.874416 6841 scope.go:122] "RemoveContainer" containerID="fc464ae15786892df57839c9378da40b0680a222196893ac50dfa544394b9a56" Oct 31 13:05:06.874700 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:05:06.874679 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc464ae15786892df57839c9378da40b0680a222196893ac50dfa544394b9a56\": container with ID starting with fc464ae15786892df57839c9378da40b0680a222196893ac50dfa544394b9a56 not found: ID does not exist" containerID="fc464ae15786892df57839c9378da40b0680a222196893ac50dfa544394b9a56" Oct 31 13:05:06.874770 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.874712 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc464ae15786892df57839c9378da40b0680a222196893ac50dfa544394b9a56"} err="failed to get container status \"fc464ae15786892df57839c9378da40b0680a222196893ac50dfa544394b9a56\": rpc error: code = NotFound desc = could not find container \"fc464ae15786892df57839c9378da40b0680a222196893ac50dfa544394b9a56\": container with ID starting with fc464ae15786892df57839c9378da40b0680a222196893ac50dfa544394b9a56 not found: ID does not exist" Oct 31 13:05:06.874770 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.874736 6841 scope.go:122] "RemoveContainer" containerID="b1523f218f8367168b69655ff93caea63b9eab3413d6a6a37850a3e8f329c552" Oct 31 13:05:06.877021 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:05:06.875227 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1523f218f8367168b69655ff93caea63b9eab3413d6a6a37850a3e8f329c552\": container with ID starting with b1523f218f8367168b69655ff93caea63b9eab3413d6a6a37850a3e8f329c552 not found: ID does not exist" containerID="b1523f218f8367168b69655ff93caea63b9eab3413d6a6a37850a3e8f329c552" Oct 31 13:05:06.877021 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.875690 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1523f218f8367168b69655ff93caea63b9eab3413d6a6a37850a3e8f329c552"} err="failed to get container status \"b1523f218f8367168b69655ff93caea63b9eab3413d6a6a37850a3e8f329c552\": rpc error: code = NotFound desc = could not find container \"b1523f218f8367168b69655ff93caea63b9eab3413d6a6a37850a3e8f329c552\": container with ID starting with b1523f218f8367168b69655ff93caea63b9eab3413d6a6a37850a3e8f329c552 not found: ID does not exist" Oct 31 13:05:06.879607 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.879583 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq"] Oct 31 13:05:06.885253 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:06.885234 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-fa2079-openshift-default-5967d769b5-x4tnq"] Oct 31 13:05:07.132828 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:07.132739 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:07.584269 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:05:07.584228 6841 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb09c0fc_2974_4e9a_8c0e_f9edd5528c49.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f31282_bb5f_426c_a167_af1e4bb39696.slice/crio-b305cb4830c27d48d2700a5b4965cd5222c5287ffe873e622dd5dfe559664f80\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f31282_bb5f_426c_a167_af1e4bb39696.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb09c0fc_2974_4e9a_8c0e_f9edd5528c49.slice/crio-213f0df112b21dce17e584231f0dff0b9fab9afad5f9126ba05b5fc42713b374\": RecentStats: unable to find data in memory cache]" Oct 31 13:05:07.642771 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:07.642741 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64dbdf9c-9faf-48af-be69-c820c44aac9f" path="/var/lib/kubelet/pods/64dbdf9c-9faf-48af-be69-c820c44aac9f/volumes" Oct 31 13:05:07.930924 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:07.930838 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:08.639959 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:08.639928 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655"] Oct 31 13:05:08.640435 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:08.640187 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655" podUID="0934f64d-5734-4600-b7af-15025a56955e" containerName="maas-api" containerID="cri-o://ca3ab5142234a3883affad6c506eef660b156657b538fb7a93bcefc266957a90" gracePeriod=30 Oct 31 13:05:08.834385 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:08.834291 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:08.921528 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:08.921499 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655" Oct 31 13:05:08.923474 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:08.923449 6841 generic.go:356] "Generic (PLEG): container finished" podID="0934f64d-5734-4600-b7af-15025a56955e" containerID="ca3ab5142234a3883affad6c506eef660b156657b538fb7a93bcefc266957a90" exitCode=0 Oct 31 13:05:08.923602 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:08.923492 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655" Oct 31 13:05:08.923602 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:08.923529 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655" event={"ID":"0934f64d-5734-4600-b7af-15025a56955e","Type":"ContainerDied","Data":"ca3ab5142234a3883affad6c506eef660b156657b538fb7a93bcefc266957a90"} Oct 31 13:05:08.923602 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:08.923592 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655" event={"ID":"0934f64d-5734-4600-b7af-15025a56955e","Type":"ContainerDied","Data":"ff4539823f71b234776f33830f83b6ad073d29dd06b1a6c7ee8338b8874a5716"} Oct 31 13:05:08.923745 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:08.923619 6841 scope.go:122] "RemoveContainer" containerID="ca3ab5142234a3883affad6c506eef660b156657b538fb7a93bcefc266957a90" Oct 31 13:05:08.933903 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:08.933882 6841 scope.go:122] "RemoveContainer" containerID="ca3ab5142234a3883affad6c506eef660b156657b538fb7a93bcefc266957a90" Oct 31 13:05:08.934201 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:05:08.934175 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca3ab5142234a3883affad6c506eef660b156657b538fb7a93bcefc266957a90\": container with ID starting with ca3ab5142234a3883affad6c506eef660b156657b538fb7a93bcefc266957a90 not found: ID does not exist" containerID="ca3ab5142234a3883affad6c506eef660b156657b538fb7a93bcefc266957a90" Oct 31 13:05:08.934299 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:08.934207 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca3ab5142234a3883affad6c506eef660b156657b538fb7a93bcefc266957a90"} err="failed to get container status \"ca3ab5142234a3883affad6c506eef660b156657b538fb7a93bcefc266957a90\": rpc error: code = NotFound desc = could not find container \"ca3ab5142234a3883affad6c506eef660b156657b538fb7a93bcefc266957a90\": container with ID starting with ca3ab5142234a3883affad6c506eef660b156657b538fb7a93bcefc266957a90 not found: ID does not exist" Oct 31 13:05:08.998096 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:08.957558 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0934f64d-5734-4600-b7af-15025a56955e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0934f64d-5734-4600-b7af-15025a56955e-metrics-certs\") pod \"0934f64d-5734-4600-b7af-15025a56955e\" (UID: \"0934f64d-5734-4600-b7af-15025a56955e\") " Oct 31 13:05:08.998096 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:08.957617 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0934f64d-5734-4600-b7af-15025a56955e-kube-api-access-r8lx9\" (UniqueName: \"kubernetes.io/projected/0934f64d-5734-4600-b7af-15025a56955e-kube-api-access-r8lx9\") pod \"0934f64d-5734-4600-b7af-15025a56955e\" (UID: \"0934f64d-5734-4600-b7af-15025a56955e\") " Oct 31 13:05:08.998096 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:08.957689 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0934f64d-5734-4600-b7af-15025a56955e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0934f64d-5734-4600-b7af-15025a56955e-openshift-service-ca\") pod \"0934f64d-5734-4600-b7af-15025a56955e\" (UID: \"0934f64d-5734-4600-b7af-15025a56955e\") " Oct 31 13:05:08.998096 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:08.957721 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0934f64d-5734-4600-b7af-15025a56955e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0934f64d-5734-4600-b7af-15025a56955e-maas-api-tls\") pod \"0934f64d-5734-4600-b7af-15025a56955e\" (UID: \"0934f64d-5734-4600-b7af-15025a56955e\") " Oct 31 13:05:08.998096 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:08.960956 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0934f64d-5734-4600-b7af-15025a56955e-openshift-service-ca" pod "0934f64d-5734-4600-b7af-15025a56955e" (UID: "0934f64d-5734-4600-b7af-15025a56955e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:05:08.998786 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:08.998745 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0934f64d-5734-4600-b7af-15025a56955e-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:09.039260 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:09.033098 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0934f64d-5734-4600-b7af-15025a56955e-metrics-certs" pod "0934f64d-5734-4600-b7af-15025a56955e" (UID: "0934f64d-5734-4600-b7af-15025a56955e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:05:09.039260 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:09.034057 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0934f64d-5734-4600-b7af-15025a56955e-kube-api-access-r8lx9" pod "0934f64d-5734-4600-b7af-15025a56955e" (UID: "0934f64d-5734-4600-b7af-15025a56955e"). InnerVolumeSpecName "kube-api-access-r8lx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:05:09.039260 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:09.034177 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0934f64d-5734-4600-b7af-15025a56955e-maas-api-tls" pod "0934f64d-5734-4600-b7af-15025a56955e" (UID: "0934f64d-5734-4600-b7af-15025a56955e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:05:09.057464 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:09.056248 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:09.118550 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:09.118512 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0934f64d-5734-4600-b7af-15025a56955e-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:09.118550 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:09.118549 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8lx9\" (UniqueName: \"kubernetes.io/projected/0934f64d-5734-4600-b7af-15025a56955e-kube-api-access-r8lx9\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:09.118754 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:09.118565 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0934f64d-5734-4600-b7af-15025a56955e-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:09.249462 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:09.249412 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655"] Oct 31 13:05:09.255863 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:09.255832 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c347ff-d44ff4d68-95655"] Oct 31 13:05:09.366754 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:09.366665 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:09.643172 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:09.643098 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0934f64d-5734-4600-b7af-15025a56955e" path="/var/lib/kubelet/pods/0934f64d-5734-4600-b7af-15025a56955e/volumes" Oct 31 13:05:10.230648 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:10.230578 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:10.939727 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:10.939668 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:11.967057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:11.964799 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:12.679263 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:12.679217 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:12.910184 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:12.910145 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:13.124713 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:13.124674 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:13.327908 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:13.327853 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx"] Oct 31 13:05:13.328144 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:13.328116 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" podUID="c0e2ba69-d8b6-4534-8113-5d4c0bfea735" containerName="istio-proxy" containerID="cri-o://c0950330646dbb70d32459735c2a3025ef02ace87c66094db6b17013c2f15941" gracePeriod=30 Oct 31 13:05:13.576790 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:13.576753 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:13.803824 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:13.803788 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:14.815383 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:14.815339 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:15.028376 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.028343 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:15.327165 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.327130 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2"] Oct 31 13:05:15.328519 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.328486 6841 cpu_manager.go:414] "containerMap: removing container" podUID="64dbdf9c-9faf-48af-be69-c820c44aac9f" containerName="istio-proxy" Oct 31 13:05:15.328519 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.328519 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64dbdf9c-9faf-48af-be69-c820c44aac9f" containerName="istio-proxy" Oct 31 13:05:15.328731 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.328544 6841 cpu_manager.go:414] "containerMap: removing container" podUID="0934f64d-5734-4600-b7af-15025a56955e" containerName="maas-api" Oct 31 13:05:15.328731 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.328552 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0934f64d-5734-4600-b7af-15025a56955e" containerName="maas-api" Oct 31 13:05:15.328731 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.328569 6841 cpu_manager.go:414] "containerMap: removing container" podUID="13c0b6da-6006-44b6-8677-d16ffd433df2" containerName="istio-proxy" Oct 31 13:05:15.328731 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.328576 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13c0b6da-6006-44b6-8677-d16ffd433df2" containerName="istio-proxy" Oct 31 13:05:15.328731 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.328588 6841 cpu_manager.go:414] "containerMap: removing container" podUID="faafb6b6-9a5a-4a60-a327-1629d2c49868" containerName="payload-pre-processing" Oct 31 13:05:15.328731 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.328596 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faafb6b6-9a5a-4a60-a327-1629d2c49868" containerName="payload-pre-processing" Oct 31 13:05:15.328731 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.328634 6841 cpu_manager.go:414] "containerMap: removing container" podUID="6a4859e8-5b79-4f2a-afe6-0b26a64cce08" containerName="payload-processing" Oct 31 13:05:15.328731 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.328642 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a4859e8-5b79-4f2a-afe6-0b26a64cce08" containerName="payload-processing" Oct 31 13:05:15.328731 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.328669 6841 cpu_manager.go:414] "containerMap: removing container" podUID="13c0b6da-6006-44b6-8677-d16ffd433df2" containerName="istio-proxy" Oct 31 13:05:15.328731 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.328676 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13c0b6da-6006-44b6-8677-d16ffd433df2" containerName="istio-proxy" Oct 31 13:05:15.328731 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.328706 6841 cpu_manager.go:414] "containerMap: removing container" podUID="3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f" containerName="maas-api" Oct 31 13:05:15.328731 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.328714 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f" containerName="maas-api" Oct 31 13:05:15.329183 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.328850 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ee1dd0e-bd6c-4c5c-a064-e1c9b814e82f" containerName="maas-api" Oct 31 13:05:15.329183 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.328869 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="64dbdf9c-9faf-48af-be69-c820c44aac9f" containerName="istio-proxy" Oct 31 13:05:15.329183 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.328881 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="0934f64d-5734-4600-b7af-15025a56955e" containerName="maas-api" Oct 31 13:05:15.329183 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.328899 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="13c0b6da-6006-44b6-8677-d16ffd433df2" containerName="istio-proxy" Oct 31 13:05:15.329183 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.328909 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="faafb6b6-9a5a-4a60-a327-1629d2c49868" containerName="payload-pre-processing" Oct 31 13:05:15.329183 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.328924 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a4859e8-5b79-4f2a-afe6-0b26a64cce08" containerName="payload-processing" Oct 31 13:05:15.329183 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.328934 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="64dbdf9c-9faf-48af-be69-c820c44aac9f" containerName="istio-proxy" Oct 31 13:05:15.334801 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.334776 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:05:15.341248 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.337016 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2"] Oct 31 13:05:15.431352 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.431314 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-7e08dc-openshift-default-dockercfg-6s755\"" Oct 31 13:05:15.461307 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.461260 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ebddec1-8fab-4709-84de-f5d99a9112d0-istiod-ca-cert\") pod \"e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2\" (UID: \"5ebddec1-8fab-4709-84de-f5d99a9112d0\") " pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:05:15.461835 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.461807 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ebddec1-8fab-4709-84de-f5d99a9112d0-istio-data\") pod \"e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2\" (UID: \"5ebddec1-8fab-4709-84de-f5d99a9112d0\") " pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:05:15.474000 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.471159 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ebddec1-8fab-4709-84de-f5d99a9112d0-credential-socket\") pod \"e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2\" (UID: \"5ebddec1-8fab-4709-84de-f5d99a9112d0\") " pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:05:15.474000 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.471366 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ebddec1-8fab-4709-84de-f5d99a9112d0-workload-socket\") pod \"e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2\" (UID: \"5ebddec1-8fab-4709-84de-f5d99a9112d0\") " pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:05:15.474000 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.471416 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ebddec1-8fab-4709-84de-f5d99a9112d0-workload-certs\") pod \"e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2\" (UID: \"5ebddec1-8fab-4709-84de-f5d99a9112d0\") " pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:05:15.474000 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.471483 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6dxr\" (UniqueName: \"kubernetes.io/projected/5ebddec1-8fab-4709-84de-f5d99a9112d0-kube-api-access-s6dxr\") pod \"e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2\" (UID: \"5ebddec1-8fab-4709-84de-f5d99a9112d0\") " pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:05:15.474000 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.471615 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ebddec1-8fab-4709-84de-f5d99a9112d0-istio-envoy\") pod \"e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2\" (UID: \"5ebddec1-8fab-4709-84de-f5d99a9112d0\") " pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:05:15.474000 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.471659 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ebddec1-8fab-4709-84de-f5d99a9112d0-istio-token\") pod \"e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2\" (UID: \"5ebddec1-8fab-4709-84de-f5d99a9112d0\") " pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:05:15.474000 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.471722 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ebddec1-8fab-4709-84de-f5d99a9112d0-istio-podinfo\") pod \"e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2\" (UID: \"5ebddec1-8fab-4709-84de-f5d99a9112d0\") " pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:05:15.491729 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.491682 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:15.573935 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.573896 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6dxr\" (UniqueName: \"kubernetes.io/projected/5ebddec1-8fab-4709-84de-f5d99a9112d0-kube-api-access-s6dxr\") pod \"e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2\" (UID: \"5ebddec1-8fab-4709-84de-f5d99a9112d0\") " pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:05:15.574431 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.574394 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ebddec1-8fab-4709-84de-f5d99a9112d0-istio-envoy\") pod \"e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2\" (UID: \"5ebddec1-8fab-4709-84de-f5d99a9112d0\") " pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:05:15.574611 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.574596 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ebddec1-8fab-4709-84de-f5d99a9112d0-istio-token\") pod \"e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2\" (UID: \"5ebddec1-8fab-4709-84de-f5d99a9112d0\") " pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:05:15.574751 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.574736 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ebddec1-8fab-4709-84de-f5d99a9112d0-istio-podinfo\") pod \"e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2\" (UID: \"5ebddec1-8fab-4709-84de-f5d99a9112d0\") " pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:05:15.574937 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.574903 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ebddec1-8fab-4709-84de-f5d99a9112d0-istiod-ca-cert\") pod \"e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2\" (UID: \"5ebddec1-8fab-4709-84de-f5d99a9112d0\") " pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:05:15.575210 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.575180 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ebddec1-8fab-4709-84de-f5d99a9112d0-istio-data\") pod \"e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2\" (UID: \"5ebddec1-8fab-4709-84de-f5d99a9112d0\") " pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:05:15.575433 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.575414 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ebddec1-8fab-4709-84de-f5d99a9112d0-credential-socket\") pod \"e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2\" (UID: \"5ebddec1-8fab-4709-84de-f5d99a9112d0\") " pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:05:15.575641 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.575626 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ebddec1-8fab-4709-84de-f5d99a9112d0-workload-socket\") pod \"e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2\" (UID: \"5ebddec1-8fab-4709-84de-f5d99a9112d0\") " pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:05:15.575777 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.575761 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ebddec1-8fab-4709-84de-f5d99a9112d0-workload-certs\") pod \"e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2\" (UID: \"5ebddec1-8fab-4709-84de-f5d99a9112d0\") " pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:05:15.580349 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.579493 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ebddec1-8fab-4709-84de-f5d99a9112d0-istio-data\") pod \"e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2\" (UID: \"5ebddec1-8fab-4709-84de-f5d99a9112d0\") " pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:05:15.580349 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.579755 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ebddec1-8fab-4709-84de-f5d99a9112d0-credential-socket\") pod \"e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2\" (UID: \"5ebddec1-8fab-4709-84de-f5d99a9112d0\") " pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:05:15.580349 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.580025 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ebddec1-8fab-4709-84de-f5d99a9112d0-workload-socket\") pod \"e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2\" (UID: \"5ebddec1-8fab-4709-84de-f5d99a9112d0\") " pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:05:15.581351 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.581313 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ebddec1-8fab-4709-84de-f5d99a9112d0-workload-certs\") pod \"e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2\" (UID: \"5ebddec1-8fab-4709-84de-f5d99a9112d0\") " pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:05:15.581574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.581553 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ebddec1-8fab-4709-84de-f5d99a9112d0-istiod-ca-cert\") pod \"e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2\" (UID: \"5ebddec1-8fab-4709-84de-f5d99a9112d0\") " pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:05:15.593235 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.591438 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ebddec1-8fab-4709-84de-f5d99a9112d0-istio-envoy\") pod \"e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2\" (UID: \"5ebddec1-8fab-4709-84de-f5d99a9112d0\") " pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:05:15.594206 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.594183 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ebddec1-8fab-4709-84de-f5d99a9112d0-istio-podinfo\") pod \"e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2\" (UID: \"5ebddec1-8fab-4709-84de-f5d99a9112d0\") " pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:05:15.635237 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.634320 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ebddec1-8fab-4709-84de-f5d99a9112d0-istio-token\") pod \"e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2\" (UID: \"5ebddec1-8fab-4709-84de-f5d99a9112d0\") " pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:05:15.637636 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.637566 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6dxr\" (UniqueName: \"kubernetes.io/projected/5ebddec1-8fab-4709-84de-f5d99a9112d0-kube-api-access-s6dxr\") pod \"e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2\" (UID: \"5ebddec1-8fab-4709-84de-f5d99a9112d0\") " pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:05:15.664194 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.663248 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:05:15.755510 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:15.755479 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:16.077203 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:16.077171 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2"] Oct 31 13:05:16.082410 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:05:16.082378 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ebddec1_8fab_4709_84de_f5d99a9112d0.slice/crio-898819c3b72b1ba1a854ab5269f3b7359d540f886c0a018c0eb33032b927f142 WatchSource:0}: Error finding container 898819c3b72b1ba1a854ab5269f3b7359d540f886c0a018c0eb33032b927f142: Status 404 returned error can't find the container with id 898819c3b72b1ba1a854ab5269f3b7359d540f886c0a018c0eb33032b927f142 Oct 31 13:05:16.084950 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:16.084915 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:05:16.085092 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:16.085069 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:05:16.085140 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:16.085120 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:05:16.099547 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:16.099518 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:16.517644 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:16.517601 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:17.014342 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:17.014307 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" event={"ID":"5ebddec1-8fab-4709-84de-f5d99a9112d0","Type":"ContainerStarted","Data":"d77b8f1b938e25220d58eb6715bbe7d0fbe7997a420a829dcd16138ef40026fd"} Oct 31 13:05:17.014525 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:17.014356 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" event={"ID":"5ebddec1-8fab-4709-84de-f5d99a9112d0","Type":"ContainerStarted","Data":"898819c3b72b1ba1a854ab5269f3b7359d540f886c0a018c0eb33032b927f142"} Oct 31 13:05:17.049813 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:17.049765 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" podStartSLOduration=2.049750176 podStartE2EDuration="2.049750176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:05:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:05:17.048853419 +0000 UTC m=+1963.531976322" watchObservedRunningTime="2026-10-31 13:05:17.049750176 +0000 UTC m=+1963.532873076" Oct 31 13:05:17.338120 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:17.338029 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:17.663787 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:17.663702 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:05:17.667620 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:05:17.667585 6841 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb09c0fc_2974_4e9a_8c0e_f9edd5528c49.slice/crio-213f0df112b21dce17e584231f0dff0b9fab9afad5f9126ba05b5fc42713b374\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f31282_bb5f_426c_a167_af1e4bb39696.slice/crio-b305cb4830c27d48d2700a5b4965cd5222c5287ffe873e622dd5dfe559664f80\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb09c0fc_2974_4e9a_8c0e_f9edd5528c49.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f31282_bb5f_426c_a167_af1e4bb39696.slice\": RecentStats: unable to find data in memory cache]" Oct 31 13:05:17.670463 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:17.670438 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:05:18.021234 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:18.021205 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:05:18.022166 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:18.022144 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:05:18.207004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:18.205208 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:18.608818 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:18.608755 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:05:18.743930 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:18.743895 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-istiod-ca-cert\") pod \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\" (UID: \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\") " Oct 31 13:05:18.744165 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:18.743960 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-workload-certs\") pod \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\" (UID: \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\") " Oct 31 13:05:18.744165 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:18.744031 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-istio-envoy\") pod \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\" (UID: \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\") " Oct 31 13:05:18.744165 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:18.744074 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-kube-api-access-lt5mc\" (UniqueName: \"kubernetes.io/projected/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-kube-api-access-lt5mc\") pod \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\" (UID: \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\") " Oct 31 13:05:18.744165 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:18.744099 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-credential-socket\") pod \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\" (UID: \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\") " Oct 31 13:05:18.744165 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:18.744131 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-istio-token\" (UniqueName: \"kubernetes.io/projected/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-istio-token\") pod \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\" (UID: \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\") " Oct 31 13:05:18.744464 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:18.744206 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-istio-data\") pod \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\" (UID: \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\") " Oct 31 13:05:18.744464 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:18.744255 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-istio-podinfo\") pod \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\" (UID: \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\") " Oct 31 13:05:18.744464 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:18.744326 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-workload-certs" pod "c0e2ba69-d8b6-4534-8113-5d4c0bfea735" (UID: "c0e2ba69-d8b6-4534-8113-5d4c0bfea735"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:05:18.744464 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:18.744381 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-workload-socket\") pod \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\" (UID: \"c0e2ba69-d8b6-4534-8113-5d4c0bfea735\") " Oct 31 13:05:18.744464 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:18.744395 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-istiod-ca-cert" pod "c0e2ba69-d8b6-4534-8113-5d4c0bfea735" (UID: "c0e2ba69-d8b6-4534-8113-5d4c0bfea735"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:05:18.744704 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:18.744498 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-istio-data" pod "c0e2ba69-d8b6-4534-8113-5d4c0bfea735" (UID: "c0e2ba69-d8b6-4534-8113-5d4c0bfea735"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:05:18.744704 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:18.744562 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-credential-socket" pod "c0e2ba69-d8b6-4534-8113-5d4c0bfea735" (UID: "c0e2ba69-d8b6-4534-8113-5d4c0bfea735"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:05:18.744813 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:18.744784 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-workload-socket" pod "c0e2ba69-d8b6-4534-8113-5d4c0bfea735" (UID: "c0e2ba69-d8b6-4534-8113-5d4c0bfea735"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:05:18.745418 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:18.745396 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:18.745544 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:18.745418 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:18.745544 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:18.745449 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:18.745544 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:18.745465 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:18.745544 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:18.745480 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:18.746683 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:18.746661 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-istio-podinfo" pod "c0e2ba69-d8b6-4534-8113-5d4c0bfea735" (UID: "c0e2ba69-d8b6-4534-8113-5d4c0bfea735"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:05:18.746956 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:18.746941 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-istio-envoy" pod "c0e2ba69-d8b6-4534-8113-5d4c0bfea735" (UID: "c0e2ba69-d8b6-4534-8113-5d4c0bfea735"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:05:18.747328 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:18.747300 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-kube-api-access-lt5mc" pod "c0e2ba69-d8b6-4534-8113-5d4c0bfea735" (UID: "c0e2ba69-d8b6-4534-8113-5d4c0bfea735"). InnerVolumeSpecName "kube-api-access-lt5mc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:05:18.747427 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:18.747341 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-istio-token" pod "c0e2ba69-d8b6-4534-8113-5d4c0bfea735" (UID: "c0e2ba69-d8b6-4534-8113-5d4c0bfea735"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:05:18.846639 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:18.846607 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:18.846639 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:18.846636 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt5mc\" (UniqueName: \"kubernetes.io/projected/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-kube-api-access-lt5mc\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:18.846639 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:18.846647 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:18.846900 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:18.846656 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e2ba69-d8b6-4534-8113-5d4c0bfea735-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:19.030052 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:19.030011 6841 generic.go:356] "Generic (PLEG): container finished" podID="c0e2ba69-d8b6-4534-8113-5d4c0bfea735" containerID="c0950330646dbb70d32459735c2a3025ef02ace87c66094db6b17013c2f15941" exitCode=0 Oct 31 13:05:19.030223 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:19.030080 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" Oct 31 13:05:19.030223 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:19.030079 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" event={"ID":"c0e2ba69-d8b6-4534-8113-5d4c0bfea735","Type":"ContainerDied","Data":"c0950330646dbb70d32459735c2a3025ef02ace87c66094db6b17013c2f15941"} Oct 31 13:05:19.030223 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:19.030176 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx" event={"ID":"c0e2ba69-d8b6-4534-8113-5d4c0bfea735","Type":"ContainerDied","Data":"938ef084b8d9e10e20022fe0210e06541cb04bbf44a567bd2bb95b21cee6b92a"} Oct 31 13:05:19.030223 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:19.030207 6841 scope.go:122] "RemoveContainer" containerID="c0950330646dbb70d32459735c2a3025ef02ace87c66094db6b17013c2f15941" Oct 31 13:05:19.044034 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:19.044013 6841 scope.go:122] "RemoveContainer" containerID="12a87e9dff5234ec8a142efacca456a19992230e502dc895e65c3cdbbf2f16b9" Oct 31 13:05:19.053442 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:19.053423 6841 scope.go:122] "RemoveContainer" containerID="c0950330646dbb70d32459735c2a3025ef02ace87c66094db6b17013c2f15941" Oct 31 13:05:19.053732 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:05:19.053707 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0950330646dbb70d32459735c2a3025ef02ace87c66094db6b17013c2f15941\": container with ID starting with c0950330646dbb70d32459735c2a3025ef02ace87c66094db6b17013c2f15941 not found: ID does not exist" containerID="c0950330646dbb70d32459735c2a3025ef02ace87c66094db6b17013c2f15941" Oct 31 13:05:19.053781 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:19.053745 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0950330646dbb70d32459735c2a3025ef02ace87c66094db6b17013c2f15941"} err="failed to get container status \"c0950330646dbb70d32459735c2a3025ef02ace87c66094db6b17013c2f15941\": rpc error: code = NotFound desc = could not find container \"c0950330646dbb70d32459735c2a3025ef02ace87c66094db6b17013c2f15941\": container with ID starting with c0950330646dbb70d32459735c2a3025ef02ace87c66094db6b17013c2f15941 not found: ID does not exist" Oct 31 13:05:19.053781 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:19.053769 6841 scope.go:122] "RemoveContainer" containerID="12a87e9dff5234ec8a142efacca456a19992230e502dc895e65c3cdbbf2f16b9" Oct 31 13:05:19.054026 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:05:19.054007 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12a87e9dff5234ec8a142efacca456a19992230e502dc895e65c3cdbbf2f16b9\": container with ID starting with 12a87e9dff5234ec8a142efacca456a19992230e502dc895e65c3cdbbf2f16b9 not found: ID does not exist" containerID="12a87e9dff5234ec8a142efacca456a19992230e502dc895e65c3cdbbf2f16b9" Oct 31 13:05:19.054077 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:19.054032 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12a87e9dff5234ec8a142efacca456a19992230e502dc895e65c3cdbbf2f16b9"} err="failed to get container status \"12a87e9dff5234ec8a142efacca456a19992230e502dc895e65c3cdbbf2f16b9\": rpc error: code = NotFound desc = could not find container \"12a87e9dff5234ec8a142efacca456a19992230e502dc895e65c3cdbbf2f16b9\": container with ID starting with 12a87e9dff5234ec8a142efacca456a19992230e502dc895e65c3cdbbf2f16b9 not found: ID does not exist" Oct 31 13:05:19.065218 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:19.065197 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx"] Oct 31 13:05:19.072248 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:19.072226 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-c347ff-openshift-default-588fd5f6d7-5q8hx"] Oct 31 13:05:19.590584 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:19.590551 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:19.723006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:19.711893 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0e2ba69-d8b6-4534-8113-5d4c0bfea735" path="/var/lib/kubelet/pods/c0e2ba69-d8b6-4534-8113-5d4c0bfea735/volumes" Oct 31 13:05:19.845542 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:19.845455 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:20.034490 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:05:20.034456 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:05:21.081432 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:21.081400 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:21.365447 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:21.365355 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:21.729153 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:05:21.729111 6841 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f31282_bb5f_426c_a167_af1e4bb39696.slice/crio-b305cb4830c27d48d2700a5b4965cd5222c5287ffe873e622dd5dfe559664f80\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb09c0fc_2974_4e9a_8c0e_f9edd5528c49.slice/crio-213f0df112b21dce17e584231f0dff0b9fab9afad5f9126ba05b5fc42713b374\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb09c0fc_2974_4e9a_8c0e_f9edd5528c49.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f31282_bb5f_426c_a167_af1e4bb39696.slice\": RecentStats: unable to find data in memory cache]" Oct 31 13:05:21.918137 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:21.918100 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:22.958581 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:22.958547 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:23.731556 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:23.731519 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:24.607239 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:24.607197 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:25.334645 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:25.334608 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:26.196340 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:26.196307 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:26.918413 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:26.918371 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:27.648842 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:27.648807 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:27.746433 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:05:27.746386 6841 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb09c0fc_2974_4e9a_8c0e_f9edd5528c49.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f31282_bb5f_426c_a167_af1e4bb39696.slice/crio-b305cb4830c27d48d2700a5b4965cd5222c5287ffe873e622dd5dfe559664f80\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f31282_bb5f_426c_a167_af1e4bb39696.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb09c0fc_2974_4e9a_8c0e_f9edd5528c49.slice/crio-213f0df112b21dce17e584231f0dff0b9fab9afad5f9126ba05b5fc42713b374\": RecentStats: unable to find data in memory cache]" Oct 31 13:05:28.537461 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:28.537302 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:29.263687 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:29.263647 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7"] Oct 31 13:05:29.264337 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:29.264294 6841 cpu_manager.go:414] "containerMap: removing container" podUID="64dbdf9c-9faf-48af-be69-c820c44aac9f" containerName="istio-proxy" Oct 31 13:05:29.264337 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:29.264308 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64dbdf9c-9faf-48af-be69-c820c44aac9f" containerName="istio-proxy" Oct 31 13:05:29.264469 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:29.264347 6841 cpu_manager.go:414] "containerMap: removing container" podUID="c0e2ba69-d8b6-4534-8113-5d4c0bfea735" containerName="istio-proxy" Oct 31 13:05:29.264469 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:29.264352 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0e2ba69-d8b6-4534-8113-5d4c0bfea735" containerName="istio-proxy" Oct 31 13:05:29.264469 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:29.264365 6841 cpu_manager.go:414] "containerMap: removing container" podUID="c0e2ba69-d8b6-4534-8113-5d4c0bfea735" containerName="istio-proxy" Oct 31 13:05:29.264469 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:29.264370 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0e2ba69-d8b6-4534-8113-5d4c0bfea735" containerName="istio-proxy" Oct 31 13:05:29.265140 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:29.265114 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="13c0b6da-6006-44b6-8677-d16ffd433df2" containerName="istio-proxy" Oct 31 13:05:29.265267 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:29.265165 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0e2ba69-d8b6-4534-8113-5d4c0bfea735" containerName="istio-proxy" Oct 31 13:05:29.286758 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:29.286724 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7" Oct 31 13:05:29.290058 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:29.289977 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7"] Oct 31 13:05:29.295554 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:29.295525 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-7e08dc-serving-cert\"" Oct 31 13:05:29.351294 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:29.351259 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c548ef73-9218-4068-bcd2-101717e5295f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7\" (UID: \"c548ef73-9218-4068-bcd2-101717e5295f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7" Oct 31 13:05:29.351512 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:29.351489 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7mvs\" (UniqueName: \"kubernetes.io/projected/c548ef73-9218-4068-bcd2-101717e5295f-kube-api-access-d7mvs\") pod \"maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7\" (UID: \"c548ef73-9218-4068-bcd2-101717e5295f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7" Oct 31 13:05:29.351649 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:29.351631 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c548ef73-9218-4068-bcd2-101717e5295f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7\" (UID: \"c548ef73-9218-4068-bcd2-101717e5295f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7" Oct 31 13:05:29.351720 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:29.351700 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c548ef73-9218-4068-bcd2-101717e5295f-metrics-certs\") pod \"maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7\" (UID: \"c548ef73-9218-4068-bcd2-101717e5295f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7" Oct 31 13:05:29.454343 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:29.454298 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7mvs\" (UniqueName: \"kubernetes.io/projected/c548ef73-9218-4068-bcd2-101717e5295f-kube-api-access-d7mvs\") pod \"maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7\" (UID: \"c548ef73-9218-4068-bcd2-101717e5295f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7" Oct 31 13:05:29.455229 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:29.455200 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c548ef73-9218-4068-bcd2-101717e5295f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7\" (UID: \"c548ef73-9218-4068-bcd2-101717e5295f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7" Oct 31 13:05:29.456039 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:29.455960 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c548ef73-9218-4068-bcd2-101717e5295f-metrics-certs\") pod \"maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7\" (UID: \"c548ef73-9218-4068-bcd2-101717e5295f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7" Oct 31 13:05:29.456953 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:29.456931 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c548ef73-9218-4068-bcd2-101717e5295f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7\" (UID: \"c548ef73-9218-4068-bcd2-101717e5295f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7" Oct 31 13:05:29.458224 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:29.457855 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c548ef73-9218-4068-bcd2-101717e5295f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7\" (UID: \"c548ef73-9218-4068-bcd2-101717e5295f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7" Oct 31 13:05:29.459386 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:29.459360 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:29.469734 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:29.469673 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c548ef73-9218-4068-bcd2-101717e5295f-metrics-certs\") pod \"maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7\" (UID: \"c548ef73-9218-4068-bcd2-101717e5295f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7" Oct 31 13:05:29.470465 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:29.470430 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c548ef73-9218-4068-bcd2-101717e5295f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7\" (UID: \"c548ef73-9218-4068-bcd2-101717e5295f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7" Oct 31 13:05:29.472504 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:29.472484 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7mvs\" (UniqueName: \"kubernetes.io/projected/c548ef73-9218-4068-bcd2-101717e5295f-kube-api-access-d7mvs\") pod \"maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7\" (UID: \"c548ef73-9218-4068-bcd2-101717e5295f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7" Oct 31 13:05:29.613568 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:29.613483 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7" Oct 31 13:05:29.677418 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:29.677359 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:29.960882 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:29.957583 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6"] Oct 31 13:05:29.960882 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:29.958945 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0e2ba69-d8b6-4534-8113-5d4c0bfea735" containerName="istio-proxy" Oct 31 13:05:29.966640 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:29.966614 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" Oct 31 13:05:29.970021 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:29.968380 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc"] Oct 31 13:05:29.993064 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:29.984968 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" Oct 31 13:05:29.993064 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:29.990144 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6"] Oct 31 13:05:30.015625 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.004477 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7"] Oct 31 13:05:30.015625 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.007505 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc"] Oct 31 13:05:30.078054 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.078011 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a108df2b-6cb1-499c-aef6-d0d200b580c1-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc\" (UID: \"a108df2b-6cb1-499c-aef6-d0d200b580c1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" Oct 31 13:05:30.078209 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.078071 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a108df2b-6cb1-499c-aef6-d0d200b580c1-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc\" (UID: \"a108df2b-6cb1-499c-aef6-d0d200b580c1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" Oct 31 13:05:30.078209 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.078170 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9vv9\" (UniqueName: \"kubernetes.io/projected/3157a099-6a36-4df2-a4cc-b006777e92b0-kube-api-access-k9vv9\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6\" (UID: \"3157a099-6a36-4df2-a4cc-b006777e92b0\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" Oct 31 13:05:30.078347 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.078244 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3157a099-6a36-4df2-a4cc-b006777e92b0-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6\" (UID: \"3157a099-6a36-4df2-a4cc-b006777e92b0\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" Oct 31 13:05:30.078347 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.078281 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a108df2b-6cb1-499c-aef6-d0d200b580c1-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc\" (UID: \"a108df2b-6cb1-499c-aef6-d0d200b580c1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" Oct 31 13:05:30.078460 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.078378 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3157a099-6a36-4df2-a4cc-b006777e92b0-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6\" (UID: \"3157a099-6a36-4df2-a4cc-b006777e92b0\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" Oct 31 13:05:30.078460 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.078414 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a108df2b-6cb1-499c-aef6-d0d200b580c1-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc\" (UID: \"a108df2b-6cb1-499c-aef6-d0d200b580c1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" Oct 31 13:05:30.078460 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.078442 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3157a099-6a36-4df2-a4cc-b006777e92b0-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6\" (UID: \"3157a099-6a36-4df2-a4cc-b006777e92b0\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" Oct 31 13:05:30.078628 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.078467 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hq49\" (UniqueName: \"kubernetes.io/projected/a108df2b-6cb1-499c-aef6-d0d200b580c1-kube-api-access-7hq49\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc\" (UID: \"a108df2b-6cb1-499c-aef6-d0d200b580c1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" Oct 31 13:05:30.078628 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.078568 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3157a099-6a36-4df2-a4cc-b006777e92b0-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6\" (UID: \"3157a099-6a36-4df2-a4cc-b006777e92b0\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" Oct 31 13:05:30.078736 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.078645 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a108df2b-6cb1-499c-aef6-d0d200b580c1-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc\" (UID: \"a108df2b-6cb1-499c-aef6-d0d200b580c1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" Oct 31 13:05:30.078792 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.078743 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a108df2b-6cb1-499c-aef6-d0d200b580c1-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc\" (UID: \"a108df2b-6cb1-499c-aef6-d0d200b580c1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" Oct 31 13:05:30.078792 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.078772 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a108df2b-6cb1-499c-aef6-d0d200b580c1-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc\" (UID: \"a108df2b-6cb1-499c-aef6-d0d200b580c1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" Oct 31 13:05:30.078906 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.078806 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a108df2b-6cb1-499c-aef6-d0d200b580c1-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc\" (UID: \"a108df2b-6cb1-499c-aef6-d0d200b580c1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" Oct 31 13:05:30.078906 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.078894 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3157a099-6a36-4df2-a4cc-b006777e92b0-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6\" (UID: \"3157a099-6a36-4df2-a4cc-b006777e92b0\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" Oct 31 13:05:30.079294 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.079020 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3157a099-6a36-4df2-a4cc-b006777e92b0-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6\" (UID: \"3157a099-6a36-4df2-a4cc-b006777e92b0\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" Oct 31 13:05:30.079294 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.079076 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3157a099-6a36-4df2-a4cc-b006777e92b0-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6\" (UID: \"3157a099-6a36-4df2-a4cc-b006777e92b0\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" Oct 31 13:05:30.079294 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.079142 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3157a099-6a36-4df2-a4cc-b006777e92b0-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6\" (UID: \"3157a099-6a36-4df2-a4cc-b006777e92b0\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" Oct 31 13:05:30.132231 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.132196 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7" event={"ID":"c548ef73-9218-4068-bcd2-101717e5295f","Type":"ContainerStarted","Data":"ea494f1b76137769b982e4a545bf4c04af73b7bf26d1af37426210549f620235"} Oct 31 13:05:30.180586 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.180548 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a108df2b-6cb1-499c-aef6-d0d200b580c1-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc\" (UID: \"a108df2b-6cb1-499c-aef6-d0d200b580c1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" Oct 31 13:05:30.180586 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.180590 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a108df2b-6cb1-499c-aef6-d0d200b580c1-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc\" (UID: \"a108df2b-6cb1-499c-aef6-d0d200b580c1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" Oct 31 13:05:30.180842 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.180613 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9vv9\" (UniqueName: \"kubernetes.io/projected/3157a099-6a36-4df2-a4cc-b006777e92b0-kube-api-access-k9vv9\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6\" (UID: \"3157a099-6a36-4df2-a4cc-b006777e92b0\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" Oct 31 13:05:30.180842 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.180679 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3157a099-6a36-4df2-a4cc-b006777e92b0-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6\" (UID: \"3157a099-6a36-4df2-a4cc-b006777e92b0\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" Oct 31 13:05:30.180842 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.180712 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a108df2b-6cb1-499c-aef6-d0d200b580c1-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc\" (UID: \"a108df2b-6cb1-499c-aef6-d0d200b580c1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" Oct 31 13:05:30.180842 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.180748 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3157a099-6a36-4df2-a4cc-b006777e92b0-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6\" (UID: \"3157a099-6a36-4df2-a4cc-b006777e92b0\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" Oct 31 13:05:30.180842 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.180769 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a108df2b-6cb1-499c-aef6-d0d200b580c1-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc\" (UID: \"a108df2b-6cb1-499c-aef6-d0d200b580c1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" Oct 31 13:05:30.180842 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.180796 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3157a099-6a36-4df2-a4cc-b006777e92b0-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6\" (UID: \"3157a099-6a36-4df2-a4cc-b006777e92b0\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" Oct 31 13:05:30.180842 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.180826 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hq49\" (UniqueName: \"kubernetes.io/projected/a108df2b-6cb1-499c-aef6-d0d200b580c1-kube-api-access-7hq49\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc\" (UID: \"a108df2b-6cb1-499c-aef6-d0d200b580c1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" Oct 31 13:05:30.181278 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.180872 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3157a099-6a36-4df2-a4cc-b006777e92b0-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6\" (UID: \"3157a099-6a36-4df2-a4cc-b006777e92b0\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" Oct 31 13:05:30.181278 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.180961 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a108df2b-6cb1-499c-aef6-d0d200b580c1-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc\" (UID: \"a108df2b-6cb1-499c-aef6-d0d200b580c1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" Oct 31 13:05:30.181278 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.181034 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a108df2b-6cb1-499c-aef6-d0d200b580c1-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc\" (UID: \"a108df2b-6cb1-499c-aef6-d0d200b580c1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" Oct 31 13:05:30.181278 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.181081 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a108df2b-6cb1-499c-aef6-d0d200b580c1-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc\" (UID: \"a108df2b-6cb1-499c-aef6-d0d200b580c1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" Oct 31 13:05:30.181278 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.181126 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a108df2b-6cb1-499c-aef6-d0d200b580c1-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc\" (UID: \"a108df2b-6cb1-499c-aef6-d0d200b580c1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" Oct 31 13:05:30.181278 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.181167 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3157a099-6a36-4df2-a4cc-b006777e92b0-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6\" (UID: \"3157a099-6a36-4df2-a4cc-b006777e92b0\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" Oct 31 13:05:30.181278 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.181190 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a108df2b-6cb1-499c-aef6-d0d200b580c1-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc\" (UID: \"a108df2b-6cb1-499c-aef6-d0d200b580c1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" Oct 31 13:05:30.181278 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.181229 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3157a099-6a36-4df2-a4cc-b006777e92b0-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6\" (UID: \"3157a099-6a36-4df2-a4cc-b006777e92b0\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" Oct 31 13:05:30.181278 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.181265 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3157a099-6a36-4df2-a4cc-b006777e92b0-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6\" (UID: \"3157a099-6a36-4df2-a4cc-b006777e92b0\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" Oct 31 13:05:30.181779 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.181341 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3157a099-6a36-4df2-a4cc-b006777e92b0-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6\" (UID: \"3157a099-6a36-4df2-a4cc-b006777e92b0\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" Oct 31 13:05:30.181779 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.181392 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a108df2b-6cb1-499c-aef6-d0d200b580c1-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc\" (UID: \"a108df2b-6cb1-499c-aef6-d0d200b580c1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" Oct 31 13:05:30.181779 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.181123 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a108df2b-6cb1-499c-aef6-d0d200b580c1-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc\" (UID: \"a108df2b-6cb1-499c-aef6-d0d200b580c1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" Oct 31 13:05:30.181779 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.181341 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3157a099-6a36-4df2-a4cc-b006777e92b0-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6\" (UID: \"3157a099-6a36-4df2-a4cc-b006777e92b0\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" Oct 31 13:05:30.181779 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.181724 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3157a099-6a36-4df2-a4cc-b006777e92b0-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6\" (UID: \"3157a099-6a36-4df2-a4cc-b006777e92b0\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" Oct 31 13:05:30.182183 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.181834 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a108df2b-6cb1-499c-aef6-d0d200b580c1-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc\" (UID: \"a108df2b-6cb1-499c-aef6-d0d200b580c1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" Oct 31 13:05:30.182183 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.181878 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3157a099-6a36-4df2-a4cc-b006777e92b0-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6\" (UID: \"3157a099-6a36-4df2-a4cc-b006777e92b0\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" Oct 31 13:05:30.182183 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.181921 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a108df2b-6cb1-499c-aef6-d0d200b580c1-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc\" (UID: \"a108df2b-6cb1-499c-aef6-d0d200b580c1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" Oct 31 13:05:30.182360 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.182233 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3157a099-6a36-4df2-a4cc-b006777e92b0-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6\" (UID: \"3157a099-6a36-4df2-a4cc-b006777e92b0\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" Oct 31 13:05:30.182360 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.182249 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3157a099-6a36-4df2-a4cc-b006777e92b0-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6\" (UID: \"3157a099-6a36-4df2-a4cc-b006777e92b0\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" Oct 31 13:05:30.184149 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.184125 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a108df2b-6cb1-499c-aef6-d0d200b580c1-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc\" (UID: \"a108df2b-6cb1-499c-aef6-d0d200b580c1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" Oct 31 13:05:30.184263 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.184190 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3157a099-6a36-4df2-a4cc-b006777e92b0-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6\" (UID: \"3157a099-6a36-4df2-a4cc-b006777e92b0\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" Oct 31 13:05:30.184309 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.184273 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a108df2b-6cb1-499c-aef6-d0d200b580c1-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc\" (UID: \"a108df2b-6cb1-499c-aef6-d0d200b580c1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" Oct 31 13:05:30.184349 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.184307 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3157a099-6a36-4df2-a4cc-b006777e92b0-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6\" (UID: \"3157a099-6a36-4df2-a4cc-b006777e92b0\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" Oct 31 13:05:30.218423 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.218352 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hq49\" (UniqueName: \"kubernetes.io/projected/a108df2b-6cb1-499c-aef6-d0d200b580c1-kube-api-access-7hq49\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc\" (UID: \"a108df2b-6cb1-499c-aef6-d0d200b580c1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" Oct 31 13:05:30.221911 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.221888 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9vv9\" (UniqueName: \"kubernetes.io/projected/3157a099-6a36-4df2-a4cc-b006777e92b0-kube-api-access-k9vv9\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6\" (UID: \"3157a099-6a36-4df2-a4cc-b006777e92b0\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" Oct 31 13:05:30.224592 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.224570 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a108df2b-6cb1-499c-aef6-d0d200b580c1-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc\" (UID: \"a108df2b-6cb1-499c-aef6-d0d200b580c1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" Oct 31 13:05:30.231647 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.231628 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3157a099-6a36-4df2-a4cc-b006777e92b0-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6\" (UID: \"3157a099-6a36-4df2-a4cc-b006777e92b0\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" Oct 31 13:05:30.362816 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.362780 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" Oct 31 13:05:30.373423 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.373398 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" Oct 31 13:05:30.416036 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.415996 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:30.659656 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.659628 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6"] Oct 31 13:05:30.661448 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:05:30.661418 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3157a099_6a36_4df2_a4cc_b006777e92b0.slice/crio-93452ff3b5183b37aada4375ff5cc2c416ecad80534b27f3de11c19e3d124755 WatchSource:0}: Error finding container 93452ff3b5183b37aada4375ff5cc2c416ecad80534b27f3de11c19e3d124755: Status 404 returned error can't find the container with id 93452ff3b5183b37aada4375ff5cc2c416ecad80534b27f3de11c19e3d124755 Oct 31 13:05:30.664523 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.664487 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:05:30.664615 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.664573 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:05:30.664656 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.664613 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:05:30.681421 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.681397 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc"] Oct 31 13:05:30.682313 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:05:30.682293 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda108df2b_6cb1_499c_aef6_d0d200b580c1.slice/crio-aff4e989227d1c2f3dd46f4d6b276eaf0a87c2d77470cda9613081f4dc1a5093 WatchSource:0}: Error finding container aff4e989227d1c2f3dd46f4d6b276eaf0a87c2d77470cda9613081f4dc1a5093: Status 404 returned error can't find the container with id aff4e989227d1c2f3dd46f4d6b276eaf0a87c2d77470cda9613081f4dc1a5093 Oct 31 13:05:30.685407 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.685374 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:05:30.685484 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.685446 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:05:30.685484 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:30.685474 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:05:31.141553 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:31.141465 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7" event={"ID":"c548ef73-9218-4068-bcd2-101717e5295f","Type":"ContainerStarted","Data":"a8114fc3a6b4c294e27ae142bbd99e2f28b2839c2daacd0d7fb78a1dc239dd49"} Oct 31 13:05:31.141553 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:31.141546 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7" Oct 31 13:05:31.143304 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:31.143277 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" event={"ID":"a108df2b-6cb1-499c-aef6-d0d200b580c1","Type":"ContainerStarted","Data":"2f6e72f4fe24dbddcf9698765c330de4cb884140f00aec5d8fddae62b1f63189"} Oct 31 13:05:31.143455 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:31.143327 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" event={"ID":"a108df2b-6cb1-499c-aef6-d0d200b580c1","Type":"ContainerStarted","Data":"aff4e989227d1c2f3dd46f4d6b276eaf0a87c2d77470cda9613081f4dc1a5093"} Oct 31 13:05:31.144940 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:31.144915 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" event={"ID":"3157a099-6a36-4df2-a4cc-b006777e92b0","Type":"ContainerStarted","Data":"263c6d69cf6cf1e8ba26ca1d69c8554371f65e1490f0beff0f81ce31eb5da80d"} Oct 31 13:05:31.145049 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:31.144963 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" event={"ID":"3157a099-6a36-4df2-a4cc-b006777e92b0","Type":"ContainerStarted","Data":"93452ff3b5183b37aada4375ff5cc2c416ecad80534b27f3de11c19e3d124755"} Oct 31 13:05:31.165587 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:31.165541 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7" podStartSLOduration=1.626627826 podStartE2EDuration="2.165528152s" totalImagesPullingTime="538.900326ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:05:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:05:31.162586868 +0000 UTC m=+1977.645709771" watchObservedRunningTime="2026-10-31 13:05:31.165528152 +0000 UTC m=+1977.648651087" Oct 31 13:05:31.188270 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:31.188230 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" podStartSLOduration=2.18821733 podStartE2EDuration="2.18821733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:05:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:05:31.187572928 +0000 UTC m=+1977.670695843" watchObservedRunningTime="2026-10-31 13:05:31.18821733 +0000 UTC m=+1977.671340231" Oct 31 13:05:31.363977 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:31.363288 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" Oct 31 13:05:31.375937 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:31.375822 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" Oct 31 13:05:31.380566 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:31.380060 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 31 13:05:31.380566 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:31.380124 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" podUID="a108df2b-6cb1-499c-aef6-d0d200b580c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 31 13:05:31.380566 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:31.380460 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 31 13:05:31.380566 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:31.380506 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" podUID="3157a099-6a36-4df2-a4cc-b006777e92b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 31 13:05:31.423098 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:31.422941 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:31.816754 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:05:31.816714 6841 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f31282_bb5f_426c_a167_af1e4bb39696.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb09c0fc_2974_4e9a_8c0e_f9edd5528c49.slice/crio-213f0df112b21dce17e584231f0dff0b9fab9afad5f9126ba05b5fc42713b374\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f31282_bb5f_426c_a167_af1e4bb39696.slice/crio-b305cb4830c27d48d2700a5b4965cd5222c5287ffe873e622dd5dfe559664f80\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb09c0fc_2974_4e9a_8c0e_f9edd5528c49.slice\": RecentStats: unable to find data in memory cache]" Oct 31 13:05:31.816754 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:05:31.816732 6841 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb09c0fc_2974_4e9a_8c0e_f9edd5528c49.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb09c0fc_2974_4e9a_8c0e_f9edd5528c49.slice/crio-213f0df112b21dce17e584231f0dff0b9fab9afad5f9126ba05b5fc42713b374\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f31282_bb5f_426c_a167_af1e4bb39696.slice/crio-b305cb4830c27d48d2700a5b4965cd5222c5287ffe873e622dd5dfe559664f80\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f31282_bb5f_426c_a167_af1e4bb39696.slice\": RecentStats: unable to find data in memory cache]" Oct 31 13:05:32.266007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:32.265671 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:32.378006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:32.376434 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 31 13:05:32.378006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:32.376500 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" podUID="a108df2b-6cb1-499c-aef6-d0d200b580c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 31 13:05:32.378006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:32.377277 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 31 13:05:32.378006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:32.377320 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" podUID="3157a099-6a36-4df2-a4cc-b006777e92b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 31 13:05:33.104226 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:33.104192 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:33.364467 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:33.364373 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 31 13:05:33.364761 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:33.364720 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" podUID="3157a099-6a36-4df2-a4cc-b006777e92b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 31 13:05:33.383310 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:33.383015 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 31 13:05:33.383876 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:33.383843 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" podUID="a108df2b-6cb1-499c-aef6-d0d200b580c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 31 13:05:33.902017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:33.900578 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:34.363400 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:34.363363 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 31 13:05:34.363710 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:34.363675 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" podUID="3157a099-6a36-4df2-a4cc-b006777e92b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 31 13:05:34.392204 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:34.392146 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 31 13:05:34.392706 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:34.392227 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" podUID="a108df2b-6cb1-499c-aef6-d0d200b580c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 31 13:05:34.822605 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:34.822566 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:35.364425 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:35.364387 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 31 13:05:35.364709 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:35.364674 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" podUID="3157a099-6a36-4df2-a4cc-b006777e92b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 31 13:05:35.384006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:35.377314 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 31 13:05:35.384006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:35.377365 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" podUID="a108df2b-6cb1-499c-aef6-d0d200b580c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 31 13:05:35.540656 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:35.540617 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:36.367006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:36.366395 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 31 13:05:36.367006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:36.366476 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" podUID="3157a099-6a36-4df2-a4cc-b006777e92b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 31 13:05:36.393005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:36.392561 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 31 13:05:36.393005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:36.392620 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" podUID="a108df2b-6cb1-499c-aef6-d0d200b580c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 31 13:05:36.766436 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:36.766391 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:37.147231 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:37.147126 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:37.253204 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:37.253164 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7" Oct 31 13:05:37.295004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:37.287191 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" podStartSLOduration=8.287171615 podStartE2EDuration="8.287171615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:05:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:05:31.211676261 +0000 UTC m=+1977.694799163" watchObservedRunningTime="2026-10-31 13:05:37.287171615 +0000 UTC m=+1983.770294521" Oct 31 13:05:37.357795 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:37.357758 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:37.365509 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:37.365478 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 31 13:05:37.365778 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:37.365744 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" podUID="3157a099-6a36-4df2-a4cc-b006777e92b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 31 13:05:37.381543 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:37.381497 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 31 13:05:37.381767 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:37.381746 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" podUID="a108df2b-6cb1-499c-aef6-d0d200b580c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 31 13:05:38.242726 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:38.242629 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:38.363431 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:38.363390 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 31 13:05:38.363745 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:38.363713 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" podUID="3157a099-6a36-4df2-a4cc-b006777e92b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 31 13:05:38.377839 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:38.377809 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 31 13:05:38.378116 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:38.378076 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" podUID="a108df2b-6cb1-499c-aef6-d0d200b580c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 31 13:05:38.413382 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:38.413348 6841 scope.go:122] "RemoveContainer" containerID="bbc75af6dc03cb69db4ada126f12254513fde386bdd1d60640143ac66b8437cb" Oct 31 13:05:38.729484 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:38.729110 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:05:38.729484 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:38.729183 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" podUID="4f4b3219-9e15-407f-9715-9865e01ea8f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:05:39.406064 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:39.354441 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:39.430327 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:39.385557 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 31 13:05:39.430592 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:39.430563 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" podUID="a108df2b-6cb1-499c-aef6-d0d200b580c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 31 13:05:39.431094 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:39.385814 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 31 13:05:39.431269 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:39.431246 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" podUID="3157a099-6a36-4df2-a4cc-b006777e92b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 31 13:05:39.638043 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:39.623437 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" podUID="688d5b2f-0756-480e-83f2-66e05c48527b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.147:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:05:40.131696 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:40.131661 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:40.389060 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:40.363920 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 31 13:05:40.389060 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:40.363995 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" podUID="3157a099-6a36-4df2-a4cc-b006777e92b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 31 13:05:40.389540 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:40.389421 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 31 13:05:40.389540 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:40.389480 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" podUID="a108df2b-6cb1-499c-aef6-d0d200b580c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 31 13:05:40.436440 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:40.436375 6841 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" podUID="688d5b2f-0756-480e-83f2-66e05c48527b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.147:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:05:40.996008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:40.983591 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:41.411848 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:41.411768 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" Oct 31 13:05:41.412289 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:41.412264 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" Oct 31 13:05:41.413096 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:41.413071 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" Oct 31 13:05:42.164843 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:42.156638 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:42.406006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:42.405131 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:05:42.406006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:42.405221 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" podUID="3157a099-6a36-4df2-a4cc-b006777e92b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:05:42.918143 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:42.918109 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:43.051664 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:43.051628 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" Oct 31 13:05:43.052135 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:43.052111 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" Oct 31 13:05:43.091944 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:43.091911 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" Oct 31 13:05:43.253713 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:43.253682 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-4989f9-67d6485cdd-xndwk_e024b9a0-c5a3-4850-914a-fe60e0c6f50e/payload-processing/0.log" Oct 31 13:05:43.619965 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:43.619869 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:43.949693 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:43.949668 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f4cdd9-b64d65f5b-rcxdp_9574b8ac-9faf-47af-a648-8ebf24027496/payload-processing/0.log" Oct 31 13:05:44.153212 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:44.153183 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f4cdd9-b64d65f5b-rcxdp_9574b8ac-9faf-47af-a648-8ebf24027496/payload-processing/0.log" Oct 31 13:05:44.557138 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:44.557105 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:44.925447 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:44.925355 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:45.374081 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:45.374040 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:45.650188 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:45.650102 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:46.069166 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:46.069133 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:46.759009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:46.726625 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:47.031222 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:47.031124 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:47.858291 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:47.858253 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:48.555726 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:48.555692 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q"] Oct 31 13:05:48.555930 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:48.555880 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" podUID="47a4ec22-fe77-4a7b-a66a-5debadbf5800" containerName="istio-proxy" containerID="cri-o://77c4cd2f036dbb6ba64e32c1632e11d9bc4d0de3b426f7eee5bca3a340c296f3" gracePeriod=30 Oct 31 13:05:48.944071 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:48.944037 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:50.036300 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:05:50.036262 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:05:50.040348 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:50.040322 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:50.446784 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:50.446752 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:50.723293 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:50.723256 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:50.996029 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:50.995928 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:51.532082 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:51.532027 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:51.974662 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:51.974618 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:52.194013 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:52.182870 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:52.662010 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:52.656728 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:53.549175 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:53.549141 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:53.845700 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:53.845678 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" Oct 31 13:05:53.906717 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:53.906682 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47a4ec22-fe77-4a7b-a66a-5debadbf5800-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a4ec22-fe77-4a7b-a66a-5debadbf5800-workload-socket\") pod \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\" (UID: \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\") " Oct 31 13:05:53.906906 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:53.906763 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47a4ec22-fe77-4a7b-a66a-5debadbf5800-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47a4ec22-fe77-4a7b-a66a-5debadbf5800-istio-podinfo\") pod \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\" (UID: \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\") " Oct 31 13:05:53.906906 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:53.906815 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47a4ec22-fe77-4a7b-a66a-5debadbf5800-kube-api-access-wvsn5\" (UniqueName: \"kubernetes.io/projected/47a4ec22-fe77-4a7b-a66a-5debadbf5800-kube-api-access-wvsn5\") pod \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\" (UID: \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\") " Oct 31 13:05:53.907094 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:53.906907 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47a4ec22-fe77-4a7b-a66a-5debadbf5800-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47a4ec22-fe77-4a7b-a66a-5debadbf5800-workload-certs\") pod \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\" (UID: \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\") " Oct 31 13:05:53.907094 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:53.906943 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47a4ec22-fe77-4a7b-a66a-5debadbf5800-workload-socket" pod "47a4ec22-fe77-4a7b-a66a-5debadbf5800" (UID: "47a4ec22-fe77-4a7b-a66a-5debadbf5800"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:05:53.907094 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:53.907037 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47a4ec22-fe77-4a7b-a66a-5debadbf5800-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47a4ec22-fe77-4a7b-a66a-5debadbf5800-istio-envoy\") pod \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\" (UID: \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\") " Oct 31 13:05:53.907251 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:53.907160 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47a4ec22-fe77-4a7b-a66a-5debadbf5800-istio-token\" (UniqueName: \"kubernetes.io/projected/47a4ec22-fe77-4a7b-a66a-5debadbf5800-istio-token\") pod \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\" (UID: \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\") " Oct 31 13:05:53.907321 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:53.907264 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47a4ec22-fe77-4a7b-a66a-5debadbf5800-workload-certs" pod "47a4ec22-fe77-4a7b-a66a-5debadbf5800" (UID: "47a4ec22-fe77-4a7b-a66a-5debadbf5800"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:05:53.907515 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:53.907478 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47a4ec22-fe77-4a7b-a66a-5debadbf5800-credential-socket" pod "47a4ec22-fe77-4a7b-a66a-5debadbf5800" (UID: "47a4ec22-fe77-4a7b-a66a-5debadbf5800"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:05:53.907515 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:53.907501 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47a4ec22-fe77-4a7b-a66a-5debadbf5800-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a4ec22-fe77-4a7b-a66a-5debadbf5800-credential-socket\") pod \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\" (UID: \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\") " Oct 31 13:05:53.907707 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:53.907637 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47a4ec22-fe77-4a7b-a66a-5debadbf5800-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47a4ec22-fe77-4a7b-a66a-5debadbf5800-istiod-ca-cert\") pod \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\" (UID: \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\") " Oct 31 13:05:53.907707 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:53.907680 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47a4ec22-fe77-4a7b-a66a-5debadbf5800-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47a4ec22-fe77-4a7b-a66a-5debadbf5800-istio-data\") pod \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\" (UID: \"47a4ec22-fe77-4a7b-a66a-5debadbf5800\") " Oct 31 13:05:53.908712 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:53.908089 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47a4ec22-fe77-4a7b-a66a-5debadbf5800-istiod-ca-cert" pod "47a4ec22-fe77-4a7b-a66a-5debadbf5800" (UID: "47a4ec22-fe77-4a7b-a66a-5debadbf5800"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:05:53.908712 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:53.908313 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47a4ec22-fe77-4a7b-a66a-5debadbf5800-istio-data" pod "47a4ec22-fe77-4a7b-a66a-5debadbf5800" (UID: "47a4ec22-fe77-4a7b-a66a-5debadbf5800"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:05:53.908712 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:53.908619 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a4ec22-fe77-4a7b-a66a-5debadbf5800-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:53.908712 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:53.908645 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47a4ec22-fe77-4a7b-a66a-5debadbf5800-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:53.908712 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:53.908662 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a4ec22-fe77-4a7b-a66a-5debadbf5800-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:53.908712 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:53.908676 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47a4ec22-fe77-4a7b-a66a-5debadbf5800-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:53.908712 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:53.908689 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47a4ec22-fe77-4a7b-a66a-5debadbf5800-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:53.909591 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:53.909562 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47a4ec22-fe77-4a7b-a66a-5debadbf5800-istio-token" pod "47a4ec22-fe77-4a7b-a66a-5debadbf5800" (UID: "47a4ec22-fe77-4a7b-a66a-5debadbf5800"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:05:53.909727 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:53.909700 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47a4ec22-fe77-4a7b-a66a-5debadbf5800-istio-podinfo" pod "47a4ec22-fe77-4a7b-a66a-5debadbf5800" (UID: "47a4ec22-fe77-4a7b-a66a-5debadbf5800"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:05:53.909808 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:53.909737 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47a4ec22-fe77-4a7b-a66a-5debadbf5800-kube-api-access-wvsn5" pod "47a4ec22-fe77-4a7b-a66a-5debadbf5800" (UID: "47a4ec22-fe77-4a7b-a66a-5debadbf5800"). InnerVolumeSpecName "kube-api-access-wvsn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:05:53.911239 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:53.911210 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47a4ec22-fe77-4a7b-a66a-5debadbf5800-istio-envoy" pod "47a4ec22-fe77-4a7b-a66a-5debadbf5800" (UID: "47a4ec22-fe77-4a7b-a66a-5debadbf5800"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:05:54.009885 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:54.009855 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47a4ec22-fe77-4a7b-a66a-5debadbf5800-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:54.009885 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:54.009883 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47a4ec22-fe77-4a7b-a66a-5debadbf5800-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:54.009885 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:54.009893 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvsn5\" (UniqueName: \"kubernetes.io/projected/47a4ec22-fe77-4a7b-a66a-5debadbf5800-kube-api-access-wvsn5\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:54.010156 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:54.009903 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47a4ec22-fe77-4a7b-a66a-5debadbf5800-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:54.130434 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:54.130327 6841 generic.go:356] "Generic (PLEG): container finished" podID="47a4ec22-fe77-4a7b-a66a-5debadbf5800" containerID="77c4cd2f036dbb6ba64e32c1632e11d9bc4d0de3b426f7eee5bca3a340c296f3" exitCode=0 Oct 31 13:05:54.130598 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:54.130447 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" Oct 31 13:05:54.130598 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:54.130464 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" event={"ID":"47a4ec22-fe77-4a7b-a66a-5debadbf5800","Type":"ContainerDied","Data":"77c4cd2f036dbb6ba64e32c1632e11d9bc4d0de3b426f7eee5bca3a340c296f3"} Oct 31 13:05:54.130598 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:54.130548 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q" event={"ID":"47a4ec22-fe77-4a7b-a66a-5debadbf5800","Type":"ContainerDied","Data":"469a7aea880d06bbd38b74d5658004d062559676d294170c09af68394bdab998"} Oct 31 13:05:54.130598 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:54.130568 6841 scope.go:122] "RemoveContainer" containerID="77c4cd2f036dbb6ba64e32c1632e11d9bc4d0de3b426f7eee5bca3a340c296f3" Oct 31 13:05:54.142858 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:54.142831 6841 scope.go:122] "RemoveContainer" containerID="77c4cd2f036dbb6ba64e32c1632e11d9bc4d0de3b426f7eee5bca3a340c296f3" Oct 31 13:05:54.143221 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:05:54.143196 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77c4cd2f036dbb6ba64e32c1632e11d9bc4d0de3b426f7eee5bca3a340c296f3\": container with ID starting with 77c4cd2f036dbb6ba64e32c1632e11d9bc4d0de3b426f7eee5bca3a340c296f3 not found: ID does not exist" containerID="77c4cd2f036dbb6ba64e32c1632e11d9bc4d0de3b426f7eee5bca3a340c296f3" Oct 31 13:05:54.143303 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:54.143233 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77c4cd2f036dbb6ba64e32c1632e11d9bc4d0de3b426f7eee5bca3a340c296f3"} err="failed to get container status \"77c4cd2f036dbb6ba64e32c1632e11d9bc4d0de3b426f7eee5bca3a340c296f3\": rpc error: code = NotFound desc = could not find container \"77c4cd2f036dbb6ba64e32c1632e11d9bc4d0de3b426f7eee5bca3a340c296f3\": container with ID starting with 77c4cd2f036dbb6ba64e32c1632e11d9bc4d0de3b426f7eee5bca3a340c296f3 not found: ID does not exist" Oct 31 13:05:54.160841 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:54.160815 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q"] Oct 31 13:05:54.176024 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:54.175208 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ssz2q"] Oct 31 13:05:54.245275 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:54.245177 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:55.035114 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:55.035073 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:55.644381 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:55.644348 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47a4ec22-fe77-4a7b-a66a-5debadbf5800" path="/var/lib/kubelet/pods/47a4ec22-fe77-4a7b-a66a-5debadbf5800/volumes" Oct 31 13:05:55.764579 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:55.764543 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:56.493846 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:56.489530 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:57.331630 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:57.330511 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:58.038872 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:58.038838 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:58.778829 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:58.778740 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:59.462041 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:59.462003 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28"] Oct 31 13:05:59.462707 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:59.462686 6841 cpu_manager.go:414] "containerMap: removing container" podUID="47a4ec22-fe77-4a7b-a66a-5debadbf5800" containerName="istio-proxy" Oct 31 13:05:59.462707 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:59.462704 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47a4ec22-fe77-4a7b-a66a-5debadbf5800" containerName="istio-proxy" Oct 31 13:05:59.462957 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:59.462942 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="47a4ec22-fe77-4a7b-a66a-5debadbf5800" containerName="istio-proxy" Oct 31 13:05:59.466801 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:59.466782 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28" Oct 31 13:05:59.477124 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:59.477064 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-3a3000-serving-cert\"" Oct 31 13:05:59.483097 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:59.483073 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28"] Oct 31 13:05:59.506941 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:59.506907 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/129ee68e-30b6-4e0c-97e8-85f780b2cab6-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28\" (UID: \"129ee68e-30b6-4e0c-97e8-85f780b2cab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28" Oct 31 13:05:59.507104 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:59.506998 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/129ee68e-30b6-4e0c-97e8-85f780b2cab6-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28\" (UID: \"129ee68e-30b6-4e0c-97e8-85f780b2cab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28" Oct 31 13:05:59.507104 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:59.507045 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/129ee68e-30b6-4e0c-97e8-85f780b2cab6-metrics-certs\") pod \"maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28\" (UID: \"129ee68e-30b6-4e0c-97e8-85f780b2cab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28" Oct 31 13:05:59.507282 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:59.507259 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znzjd\" (UniqueName: \"kubernetes.io/projected/129ee68e-30b6-4e0c-97e8-85f780b2cab6-kube-api-access-znzjd\") pod \"maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28\" (UID: \"129ee68e-30b6-4e0c-97e8-85f780b2cab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28" Oct 31 13:05:59.619523 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:59.619485 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znzjd\" (UniqueName: \"kubernetes.io/projected/129ee68e-30b6-4e0c-97e8-85f780b2cab6-kube-api-access-znzjd\") pod \"maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28\" (UID: \"129ee68e-30b6-4e0c-97e8-85f780b2cab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28" Oct 31 13:05:59.620500 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:59.620466 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/129ee68e-30b6-4e0c-97e8-85f780b2cab6-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28\" (UID: \"129ee68e-30b6-4e0c-97e8-85f780b2cab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28" Oct 31 13:05:59.620672 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:59.620576 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/129ee68e-30b6-4e0c-97e8-85f780b2cab6-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28\" (UID: \"129ee68e-30b6-4e0c-97e8-85f780b2cab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28" Oct 31 13:05:59.620672 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:59.620608 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/129ee68e-30b6-4e0c-97e8-85f780b2cab6-metrics-certs\") pod \"maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28\" (UID: \"129ee68e-30b6-4e0c-97e8-85f780b2cab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28" Oct 31 13:05:59.621853 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:59.621825 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/129ee68e-30b6-4e0c-97e8-85f780b2cab6-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28\" (UID: \"129ee68e-30b6-4e0c-97e8-85f780b2cab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28" Oct 31 13:05:59.623087 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:59.622371 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:05:59.657003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:59.654717 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/129ee68e-30b6-4e0c-97e8-85f780b2cab6-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28\" (UID: \"129ee68e-30b6-4e0c-97e8-85f780b2cab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28" Oct 31 13:05:59.699005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:59.682548 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znzjd\" (UniqueName: \"kubernetes.io/projected/129ee68e-30b6-4e0c-97e8-85f780b2cab6-kube-api-access-znzjd\") pod \"maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28\" (UID: \"129ee68e-30b6-4e0c-97e8-85f780b2cab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28" Oct 31 13:05:59.699005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:59.693653 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/129ee68e-30b6-4e0c-97e8-85f780b2cab6-metrics-certs\") pod \"maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28\" (UID: \"129ee68e-30b6-4e0c-97e8-85f780b2cab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28" Oct 31 13:05:59.785264 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:59.785184 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28" Oct 31 13:05:59.827375 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:05:59.827345 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:00.217276 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:00.217243 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28"] Oct 31 13:06:00.218707 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:00.218666 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28" event={"ID":"129ee68e-30b6-4e0c-97e8-85f780b2cab6","Type":"ContainerStarted","Data":"00a9ea9a9bb63dbbd9033732fa9322ba72b82bdc20f2924e143fb10b725dc108"} Oct 31 13:06:00.663472 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:00.663420 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:01.227858 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:01.227815 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28" event={"ID":"129ee68e-30b6-4e0c-97e8-85f780b2cab6","Type":"ContainerStarted","Data":"04ee0f7e7a2320a1554d9a0bde753ca2cb701654f3fbfbba9e6f1b5432d51049"} Oct 31 13:06:01.228083 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:01.227947 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28" Oct 31 13:06:01.251056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:01.251002 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28" podStartSLOduration=1.695454873 podStartE2EDuration="2.250972284s" totalImagesPullingTime="555.517411ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:05:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:06:01.248835252 +0000 UTC m=+2007.731958154" watchObservedRunningTime="2026-10-31 13:06:01.250972284 +0000 UTC m=+2007.734095184" Oct 31 13:06:02.264008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:02.259099 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:03.555114 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:03.555062 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:04.224145 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:04.224089 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" podUID="da83bd57-52af-48d3-bd76-42e36c18112f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:04.572064 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:04.553561 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:05.236020 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:05.221383 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:05.236020 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:05.233339 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" podUID="da83bd57-52af-48d3-bd76-42e36c18112f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:05.245007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:05.236972 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" Oct 31 13:06:05.531051 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:05.527176 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" Oct 31 13:06:05.598093 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:05.598056 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt"] Oct 31 13:06:05.613478 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:05.613425 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" Oct 31 13:06:05.633111 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:05.633077 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt"] Oct 31 13:06:05.784936 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:05.784856 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c763c4c-2410-4735-9e8d-98d8ec38142c-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt\" (UID: \"8c763c4c-2410-4735-9e8d-98d8ec38142c\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" Oct 31 13:06:05.784936 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:05.784918 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c763c4c-2410-4735-9e8d-98d8ec38142c-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt\" (UID: \"8c763c4c-2410-4735-9e8d-98d8ec38142c\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" Oct 31 13:06:05.785229 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:05.785192 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c763c4c-2410-4735-9e8d-98d8ec38142c-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt\" (UID: \"8c763c4c-2410-4735-9e8d-98d8ec38142c\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" Oct 31 13:06:05.785315 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:05.785287 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c763c4c-2410-4735-9e8d-98d8ec38142c-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt\" (UID: \"8c763c4c-2410-4735-9e8d-98d8ec38142c\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" Oct 31 13:06:05.785375 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:05.785358 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb5mg\" (UniqueName: \"kubernetes.io/projected/8c763c4c-2410-4735-9e8d-98d8ec38142c-kube-api-access-xb5mg\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt\" (UID: \"8c763c4c-2410-4735-9e8d-98d8ec38142c\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" Oct 31 13:06:05.785483 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:05.785466 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c763c4c-2410-4735-9e8d-98d8ec38142c-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt\" (UID: \"8c763c4c-2410-4735-9e8d-98d8ec38142c\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" Oct 31 13:06:05.785525 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:05.785503 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c763c4c-2410-4735-9e8d-98d8ec38142c-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt\" (UID: \"8c763c4c-2410-4735-9e8d-98d8ec38142c\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" Oct 31 13:06:05.785562 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:05.785542 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c763c4c-2410-4735-9e8d-98d8ec38142c-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt\" (UID: \"8c763c4c-2410-4735-9e8d-98d8ec38142c\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" Oct 31 13:06:05.785741 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:05.785726 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c763c4c-2410-4735-9e8d-98d8ec38142c-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt\" (UID: \"8c763c4c-2410-4735-9e8d-98d8ec38142c\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" Oct 31 13:06:05.887265 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:05.887226 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c763c4c-2410-4735-9e8d-98d8ec38142c-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt\" (UID: \"8c763c4c-2410-4735-9e8d-98d8ec38142c\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" Oct 31 13:06:05.887465 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:05.887348 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c763c4c-2410-4735-9e8d-98d8ec38142c-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt\" (UID: \"8c763c4c-2410-4735-9e8d-98d8ec38142c\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" Oct 31 13:06:05.887465 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:05.887395 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c763c4c-2410-4735-9e8d-98d8ec38142c-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt\" (UID: \"8c763c4c-2410-4735-9e8d-98d8ec38142c\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" Oct 31 13:06:05.887465 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:05.887454 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c763c4c-2410-4735-9e8d-98d8ec38142c-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt\" (UID: \"8c763c4c-2410-4735-9e8d-98d8ec38142c\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" Oct 31 13:06:05.887656 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:05.887482 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c763c4c-2410-4735-9e8d-98d8ec38142c-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt\" (UID: \"8c763c4c-2410-4735-9e8d-98d8ec38142c\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" Oct 31 13:06:05.887656 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:05.887505 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb5mg\" (UniqueName: \"kubernetes.io/projected/8c763c4c-2410-4735-9e8d-98d8ec38142c-kube-api-access-xb5mg\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt\" (UID: \"8c763c4c-2410-4735-9e8d-98d8ec38142c\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" Oct 31 13:06:05.887656 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:05.887585 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c763c4c-2410-4735-9e8d-98d8ec38142c-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt\" (UID: \"8c763c4c-2410-4735-9e8d-98d8ec38142c\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" Oct 31 13:06:05.887656 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:05.887619 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c763c4c-2410-4735-9e8d-98d8ec38142c-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt\" (UID: \"8c763c4c-2410-4735-9e8d-98d8ec38142c\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" Oct 31 13:06:05.887656 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:05.887645 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c763c4c-2410-4735-9e8d-98d8ec38142c-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt\" (UID: \"8c763c4c-2410-4735-9e8d-98d8ec38142c\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" Oct 31 13:06:05.888053 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:05.887714 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c763c4c-2410-4735-9e8d-98d8ec38142c-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt\" (UID: \"8c763c4c-2410-4735-9e8d-98d8ec38142c\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" Oct 31 13:06:05.888053 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:05.887924 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c763c4c-2410-4735-9e8d-98d8ec38142c-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt\" (UID: \"8c763c4c-2410-4735-9e8d-98d8ec38142c\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" Oct 31 13:06:05.888053 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:05.887924 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c763c4c-2410-4735-9e8d-98d8ec38142c-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt\" (UID: \"8c763c4c-2410-4735-9e8d-98d8ec38142c\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" Oct 31 13:06:05.888247 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:05.888175 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c763c4c-2410-4735-9e8d-98d8ec38142c-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt\" (UID: \"8c763c4c-2410-4735-9e8d-98d8ec38142c\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" Oct 31 13:06:05.888768 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:05.888741 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c763c4c-2410-4735-9e8d-98d8ec38142c-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt\" (UID: \"8c763c4c-2410-4735-9e8d-98d8ec38142c\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" Oct 31 13:06:05.890281 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:05.890257 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c763c4c-2410-4735-9e8d-98d8ec38142c-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt\" (UID: \"8c763c4c-2410-4735-9e8d-98d8ec38142c\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" Oct 31 13:06:05.891027 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:05.891004 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c763c4c-2410-4735-9e8d-98d8ec38142c-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt\" (UID: \"8c763c4c-2410-4735-9e8d-98d8ec38142c\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" Oct 31 13:06:05.900621 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:05.900593 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c763c4c-2410-4735-9e8d-98d8ec38142c-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt\" (UID: \"8c763c4c-2410-4735-9e8d-98d8ec38142c\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" Oct 31 13:06:05.900972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:05.900951 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb5mg\" (UniqueName: \"kubernetes.io/projected/8c763c4c-2410-4735-9e8d-98d8ec38142c-kube-api-access-xb5mg\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt\" (UID: \"8c763c4c-2410-4735-9e8d-98d8ec38142c\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" Oct 31 13:06:05.996226 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:05.996194 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" Oct 31 13:06:06.242361 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:06.231612 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:06.322493 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:06.322464 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt"] Oct 31 13:06:06.323183 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:06:06.323156 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c763c4c_2410_4735_9e8d_98d8ec38142c.slice/crio-fcea4e10fc085ddeff7218ad736ca3e2b7846ee811b37ee2a112e103014da32e WatchSource:0}: Error finding container fcea4e10fc085ddeff7218ad736ca3e2b7846ee811b37ee2a112e103014da32e: Status 404 returned error can't find the container with id fcea4e10fc085ddeff7218ad736ca3e2b7846ee811b37ee2a112e103014da32e Oct 31 13:06:06.326160 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:06.326129 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:06:06.326238 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:06.326194 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:06:06.326238 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:06.326224 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:06:06.602842 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:06.602804 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" event={"ID":"8c763c4c-2410-4735-9e8d-98d8ec38142c","Type":"ContainerStarted","Data":"e1b923b781d65b19a56188d0fcd7efbfe3583d5d54eb0096a2caabb79c0e1bf9"} Oct 31 13:06:06.603364 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:06.602864 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" event={"ID":"8c763c4c-2410-4735-9e8d-98d8ec38142c","Type":"ContainerStarted","Data":"fcea4e10fc085ddeff7218ad736ca3e2b7846ee811b37ee2a112e103014da32e"} Oct 31 13:06:06.626580 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:06.626474 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" podStartSLOduration=1.626458156 podStartE2EDuration="1.626458156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:06:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:06:06.625310809 +0000 UTC m=+2013.108433713" watchObservedRunningTime="2026-10-31 13:06:06.626458156 +0000 UTC m=+2013.109581058" Oct 31 13:06:06.769006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:06.768522 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:07.004514 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:06.999870 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" Oct 31 13:06:07.037790 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:07.031398 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:07.291184 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:07.291098 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28" Oct 31 13:06:07.964618 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:07.964584 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:08.010817 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:08.010781 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 31 13:06:08.011122 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:08.011087 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" podUID="8c763c4c-2410-4735-9e8d-98d8ec38142c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 31 13:06:08.893005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:08.890663 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:09.005730 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:09.005212 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 31 13:06:09.006419 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:09.006383 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" podUID="8c763c4c-2410-4735-9e8d-98d8ec38142c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 31 13:06:09.772034 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:09.753484 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:09.996949 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:09.996910 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 31 13:06:09.997135 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:09.996968 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" podUID="8c763c4c-2410-4735-9e8d-98d8ec38142c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 31 13:06:10.489113 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:10.489074 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:10.731044 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:10.727249 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:10.997470 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:10.997428 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 31 13:06:10.997655 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:10.997489 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" podUID="8c763c4c-2410-4735-9e8d-98d8ec38142c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 31 13:06:11.997235 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:11.997196 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 31 13:06:11.997645 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:11.997264 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" podUID="8c763c4c-2410-4735-9e8d-98d8ec38142c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 31 13:06:12.003932 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:12.003891 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:13.000056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:12.999477 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 31 13:06:13.000056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:12.999580 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" podUID="8c763c4c-2410-4735-9e8d-98d8ec38142c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 31 13:06:13.222952 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:13.222913 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:14.233010 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:14.227487 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:14.450103 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:14.450055 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" Oct 31 13:06:14.451191 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:14.451161 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" Oct 31 13:06:14.451944 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:14.451922 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" Oct 31 13:06:15.121119 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:15.121024 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:15.326894 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:15.326860 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh"] Oct 31 13:06:15.518246 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:15.518199 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:15.708926 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:15.708757 6841 generic.go:356] "Generic (PLEG): container finished" podID="5ebddec1-8fab-4709-84de-f5d99a9112d0" containerID="d77b8f1b938e25220d58eb6715bbe7d0fbe7997a420a829dcd16138ef40026fd" exitCode=0 Oct 31 13:06:15.708926 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:15.708811 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" event={"ID":"5ebddec1-8fab-4709-84de-f5d99a9112d0","Type":"ContainerDied","Data":"d77b8f1b938e25220d58eb6715bbe7d0fbe7997a420a829dcd16138ef40026fd"} Oct 31 13:06:15.709533 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:15.709427 6841 scope.go:122] "RemoveContainer" containerID="d77b8f1b938e25220d58eb6715bbe7d0fbe7997a420a829dcd16138ef40026fd" Oct 31 13:06:15.711252 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:15.711215 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:06:15.711369 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:15.711297 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:06:15.711369 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:15.711339 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:06:16.212776 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:16.212740 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:16.595831 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:16.595745 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:16.718555 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:16.718522 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" event={"ID":"5ebddec1-8fab-4709-84de-f5d99a9112d0","Type":"ContainerStarted","Data":"748689c72487261e6e22fc1cf5e3d8a77c9097ebc0c732b4d40b3c95be11a3ae"} Oct 31 13:06:17.364016 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:17.363855 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:17.667478 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:17.667388 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:06:17.668618 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:17.668588 6841 patch_prober.go:29] interesting pod/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 31 13:06:17.668846 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:17.668808 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" podUID="5ebddec1-8fab-4709-84de-f5d99a9112d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 31 13:06:17.682233 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:17.682199 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:18.022424 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:18.022386 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:06:18.254032 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:18.253627 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:18.666400 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:18.666355 6841 patch_prober.go:29] interesting pod/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 31 13:06:18.666620 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:18.666430 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" podUID="5ebddec1-8fab-4709-84de-f5d99a9112d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 31 13:06:19.032047 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:19.031528 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:19.667295 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:19.667258 6841 patch_prober.go:29] interesting pod/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 31 13:06:19.667584 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:19.667549 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" podUID="5ebddec1-8fab-4709-84de-f5d99a9112d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 31 13:06:19.728219 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:19.728180 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:20.035944 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:06:20.035911 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:06:20.420967 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:20.420814 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:20.666445 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:20.666397 6841 patch_prober.go:29] interesting pod/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 31 13:06:20.666636 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:20.666489 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" podUID="5ebddec1-8fab-4709-84de-f5d99a9112d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 31 13:06:21.147576 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:21.147542 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:21.666014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:21.665447 6841 patch_prober.go:29] interesting pod/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 31 13:06:21.666014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:21.665525 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" podUID="5ebddec1-8fab-4709-84de-f5d99a9112d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 31 13:06:22.132005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:22.127945 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:22.670561 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:22.670528 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:06:22.671958 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:22.671933 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:06:22.754950 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:22.754902 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:22.978789 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:22.978758 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:23.968076 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:23.968043 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:24.942004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:24.941621 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7"] Oct 31 13:06:24.942004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:24.941863 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7" podUID="c548ef73-9218-4068-bcd2-101717e5295f" containerName="maas-api" containerID="cri-o://a8114fc3a6b4c294e27ae142bbd99e2f28b2839c2daacd0d7fb78a1dc239dd49" gracePeriod=30 Oct 31 13:06:25.130267 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:25.130194 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:25.393010 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:25.389694 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:25.666723 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:25.663701 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:26.335956 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:26.335728 6841 generic.go:356] "Generic (PLEG): container finished" podID="c548ef73-9218-4068-bcd2-101717e5295f" containerID="a8114fc3a6b4c294e27ae142bbd99e2f28b2839c2daacd0d7fb78a1dc239dd49" exitCode=0 Oct 31 13:06:26.335956 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:26.335862 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7" event={"ID":"c548ef73-9218-4068-bcd2-101717e5295f","Type":"ContainerDied","Data":"a8114fc3a6b4c294e27ae142bbd99e2f28b2839c2daacd0d7fb78a1dc239dd49"} Oct 31 13:06:26.396767 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:26.369193 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7" Oct 31 13:06:26.465023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:26.425930 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c548ef73-9218-4068-bcd2-101717e5295f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c548ef73-9218-4068-bcd2-101717e5295f-metrics-certs\") pod \"c548ef73-9218-4068-bcd2-101717e5295f\" (UID: \"c548ef73-9218-4068-bcd2-101717e5295f\") " Oct 31 13:06:26.465023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:26.426008 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c548ef73-9218-4068-bcd2-101717e5295f-kube-api-access-d7mvs\" (UniqueName: \"kubernetes.io/projected/c548ef73-9218-4068-bcd2-101717e5295f-kube-api-access-d7mvs\") pod \"c548ef73-9218-4068-bcd2-101717e5295f\" (UID: \"c548ef73-9218-4068-bcd2-101717e5295f\") " Oct 31 13:06:26.465023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:26.426156 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c548ef73-9218-4068-bcd2-101717e5295f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c548ef73-9218-4068-bcd2-101717e5295f-openshift-service-ca\") pod \"c548ef73-9218-4068-bcd2-101717e5295f\" (UID: \"c548ef73-9218-4068-bcd2-101717e5295f\") " Oct 31 13:06:26.465023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:26.426220 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c548ef73-9218-4068-bcd2-101717e5295f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c548ef73-9218-4068-bcd2-101717e5295f-maas-api-tls\") pod \"c548ef73-9218-4068-bcd2-101717e5295f\" (UID: \"c548ef73-9218-4068-bcd2-101717e5295f\") " Oct 31 13:06:26.686280 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:26.622067 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" podUID="688d5b2f-0756-480e-83f2-66e05c48527b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.147:8000/ready\": context deadline exceeded" Oct 31 13:06:26.686280 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:26.624229 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c548ef73-9218-4068-bcd2-101717e5295f-openshift-service-ca" pod "c548ef73-9218-4068-bcd2-101717e5295f" (UID: "c548ef73-9218-4068-bcd2-101717e5295f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:06:26.686280 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:26.682457 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c548ef73-9218-4068-bcd2-101717e5295f-metrics-certs" pod "c548ef73-9218-4068-bcd2-101717e5295f" (UID: "c548ef73-9218-4068-bcd2-101717e5295f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:06:26.816017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:26.741210 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" podUID="da83bd57-52af-48d3-bd76-42e36c18112f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:26.816017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:26.813764 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c548ef73-9218-4068-bcd2-101717e5295f-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:26.816017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:26.813802 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c548ef73-9218-4068-bcd2-101717e5295f-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:26.832010 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:26.816844 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c548ef73-9218-4068-bcd2-101717e5295f-maas-api-tls" pod "c548ef73-9218-4068-bcd2-101717e5295f" (UID: "c548ef73-9218-4068-bcd2-101717e5295f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:06:26.916396 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:26.916357 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c548ef73-9218-4068-bcd2-101717e5295f-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:27.098008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:27.090732 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c548ef73-9218-4068-bcd2-101717e5295f-kube-api-access-d7mvs" pod "c548ef73-9218-4068-bcd2-101717e5295f" (UID: "c548ef73-9218-4068-bcd2-101717e5295f"). InnerVolumeSpecName "kube-api-access-d7mvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:06:27.288016 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:27.273617 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c548ef73-9218-4068-bcd2-101717e5295f-kube-api-access-d7mvs\" (UniqueName: \"kubernetes.io/projected/c548ef73-9218-4068-bcd2-101717e5295f-kube-api-access-d7mvs\") pod \"c548ef73-9218-4068-bcd2-101717e5295f\" (UID: \"c548ef73-9218-4068-bcd2-101717e5295f\") " Oct 31 13:06:27.288016 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:06:27.274925 6841 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/c548ef73-9218-4068-bcd2-101717e5295f/volumes/kubernetes.io~projected/kube-api-access-d7mvs Oct 31 13:06:27.288016 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:27.275013 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c548ef73-9218-4068-bcd2-101717e5295f-kube-api-access-d7mvs" pod "c548ef73-9218-4068-bcd2-101717e5295f" (UID: "c548ef73-9218-4068-bcd2-101717e5295f"). InnerVolumeSpecName "kube-api-access-d7mvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:06:27.406105 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:27.383624 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:06:27.406105 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:27.383725 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" podUID="61c4f511-5495-4b14-ae0f-b019d826ae7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:27.548206 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:27.548151 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7mvs\" (UniqueName: \"kubernetes.io/projected/c548ef73-9218-4068-bcd2-101717e5295f-kube-api-access-d7mvs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:27.638147 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:27.638095 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" podUID="688d5b2f-0756-480e-83f2-66e05c48527b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.147:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:27.714340 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:27.714310 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" Oct 31 13:06:27.760435 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:27.748056 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" podUID="da83bd57-52af-48d3-bd76-42e36c18112f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:27.879008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:27.854716 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7" Oct 31 13:06:28.057400 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:28.057313 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:28.142876 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:28.142840 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" Oct 31 13:06:28.152702 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:28.152668 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7" event={"ID":"c548ef73-9218-4068-bcd2-101717e5295f","Type":"ContainerDied","Data":"ea494f1b76137769b982e4a545bf4c04af73b7bf26d1af37426210549f620235"} Oct 31 13:06:28.180326 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:28.180287 6841 scope.go:122] "RemoveContainer" containerID="a8114fc3a6b4c294e27ae142bbd99e2f28b2839c2daacd0d7fb78a1dc239dd49" Oct 31 13:06:28.207181 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:28.207122 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7"] Oct 31 13:06:28.353060 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:28.344235 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" Oct 31 13:06:28.415055 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:28.412557 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7e08dc-686f7d7995-scvf7"] Oct 31 13:06:28.465635 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:28.465600 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" Oct 31 13:06:28.762427 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:28.762392 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:28.969441 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:28.969405 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:29.643730 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:29.643694 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c548ef73-9218-4068-bcd2-101717e5295f" path="/var/lib/kubelet/pods/c548ef73-9218-4068-bcd2-101717e5295f/volumes" Oct 31 13:06:30.384067 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:30.382177 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" podUID="688d5b2f-0756-480e-83f2-66e05c48527b" containerName="main" containerID="cri-o://948fc3e8d748982dab2910d12204f0a9e5cf3a86fcdf7f653ce1e30973023fbd" gracePeriod=45 Oct 31 13:06:30.393529 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:30.393365 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:30.531392 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:30.531353 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" podUID="688d5b2f-0756-480e-83f2-66e05c48527b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.147:8000/ready\": dial tcp 10.134.0.147:8000: connect: connection refused" Oct 31 13:06:30.678569 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:30.678542 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" Oct 31 13:06:30.781791 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:30.781760 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/688d5b2f-0756-480e-83f2-66e05c48527b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/688d5b2f-0756-480e-83f2-66e05c48527b-model-cache\") pod \"688d5b2f-0756-480e-83f2-66e05c48527b\" (UID: \"688d5b2f-0756-480e-83f2-66e05c48527b\") " Oct 31 13:06:30.781956 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:30.781799 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/688d5b2f-0756-480e-83f2-66e05c48527b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/688d5b2f-0756-480e-83f2-66e05c48527b-tmp-dir\") pod \"688d5b2f-0756-480e-83f2-66e05c48527b\" (UID: \"688d5b2f-0756-480e-83f2-66e05c48527b\") " Oct 31 13:06:30.781956 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:30.781901 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/688d5b2f-0756-480e-83f2-66e05c48527b-kube-api-access-tcs4f\" (UniqueName: \"kubernetes.io/projected/688d5b2f-0756-480e-83f2-66e05c48527b-kube-api-access-tcs4f\") pod \"688d5b2f-0756-480e-83f2-66e05c48527b\" (UID: \"688d5b2f-0756-480e-83f2-66e05c48527b\") " Oct 31 13:06:30.781956 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:30.781932 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/688d5b2f-0756-480e-83f2-66e05c48527b-home\" (UniqueName: \"kubernetes.io/empty-dir/688d5b2f-0756-480e-83f2-66e05c48527b-home\") pod \"688d5b2f-0756-480e-83f2-66e05c48527b\" (UID: \"688d5b2f-0756-480e-83f2-66e05c48527b\") " Oct 31 13:06:30.782169 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:30.782087 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/688d5b2f-0756-480e-83f2-66e05c48527b-tls-certs\" (UniqueName: \"kubernetes.io/secret/688d5b2f-0756-480e-83f2-66e05c48527b-tls-certs\") pod \"688d5b2f-0756-480e-83f2-66e05c48527b\" (UID: \"688d5b2f-0756-480e-83f2-66e05c48527b\") " Oct 31 13:06:30.782169 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:30.782081 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/688d5b2f-0756-480e-83f2-66e05c48527b-tmp-dir" pod "688d5b2f-0756-480e-83f2-66e05c48527b" (UID: "688d5b2f-0756-480e-83f2-66e05c48527b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:06:30.782169 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:30.782088 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/688d5b2f-0756-480e-83f2-66e05c48527b-model-cache" pod "688d5b2f-0756-480e-83f2-66e05c48527b" (UID: "688d5b2f-0756-480e-83f2-66e05c48527b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:06:30.782169 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:30.782130 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/688d5b2f-0756-480e-83f2-66e05c48527b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/688d5b2f-0756-480e-83f2-66e05c48527b-dshm\") pod \"688d5b2f-0756-480e-83f2-66e05c48527b\" (UID: \"688d5b2f-0756-480e-83f2-66e05c48527b\") " Oct 31 13:06:30.782372 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:30.782227 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/688d5b2f-0756-480e-83f2-66e05c48527b-home" pod "688d5b2f-0756-480e-83f2-66e05c48527b" (UID: "688d5b2f-0756-480e-83f2-66e05c48527b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:06:30.782646 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:30.782625 6841 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/688d5b2f-0756-480e-83f2-66e05c48527b-model-cache\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:30.782646 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:30.782646 6841 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/688d5b2f-0756-480e-83f2-66e05c48527b-tmp-dir\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:30.782849 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:30.782669 6841 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/688d5b2f-0756-480e-83f2-66e05c48527b-home\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:30.784341 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:30.784313 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/688d5b2f-0756-480e-83f2-66e05c48527b-tls-certs" pod "688d5b2f-0756-480e-83f2-66e05c48527b" (UID: "688d5b2f-0756-480e-83f2-66e05c48527b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:06:30.784441 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:30.784337 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/688d5b2f-0756-480e-83f2-66e05c48527b-dshm" pod "688d5b2f-0756-480e-83f2-66e05c48527b" (UID: "688d5b2f-0756-480e-83f2-66e05c48527b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:06:30.784441 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:30.784359 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/688d5b2f-0756-480e-83f2-66e05c48527b-kube-api-access-tcs4f" pod "688d5b2f-0756-480e-83f2-66e05c48527b" (UID: "688d5b2f-0756-480e-83f2-66e05c48527b"). InnerVolumeSpecName "kube-api-access-tcs4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:06:30.883712 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:30.883657 6841 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/688d5b2f-0756-480e-83f2-66e05c48527b-tls-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:30.883712 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:30.883705 6841 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/688d5b2f-0756-480e-83f2-66e05c48527b-dshm\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:30.883712 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:30.883718 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcs4f\" (UniqueName: \"kubernetes.io/projected/688d5b2f-0756-480e-83f2-66e05c48527b-kube-api-access-tcs4f\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:30.896568 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:30.896535 6841 generic.go:356] "Generic (PLEG): container finished" podID="688d5b2f-0756-480e-83f2-66e05c48527b" containerID="948fc3e8d748982dab2910d12204f0a9e5cf3a86fcdf7f653ce1e30973023fbd" exitCode=0 Oct 31 13:06:30.896741 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:30.896597 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" Oct 31 13:06:30.896741 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:30.896612 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" event={"ID":"688d5b2f-0756-480e-83f2-66e05c48527b","Type":"ContainerDied","Data":"948fc3e8d748982dab2910d12204f0a9e5cf3a86fcdf7f653ce1e30973023fbd"} Oct 31 13:06:30.896741 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:30.896679 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh" event={"ID":"688d5b2f-0756-480e-83f2-66e05c48527b","Type":"ContainerDied","Data":"488ee03cbc25cc26aff7f1924ebfa573a6b174202d2e11644e23d4521f8ff323"} Oct 31 13:06:30.896741 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:30.896711 6841 scope.go:122] "RemoveContainer" containerID="948fc3e8d748982dab2910d12204f0a9e5cf3a86fcdf7f653ce1e30973023fbd" Oct 31 13:06:30.908220 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:30.908200 6841 scope.go:122] "RemoveContainer" containerID="948fc3e8d748982dab2910d12204f0a9e5cf3a86fcdf7f653ce1e30973023fbd" Oct 31 13:06:30.908491 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:06:30.908468 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"948fc3e8d748982dab2910d12204f0a9e5cf3a86fcdf7f653ce1e30973023fbd\": container with ID starting with 948fc3e8d748982dab2910d12204f0a9e5cf3a86fcdf7f653ce1e30973023fbd not found: ID does not exist" containerID="948fc3e8d748982dab2910d12204f0a9e5cf3a86fcdf7f653ce1e30973023fbd" Oct 31 13:06:30.908586 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:30.908503 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"948fc3e8d748982dab2910d12204f0a9e5cf3a86fcdf7f653ce1e30973023fbd"} err="failed to get container status \"948fc3e8d748982dab2910d12204f0a9e5cf3a86fcdf7f653ce1e30973023fbd\": rpc error: code = NotFound desc = could not find container \"948fc3e8d748982dab2910d12204f0a9e5cf3a86fcdf7f653ce1e30973023fbd\": container with ID starting with 948fc3e8d748982dab2910d12204f0a9e5cf3a86fcdf7f653ce1e30973023fbd not found: ID does not exist" Oct 31 13:06:30.923144 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:30.923120 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh"] Oct 31 13:06:30.928100 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:30.928071 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-758f48/praxis-route-758f48-kserve-7bc886646f-rsfdh"] Oct 31 13:06:31.549092 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:31.549043 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:31.643522 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:31.643490 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="688d5b2f-0756-480e-83f2-66e05c48527b" path="/var/lib/kubelet/pods/688d5b2f-0756-480e-83f2-66e05c48527b/volumes" Oct 31 13:06:31.931885 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:31.931803 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:32.139215 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:32.139182 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:32.398114 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:32.398074 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:32.613064 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:32.613028 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:32.699360 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:32.699326 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2"] Oct 31 13:06:32.699628 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:32.699597 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" podUID="5ebddec1-8fab-4709-84de-f5d99a9112d0" containerName="istio-proxy" containerID="cri-o://748689c72487261e6e22fc1cf5e3d8a77c9097ebc0c732b4d40b3c95be11a3ae" gracePeriod=30 Oct 31 13:06:32.858580 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:32.858543 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:33.483661 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:33.483626 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:33.716624 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:33.716580 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:34.230520 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:34.230340 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:35.103633 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:35.103599 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:35.640186 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:35.640155 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:35.919259 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:35.919164 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:36.741375 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:36.741343 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:37.182679 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:37.182587 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:37.402541 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:37.402508 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:37.607150 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:37.607111 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:37.672791 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:37.672756 6841 patch_prober.go:29] interesting pod/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:06:37.673005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:37.672823 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" podUID="5ebddec1-8fab-4709-84de-f5d99a9112d0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:06:38.001822 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.001797 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:06:38.057747 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.057711 6841 generic.go:356] "Generic (PLEG): container finished" podID="5ebddec1-8fab-4709-84de-f5d99a9112d0" containerID="748689c72487261e6e22fc1cf5e3d8a77c9097ebc0c732b4d40b3c95be11a3ae" exitCode=0 Oct 31 13:06:38.057933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.057765 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" Oct 31 13:06:38.057933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.057764 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" event={"ID":"5ebddec1-8fab-4709-84de-f5d99a9112d0","Type":"ContainerDied","Data":"748689c72487261e6e22fc1cf5e3d8a77c9097ebc0c732b4d40b3c95be11a3ae"} Oct 31 13:06:38.057933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.057899 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2" event={"ID":"5ebddec1-8fab-4709-84de-f5d99a9112d0","Type":"ContainerDied","Data":"898819c3b72b1ba1a854ab5269f3b7359d540f886c0a018c0eb33032b927f142"} Oct 31 13:06:38.057933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.057932 6841 scope.go:122] "RemoveContainer" containerID="748689c72487261e6e22fc1cf5e3d8a77c9097ebc0c732b4d40b3c95be11a3ae" Oct 31 13:06:38.069608 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.069575 6841 scope.go:122] "RemoveContainer" containerID="d77b8f1b938e25220d58eb6715bbe7d0fbe7997a420a829dcd16138ef40026fd" Oct 31 13:06:38.079375 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.079356 6841 scope.go:122] "RemoveContainer" containerID="748689c72487261e6e22fc1cf5e3d8a77c9097ebc0c732b4d40b3c95be11a3ae" Oct 31 13:06:38.079675 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:06:38.079647 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"748689c72487261e6e22fc1cf5e3d8a77c9097ebc0c732b4d40b3c95be11a3ae\": container with ID starting with 748689c72487261e6e22fc1cf5e3d8a77c9097ebc0c732b4d40b3c95be11a3ae not found: ID does not exist" containerID="748689c72487261e6e22fc1cf5e3d8a77c9097ebc0c732b4d40b3c95be11a3ae" Oct 31 13:06:38.079742 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.079684 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"748689c72487261e6e22fc1cf5e3d8a77c9097ebc0c732b4d40b3c95be11a3ae"} err="failed to get container status \"748689c72487261e6e22fc1cf5e3d8a77c9097ebc0c732b4d40b3c95be11a3ae\": rpc error: code = NotFound desc = could not find container \"748689c72487261e6e22fc1cf5e3d8a77c9097ebc0c732b4d40b3c95be11a3ae\": container with ID starting with 748689c72487261e6e22fc1cf5e3d8a77c9097ebc0c732b4d40b3c95be11a3ae not found: ID does not exist" Oct 31 13:06:38.079742 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.079702 6841 scope.go:122] "RemoveContainer" containerID="d77b8f1b938e25220d58eb6715bbe7d0fbe7997a420a829dcd16138ef40026fd" Oct 31 13:06:38.080016 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:06:38.079975 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d77b8f1b938e25220d58eb6715bbe7d0fbe7997a420a829dcd16138ef40026fd\": container with ID starting with d77b8f1b938e25220d58eb6715bbe7d0fbe7997a420a829dcd16138ef40026fd not found: ID does not exist" containerID="d77b8f1b938e25220d58eb6715bbe7d0fbe7997a420a829dcd16138ef40026fd" Oct 31 13:06:38.080105 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.080024 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d77b8f1b938e25220d58eb6715bbe7d0fbe7997a420a829dcd16138ef40026fd"} err="failed to get container status \"d77b8f1b938e25220d58eb6715bbe7d0fbe7997a420a829dcd16138ef40026fd\": rpc error: code = NotFound desc = could not find container \"d77b8f1b938e25220d58eb6715bbe7d0fbe7997a420a829dcd16138ef40026fd\": container with ID starting with d77b8f1b938e25220d58eb6715bbe7d0fbe7997a420a829dcd16138ef40026fd not found: ID does not exist" Oct 31 13:06:38.121611 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.121583 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ebddec1-8fab-4709-84de-f5d99a9112d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ebddec1-8fab-4709-84de-f5d99a9112d0-workload-certs\") pod \"5ebddec1-8fab-4709-84de-f5d99a9112d0\" (UID: \"5ebddec1-8fab-4709-84de-f5d99a9112d0\") " Oct 31 13:06:38.121754 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.121681 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ebddec1-8fab-4709-84de-f5d99a9112d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ebddec1-8fab-4709-84de-f5d99a9112d0-istio-data\") pod \"5ebddec1-8fab-4709-84de-f5d99a9112d0\" (UID: \"5ebddec1-8fab-4709-84de-f5d99a9112d0\") " Oct 31 13:06:38.121754 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.121736 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ebddec1-8fab-4709-84de-f5d99a9112d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ebddec1-8fab-4709-84de-f5d99a9112d0-credential-socket\") pod \"5ebddec1-8fab-4709-84de-f5d99a9112d0\" (UID: \"5ebddec1-8fab-4709-84de-f5d99a9112d0\") " Oct 31 13:06:38.121879 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.121851 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ebddec1-8fab-4709-84de-f5d99a9112d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ebddec1-8fab-4709-84de-f5d99a9112d0-workload-socket\") pod \"5ebddec1-8fab-4709-84de-f5d99a9112d0\" (UID: \"5ebddec1-8fab-4709-84de-f5d99a9112d0\") " Oct 31 13:06:38.121943 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.121917 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ebddec1-8fab-4709-84de-f5d99a9112d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ebddec1-8fab-4709-84de-f5d99a9112d0-istiod-ca-cert\") pod \"5ebddec1-8fab-4709-84de-f5d99a9112d0\" (UID: \"5ebddec1-8fab-4709-84de-f5d99a9112d0\") " Oct 31 13:06:38.122032 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.121942 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ebddec1-8fab-4709-84de-f5d99a9112d0-istio-token\" (UniqueName: \"kubernetes.io/projected/5ebddec1-8fab-4709-84de-f5d99a9112d0-istio-token\") pod \"5ebddec1-8fab-4709-84de-f5d99a9112d0\" (UID: \"5ebddec1-8fab-4709-84de-f5d99a9112d0\") " Oct 31 13:06:38.122032 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.121842 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ebddec1-8fab-4709-84de-f5d99a9112d0-workload-certs" pod "5ebddec1-8fab-4709-84de-f5d99a9112d0" (UID: "5ebddec1-8fab-4709-84de-f5d99a9112d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:06:38.122032 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.121978 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ebddec1-8fab-4709-84de-f5d99a9112d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ebddec1-8fab-4709-84de-f5d99a9112d0-istio-podinfo\") pod \"5ebddec1-8fab-4709-84de-f5d99a9112d0\" (UID: \"5ebddec1-8fab-4709-84de-f5d99a9112d0\") " Oct 31 13:06:38.122190 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.122046 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ebddec1-8fab-4709-84de-f5d99a9112d0-istio-data" pod "5ebddec1-8fab-4709-84de-f5d99a9112d0" (UID: "5ebddec1-8fab-4709-84de-f5d99a9112d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:06:38.122190 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.122083 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ebddec1-8fab-4709-84de-f5d99a9112d0-credential-socket" pod "5ebddec1-8fab-4709-84de-f5d99a9112d0" (UID: "5ebddec1-8fab-4709-84de-f5d99a9112d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:06:38.122190 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.122106 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ebddec1-8fab-4709-84de-f5d99a9112d0-workload-socket" pod "5ebddec1-8fab-4709-84de-f5d99a9112d0" (UID: "5ebddec1-8fab-4709-84de-f5d99a9112d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:06:38.122190 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.122157 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ebddec1-8fab-4709-84de-f5d99a9112d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ebddec1-8fab-4709-84de-f5d99a9112d0-istio-envoy\") pod \"5ebddec1-8fab-4709-84de-f5d99a9112d0\" (UID: \"5ebddec1-8fab-4709-84de-f5d99a9112d0\") " Oct 31 13:06:38.122360 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.122199 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ebddec1-8fab-4709-84de-f5d99a9112d0-kube-api-access-s6dxr\" (UniqueName: \"kubernetes.io/projected/5ebddec1-8fab-4709-84de-f5d99a9112d0-kube-api-access-s6dxr\") pod \"5ebddec1-8fab-4709-84de-f5d99a9112d0\" (UID: \"5ebddec1-8fab-4709-84de-f5d99a9112d0\") " Oct 31 13:06:38.122360 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.122288 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ebddec1-8fab-4709-84de-f5d99a9112d0-istiod-ca-cert" pod "5ebddec1-8fab-4709-84de-f5d99a9112d0" (UID: "5ebddec1-8fab-4709-84de-f5d99a9112d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:06:38.122838 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.122816 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ebddec1-8fab-4709-84de-f5d99a9112d0-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:38.122838 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.122837 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ebddec1-8fab-4709-84de-f5d99a9112d0-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:38.123047 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.122846 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ebddec1-8fab-4709-84de-f5d99a9112d0-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:38.123047 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.122855 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ebddec1-8fab-4709-84de-f5d99a9112d0-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:38.123047 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.122863 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ebddec1-8fab-4709-84de-f5d99a9112d0-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:38.124201 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.124147 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ebddec1-8fab-4709-84de-f5d99a9112d0-istio-podinfo" pod "5ebddec1-8fab-4709-84de-f5d99a9112d0" (UID: "5ebddec1-8fab-4709-84de-f5d99a9112d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:06:38.124333 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.124284 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ebddec1-8fab-4709-84de-f5d99a9112d0-istio-envoy" pod "5ebddec1-8fab-4709-84de-f5d99a9112d0" (UID: "5ebddec1-8fab-4709-84de-f5d99a9112d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:06:38.124623 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.124606 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ebddec1-8fab-4709-84de-f5d99a9112d0-istio-token" pod "5ebddec1-8fab-4709-84de-f5d99a9112d0" (UID: "5ebddec1-8fab-4709-84de-f5d99a9112d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:06:38.124665 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.124652 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ebddec1-8fab-4709-84de-f5d99a9112d0-kube-api-access-s6dxr" pod "5ebddec1-8fab-4709-84de-f5d99a9112d0" (UID: "5ebddec1-8fab-4709-84de-f5d99a9112d0"). InnerVolumeSpecName "kube-api-access-s6dxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:06:38.224050 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.224020 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ebddec1-8fab-4709-84de-f5d99a9112d0-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:38.224050 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.224047 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ebddec1-8fab-4709-84de-f5d99a9112d0-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:38.224050 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.224057 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ebddec1-8fab-4709-84de-f5d99a9112d0-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:38.224286 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.224077 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6dxr\" (UniqueName: \"kubernetes.io/projected/5ebddec1-8fab-4709-84de-f5d99a9112d0-kube-api-access-s6dxr\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:38.425156 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.421776 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2"] Oct 31 13:06:38.427061 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.426366 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:38.436350 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:38.436320 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-7e08dc-openshift-default-f577fb8cc-2n4v2"] Oct 31 13:06:39.203044 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:39.202360 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:39.483443 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:39.483368 6841 scope.go:122] "RemoveContainer" containerID="e643d407546465bed47e05556bfd61678943f64bb5b28d315a8431a8c985cbf4" Oct 31 13:06:39.643686 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:39.643654 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ebddec1-8fab-4709-84de-f5d99a9112d0" path="/var/lib/kubelet/pods/5ebddec1-8fab-4709-84de-f5d99a9112d0/volumes" Oct 31 13:06:39.859202 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:39.859122 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:40.083004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:40.077674 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:40.281592 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:40.281549 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:40.492957 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:40.492925 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:41.197615 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:41.197568 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:42.101643 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:42.101597 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:42.771614 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:42.771580 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:43.023137 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:43.023048 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:43.830609 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:43.830567 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:44.514920 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:44.514880 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:45.106147 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:45.106107 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28"] Oct 31 13:06:45.106724 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:45.106691 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28" podUID="129ee68e-30b6-4e0c-97e8-85f780b2cab6" containerName="maas-api" containerID="cri-o://04ee0f7e7a2320a1554d9a0bde753ca2cb701654f3fbfbba9e6f1b5432d51049" gracePeriod=30 Oct 31 13:06:45.247384 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:45.247354 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:45.632314 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:45.632274 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28" Oct 31 13:06:45.687908 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:45.687877 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:45.814684 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:45.814574 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/129ee68e-30b6-4e0c-97e8-85f780b2cab6-kube-api-access-znzjd\" (UniqueName: \"kubernetes.io/projected/129ee68e-30b6-4e0c-97e8-85f780b2cab6-kube-api-access-znzjd\") pod \"129ee68e-30b6-4e0c-97e8-85f780b2cab6\" (UID: \"129ee68e-30b6-4e0c-97e8-85f780b2cab6\") " Oct 31 13:06:45.815032 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:45.815003 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/129ee68e-30b6-4e0c-97e8-85f780b2cab6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/129ee68e-30b6-4e0c-97e8-85f780b2cab6-maas-api-tls\") pod \"129ee68e-30b6-4e0c-97e8-85f780b2cab6\" (UID: \"129ee68e-30b6-4e0c-97e8-85f780b2cab6\") " Oct 31 13:06:45.815441 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:45.815420 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/129ee68e-30b6-4e0c-97e8-85f780b2cab6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/129ee68e-30b6-4e0c-97e8-85f780b2cab6-metrics-certs\") pod \"129ee68e-30b6-4e0c-97e8-85f780b2cab6\" (UID: \"129ee68e-30b6-4e0c-97e8-85f780b2cab6\") " Oct 31 13:06:45.817166 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:45.817126 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/129ee68e-30b6-4e0c-97e8-85f780b2cab6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/129ee68e-30b6-4e0c-97e8-85f780b2cab6-openshift-service-ca\") pod \"129ee68e-30b6-4e0c-97e8-85f780b2cab6\" (UID: \"129ee68e-30b6-4e0c-97e8-85f780b2cab6\") " Oct 31 13:06:45.817768 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:45.817720 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/129ee68e-30b6-4e0c-97e8-85f780b2cab6-openshift-service-ca" pod "129ee68e-30b6-4e0c-97e8-85f780b2cab6" (UID: "129ee68e-30b6-4e0c-97e8-85f780b2cab6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:06:45.817902 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:45.817883 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/129ee68e-30b6-4e0c-97e8-85f780b2cab6-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:45.819243 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:45.819216 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/129ee68e-30b6-4e0c-97e8-85f780b2cab6-maas-api-tls" pod "129ee68e-30b6-4e0c-97e8-85f780b2cab6" (UID: "129ee68e-30b6-4e0c-97e8-85f780b2cab6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:06:45.819479 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:45.819457 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/129ee68e-30b6-4e0c-97e8-85f780b2cab6-metrics-certs" pod "129ee68e-30b6-4e0c-97e8-85f780b2cab6" (UID: "129ee68e-30b6-4e0c-97e8-85f780b2cab6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:06:45.819905 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:45.819879 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/129ee68e-30b6-4e0c-97e8-85f780b2cab6-kube-api-access-znzjd" pod "129ee68e-30b6-4e0c-97e8-85f780b2cab6" (UID: "129ee68e-30b6-4e0c-97e8-85f780b2cab6"). InnerVolumeSpecName "kube-api-access-znzjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:06:45.919597 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:45.919568 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znzjd\" (UniqueName: \"kubernetes.io/projected/129ee68e-30b6-4e0c-97e8-85f780b2cab6-kube-api-access-znzjd\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:45.919597 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:45.919594 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/129ee68e-30b6-4e0c-97e8-85f780b2cab6-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:45.919765 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:45.919604 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/129ee68e-30b6-4e0c-97e8-85f780b2cab6-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:46.162271 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:46.162178 6841 generic.go:356] "Generic (PLEG): container finished" podID="129ee68e-30b6-4e0c-97e8-85f780b2cab6" containerID="04ee0f7e7a2320a1554d9a0bde753ca2cb701654f3fbfbba9e6f1b5432d51049" exitCode=0 Oct 31 13:06:46.162271 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:46.162243 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28" event={"ID":"129ee68e-30b6-4e0c-97e8-85f780b2cab6","Type":"ContainerDied","Data":"04ee0f7e7a2320a1554d9a0bde753ca2cb701654f3fbfbba9e6f1b5432d51049"} Oct 31 13:06:46.162271 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:46.162250 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28" Oct 31 13:06:46.163087 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:46.162311 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28" event={"ID":"129ee68e-30b6-4e0c-97e8-85f780b2cab6","Type":"ContainerDied","Data":"00a9ea9a9bb63dbbd9033732fa9322ba72b82bdc20f2924e143fb10b725dc108"} Oct 31 13:06:46.163087 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:46.162333 6841 scope.go:122] "RemoveContainer" containerID="04ee0f7e7a2320a1554d9a0bde753ca2cb701654f3fbfbba9e6f1b5432d51049" Oct 31 13:06:46.173565 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:46.173539 6841 scope.go:122] "RemoveContainer" containerID="04ee0f7e7a2320a1554d9a0bde753ca2cb701654f3fbfbba9e6f1b5432d51049" Oct 31 13:06:46.173842 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:06:46.173823 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04ee0f7e7a2320a1554d9a0bde753ca2cb701654f3fbfbba9e6f1b5432d51049\": container with ID starting with 04ee0f7e7a2320a1554d9a0bde753ca2cb701654f3fbfbba9e6f1b5432d51049 not found: ID does not exist" containerID="04ee0f7e7a2320a1554d9a0bde753ca2cb701654f3fbfbba9e6f1b5432d51049" Oct 31 13:06:46.173905 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:46.173851 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04ee0f7e7a2320a1554d9a0bde753ca2cb701654f3fbfbba9e6f1b5432d51049"} err="failed to get container status \"04ee0f7e7a2320a1554d9a0bde753ca2cb701654f3fbfbba9e6f1b5432d51049\": rpc error: code = NotFound desc = could not find container \"04ee0f7e7a2320a1554d9a0bde753ca2cb701654f3fbfbba9e6f1b5432d51049\": container with ID starting with 04ee0f7e7a2320a1554d9a0bde753ca2cb701654f3fbfbba9e6f1b5432d51049 not found: ID does not exist" Oct 31 13:06:46.188406 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:46.188380 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28"] Oct 31 13:06:46.192284 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:46.192262 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a3000-89788d5fd-7sl28"] Oct 31 13:06:46.551004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:46.539334 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4989f9-67d6485cdd-xndwk"] Oct 31 13:06:46.551004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:46.539567 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-4989f9-67d6485cdd-xndwk" podUID="e024b9a0-c5a3-4850-914a-fe60e0c6f50e" containerName="payload-processing" containerID="cri-o://d7ef6d4a67dcfb8f9dc0b7aca5be7cf45cac2bf4bb0a6c9ec81cc6fbc86bed52" gracePeriod=30 Oct 31 13:06:46.556515 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:46.556478 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-6f58649d67-qjsfw"] Oct 31 13:06:46.556702 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:46.556681 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-6f58649d67-qjsfw" podUID="d03bab9c-ade2-43da-8bb1-37fc53fd93f9" containerName="payload-pre-processing" containerID="cri-o://500b0e45b909b37cffe613d3f3b32e10882ba995055d5722fcc0f9233618774f" gracePeriod=30 Oct 31 13:06:46.607620 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:46.606946 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:47.446438 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:47.441210 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-6f58649d67-qjsfw" Oct 31 13:06:47.481006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:47.472921 6841 generic.go:356] "Generic (PLEG): container finished" podID="e024b9a0-c5a3-4850-914a-fe60e0c6f50e" containerID="d7ef6d4a67dcfb8f9dc0b7aca5be7cf45cac2bf4bb0a6c9ec81cc6fbc86bed52" exitCode=0 Oct 31 13:06:47.481006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:47.473034 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4989f9-67d6485cdd-xndwk" event={"ID":"e024b9a0-c5a3-4850-914a-fe60e0c6f50e","Type":"ContainerDied","Data":"d7ef6d4a67dcfb8f9dc0b7aca5be7cf45cac2bf4bb0a6c9ec81cc6fbc86bed52"} Oct 31 13:06:47.515021 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:47.505093 6841 generic.go:356] "Generic (PLEG): container finished" podID="d03bab9c-ade2-43da-8bb1-37fc53fd93f9" containerID="500b0e45b909b37cffe613d3f3b32e10882ba995055d5722fcc0f9233618774f" exitCode=0 Oct 31 13:06:47.515021 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:47.505141 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-6f58649d67-qjsfw" event={"ID":"d03bab9c-ade2-43da-8bb1-37fc53fd93f9","Type":"ContainerDied","Data":"500b0e45b909b37cffe613d3f3b32e10882ba995055d5722fcc0f9233618774f"} Oct 31 13:06:47.515021 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:47.505197 6841 scope.go:122] "RemoveContainer" containerID="500b0e45b909b37cffe613d3f3b32e10882ba995055d5722fcc0f9233618774f" Oct 31 13:06:47.515021 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:47.505300 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-6f58649d67-qjsfw" Oct 31 13:06:47.555465 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:47.546190 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4989f9-67d6485cdd-xndwk" Oct 31 13:06:47.555465 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:47.549796 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d03bab9c-ade2-43da-8bb1-37fc53fd93f9-kube-api-access-2f47h\" (UniqueName: \"kubernetes.io/projected/d03bab9c-ade2-43da-8bb1-37fc53fd93f9-kube-api-access-2f47h\") pod \"d03bab9c-ade2-43da-8bb1-37fc53fd93f9\" (UID: \"d03bab9c-ade2-43da-8bb1-37fc53fd93f9\") " Oct 31 13:06:47.555465 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:47.549932 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d03bab9c-ade2-43da-8bb1-37fc53fd93f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d03bab9c-ade2-43da-8bb1-37fc53fd93f9-plugins-config-volume\") pod \"d03bab9c-ade2-43da-8bb1-37fc53fd93f9\" (UID: \"d03bab9c-ade2-43da-8bb1-37fc53fd93f9\") " Oct 31 13:06:47.555465 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:47.551272 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d03bab9c-ade2-43da-8bb1-37fc53fd93f9-plugins-config-volume" pod "d03bab9c-ade2-43da-8bb1-37fc53fd93f9" (UID: "d03bab9c-ade2-43da-8bb1-37fc53fd93f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:06:47.683548 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:47.658338 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e024b9a0-c5a3-4850-914a-fe60e0c6f50e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e024b9a0-c5a3-4850-914a-fe60e0c6f50e-plugins-config-volume\") pod \"e024b9a0-c5a3-4850-914a-fe60e0c6f50e\" (UID: \"e024b9a0-c5a3-4850-914a-fe60e0c6f50e\") " Oct 31 13:06:47.683548 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:47.658441 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e024b9a0-c5a3-4850-914a-fe60e0c6f50e-kube-api-access-qd92l\" (UniqueName: \"kubernetes.io/projected/e024b9a0-c5a3-4850-914a-fe60e0c6f50e-kube-api-access-qd92l\") pod \"e024b9a0-c5a3-4850-914a-fe60e0c6f50e\" (UID: \"e024b9a0-c5a3-4850-914a-fe60e0c6f50e\") " Oct 31 13:06:47.683548 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:47.659002 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d03bab9c-ade2-43da-8bb1-37fc53fd93f9-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:47.684760 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:47.684122 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e024b9a0-c5a3-4850-914a-fe60e0c6f50e-plugins-config-volume" pod "e024b9a0-c5a3-4850-914a-fe60e0c6f50e" (UID: "e024b9a0-c5a3-4850-914a-fe60e0c6f50e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:06:47.706007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:47.693126 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d03bab9c-ade2-43da-8bb1-37fc53fd93f9-kube-api-access-2f47h" pod "d03bab9c-ade2-43da-8bb1-37fc53fd93f9" (UID: "d03bab9c-ade2-43da-8bb1-37fc53fd93f9"). InnerVolumeSpecName "kube-api-access-2f47h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:06:47.748297 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:47.744781 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e024b9a0-c5a3-4850-914a-fe60e0c6f50e-kube-api-access-qd92l" pod "e024b9a0-c5a3-4850-914a-fe60e0c6f50e" (UID: "e024b9a0-c5a3-4850-914a-fe60e0c6f50e"). InnerVolumeSpecName "kube-api-access-qd92l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:06:47.802950 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:47.802918 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e024b9a0-c5a3-4850-914a-fe60e0c6f50e-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:47.803235 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:47.803214 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd92l\" (UniqueName: \"kubernetes.io/projected/e024b9a0-c5a3-4850-914a-fe60e0c6f50e-kube-api-access-qd92l\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:47.841057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:47.838061 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f47h\" (UniqueName: \"kubernetes.io/projected/d03bab9c-ade2-43da-8bb1-37fc53fd93f9-kube-api-access-2f47h\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:47.841057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:47.831671 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="129ee68e-30b6-4e0c-97e8-85f780b2cab6" path="/var/lib/kubelet/pods/129ee68e-30b6-4e0c-97e8-85f780b2cab6/volumes" Oct 31 13:06:48.008600 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:48.008166 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-6f58649d67-qjsfw"] Oct 31 13:06:48.064009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:48.058965 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-6f58649d67-qjsfw"] Oct 31 13:06:48.388813 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:48.388716 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:48.578140 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:48.569753 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4989f9-67d6485cdd-xndwk" event={"ID":"e024b9a0-c5a3-4850-914a-fe60e0c6f50e","Type":"ContainerDied","Data":"21328748cb76e12e8dffd16d90e91dc9b57168332c319a2b305b0e6f075e319e"} Oct 31 13:06:48.578140 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:48.569832 6841 scope.go:122] "RemoveContainer" containerID="d7ef6d4a67dcfb8f9dc0b7aca5be7cf45cac2bf4bb0a6c9ec81cc6fbc86bed52" Oct 31 13:06:48.578140 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:48.569964 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4989f9-67d6485cdd-xndwk" Oct 31 13:06:48.734363 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:48.730176 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4989f9-67d6485cdd-xndwk"] Oct 31 13:06:48.734363 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:48.734043 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:48.764054 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:48.764018 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4989f9-67d6485cdd-xndwk"] Oct 31 13:06:49.842008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:49.819840 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:49.962018 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:49.952684 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d03bab9c-ade2-43da-8bb1-37fc53fd93f9" path="/var/lib/kubelet/pods/d03bab9c-ade2-43da-8bb1-37fc53fd93f9/volumes" Oct 31 13:06:49.962018 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:49.953545 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e024b9a0-c5a3-4850-914a-fe60e0c6f50e" path="/var/lib/kubelet/pods/e024b9a0-c5a3-4850-914a-fe60e0c6f50e/volumes" Oct 31 13:06:50.248830 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:50.248795 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:50.400341 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:06:50.400294 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:06:50.626684 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:50.626594 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:50.895732 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:50.895616 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:51.136014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:51.135937 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:51.432777 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:51.432740 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf"] Oct 31 13:06:51.433038 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:51.433012 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" podUID="84a47faf-ed37-4499-abcb-cd5fec47dc13" containerName="istio-proxy" containerID="cri-o://162fd22a616ae574ea364983c82388208f3b2d79d12c7f32996881784a6bd0a3" gracePeriod=30 Oct 31 13:06:51.606101 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:51.601886 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:51.831005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:51.830534 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:52.148610 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:52.148444 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:53.133591 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:53.130657 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:54.167546 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:54.167511 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:54.631136 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:54.631098 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:54.963921 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:54.963891 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:55.325734 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:55.325653 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:55.590056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:55.589940 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:56.242915 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:56.242882 6841 patch_prober.go:29] interesting pod/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:06:56.243107 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:56.242947 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" podUID="84a47faf-ed37-4499-abcb-cd5fec47dc13" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:06:56.441805 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:56.441774 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:56.701849 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:56.701821 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" Oct 31 13:06:56.742184 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:56.742151 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a47faf-ed37-4499-abcb-cd5fec47dc13-istio-token\" (UniqueName: \"kubernetes.io/projected/84a47faf-ed37-4499-abcb-cd5fec47dc13-istio-token\") pod \"84a47faf-ed37-4499-abcb-cd5fec47dc13\" (UID: \"84a47faf-ed37-4499-abcb-cd5fec47dc13\") " Oct 31 13:06:56.742363 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:56.742256 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84a47faf-ed37-4499-abcb-cd5fec47dc13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a47faf-ed37-4499-abcb-cd5fec47dc13-istiod-ca-cert\") pod \"84a47faf-ed37-4499-abcb-cd5fec47dc13\" (UID: \"84a47faf-ed37-4499-abcb-cd5fec47dc13\") " Oct 31 13:06:56.742363 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:56.742307 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a47faf-ed37-4499-abcb-cd5fec47dc13-kube-api-access-fvls5\" (UniqueName: \"kubernetes.io/projected/84a47faf-ed37-4499-abcb-cd5fec47dc13-kube-api-access-fvls5\") pod \"84a47faf-ed37-4499-abcb-cd5fec47dc13\" (UID: \"84a47faf-ed37-4499-abcb-cd5fec47dc13\") " Oct 31 13:06:56.742363 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:56.742339 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a47faf-ed37-4499-abcb-cd5fec47dc13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a47faf-ed37-4499-abcb-cd5fec47dc13-workload-certs\") pod \"84a47faf-ed37-4499-abcb-cd5fec47dc13\" (UID: \"84a47faf-ed37-4499-abcb-cd5fec47dc13\") " Oct 31 13:06:56.742548 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:56.742389 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a47faf-ed37-4499-abcb-cd5fec47dc13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a47faf-ed37-4499-abcb-cd5fec47dc13-workload-socket\") pod \"84a47faf-ed37-4499-abcb-cd5fec47dc13\" (UID: \"84a47faf-ed37-4499-abcb-cd5fec47dc13\") " Oct 31 13:06:56.742548 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:56.742451 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84a47faf-ed37-4499-abcb-cd5fec47dc13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a47faf-ed37-4499-abcb-cd5fec47dc13-istio-podinfo\") pod \"84a47faf-ed37-4499-abcb-cd5fec47dc13\" (UID: \"84a47faf-ed37-4499-abcb-cd5fec47dc13\") " Oct 31 13:06:56.742548 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:56.742483 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a47faf-ed37-4499-abcb-cd5fec47dc13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a47faf-ed37-4499-abcb-cd5fec47dc13-credential-socket\") pod \"84a47faf-ed37-4499-abcb-cd5fec47dc13\" (UID: \"84a47faf-ed37-4499-abcb-cd5fec47dc13\") " Oct 31 13:06:56.742739 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:56.742551 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a47faf-ed37-4499-abcb-cd5fec47dc13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a47faf-ed37-4499-abcb-cd5fec47dc13-istio-envoy\") pod \"84a47faf-ed37-4499-abcb-cd5fec47dc13\" (UID: \"84a47faf-ed37-4499-abcb-cd5fec47dc13\") " Oct 31 13:06:56.742739 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:56.742644 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a47faf-ed37-4499-abcb-cd5fec47dc13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a47faf-ed37-4499-abcb-cd5fec47dc13-istio-data\") pod \"84a47faf-ed37-4499-abcb-cd5fec47dc13\" (UID: \"84a47faf-ed37-4499-abcb-cd5fec47dc13\") " Oct 31 13:06:56.742739 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:56.742697 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84a47faf-ed37-4499-abcb-cd5fec47dc13-istiod-ca-cert" pod "84a47faf-ed37-4499-abcb-cd5fec47dc13" (UID: "84a47faf-ed37-4499-abcb-cd5fec47dc13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:06:56.742974 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:56.742953 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a47faf-ed37-4499-abcb-cd5fec47dc13-workload-socket" pod "84a47faf-ed37-4499-abcb-cd5fec47dc13" (UID: "84a47faf-ed37-4499-abcb-cd5fec47dc13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:06:56.743210 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:56.743188 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a47faf-ed37-4499-abcb-cd5fec47dc13-istio-data" pod "84a47faf-ed37-4499-abcb-cd5fec47dc13" (UID: "84a47faf-ed37-4499-abcb-cd5fec47dc13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:06:56.743417 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:56.743397 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a47faf-ed37-4499-abcb-cd5fec47dc13-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:56.743488 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:56.743421 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a47faf-ed37-4499-abcb-cd5fec47dc13-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:56.743488 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:56.743436 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a47faf-ed37-4499-abcb-cd5fec47dc13-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:56.743901 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:56.743620 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a47faf-ed37-4499-abcb-cd5fec47dc13-credential-socket" pod "84a47faf-ed37-4499-abcb-cd5fec47dc13" (UID: "84a47faf-ed37-4499-abcb-cd5fec47dc13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:06:56.743901 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:56.743686 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a47faf-ed37-4499-abcb-cd5fec47dc13-workload-certs" pod "84a47faf-ed37-4499-abcb-cd5fec47dc13" (UID: "84a47faf-ed37-4499-abcb-cd5fec47dc13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:06:56.744975 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:56.744947 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a47faf-ed37-4499-abcb-cd5fec47dc13-istio-token" pod "84a47faf-ed37-4499-abcb-cd5fec47dc13" (UID: "84a47faf-ed37-4499-abcb-cd5fec47dc13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:06:56.745865 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:56.745835 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a47faf-ed37-4499-abcb-cd5fec47dc13-kube-api-access-fvls5" pod "84a47faf-ed37-4499-abcb-cd5fec47dc13" (UID: "84a47faf-ed37-4499-abcb-cd5fec47dc13"). InnerVolumeSpecName "kube-api-access-fvls5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:06:56.745966 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:56.745918 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a47faf-ed37-4499-abcb-cd5fec47dc13-istio-envoy" pod "84a47faf-ed37-4499-abcb-cd5fec47dc13" (UID: "84a47faf-ed37-4499-abcb-cd5fec47dc13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:06:56.746324 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:56.746295 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84a47faf-ed37-4499-abcb-cd5fec47dc13-istio-podinfo" pod "84a47faf-ed37-4499-abcb-cd5fec47dc13" (UID: "84a47faf-ed37-4499-abcb-cd5fec47dc13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:06:56.844850 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:56.844763 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a47faf-ed37-4499-abcb-cd5fec47dc13-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:56.844850 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:56.844794 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvls5\" (UniqueName: \"kubernetes.io/projected/84a47faf-ed37-4499-abcb-cd5fec47dc13-kube-api-access-fvls5\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:56.844850 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:56.844810 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a47faf-ed37-4499-abcb-cd5fec47dc13-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:56.844850 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:56.844822 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a47faf-ed37-4499-abcb-cd5fec47dc13-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:56.844850 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:56.844835 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a47faf-ed37-4499-abcb-cd5fec47dc13-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:56.844850 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:56.844847 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a47faf-ed37-4499-abcb-cd5fec47dc13-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:57.000658 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:57.000626 6841 generic.go:356] "Generic (PLEG): container finished" podID="84a47faf-ed37-4499-abcb-cd5fec47dc13" containerID="162fd22a616ae574ea364983c82388208f3b2d79d12c7f32996881784a6bd0a3" exitCode=0 Oct 31 13:06:57.000830 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:57.000672 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" Oct 31 13:06:57.000830 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:57.000710 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" event={"ID":"84a47faf-ed37-4499-abcb-cd5fec47dc13","Type":"ContainerDied","Data":"162fd22a616ae574ea364983c82388208f3b2d79d12c7f32996881784a6bd0a3"} Oct 31 13:06:57.000830 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:57.000776 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf" event={"ID":"84a47faf-ed37-4499-abcb-cd5fec47dc13","Type":"ContainerDied","Data":"2511901b3319532630b21d37c63947ed59b6db7e435a3d87f6c5e7aed4655588"} Oct 31 13:06:57.000830 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:57.000801 6841 scope.go:122] "RemoveContainer" containerID="162fd22a616ae574ea364983c82388208f3b2d79d12c7f32996881784a6bd0a3" Oct 31 13:06:57.011931 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:57.011916 6841 scope.go:122] "RemoveContainer" containerID="162fd22a616ae574ea364983c82388208f3b2d79d12c7f32996881784a6bd0a3" Oct 31 13:06:57.012203 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:06:57.012186 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"162fd22a616ae574ea364983c82388208f3b2d79d12c7f32996881784a6bd0a3\": container with ID starting with 162fd22a616ae574ea364983c82388208f3b2d79d12c7f32996881784a6bd0a3 not found: ID does not exist" containerID="162fd22a616ae574ea364983c82388208f3b2d79d12c7f32996881784a6bd0a3" Oct 31 13:06:57.012246 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:57.012211 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"162fd22a616ae574ea364983c82388208f3b2d79d12c7f32996881784a6bd0a3"} err="failed to get container status \"162fd22a616ae574ea364983c82388208f3b2d79d12c7f32996881784a6bd0a3\": rpc error: code = NotFound desc = could not find container \"162fd22a616ae574ea364983c82388208f3b2d79d12c7f32996881784a6bd0a3\": container with ID starting with 162fd22a616ae574ea364983c82388208f3b2d79d12c7f32996881784a6bd0a3 not found: ID does not exist" Oct 31 13:06:57.025841 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:57.025812 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf"] Oct 31 13:06:57.034169 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:57.034145 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-f4fsf"] Oct 31 13:06:57.231127 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:57.231049 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:57.448969 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:57.448921 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:57.644183 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:57.644104 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84a47faf-ed37-4499-abcb-cd5fec47dc13" path="/var/lib/kubelet/pods/84a47faf-ed37-4499-abcb-cd5fec47dc13/volumes" Oct 31 13:06:58.347880 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:58.347844 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:59.084617 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:59.084580 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:06:59.836862 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:06:59.836824 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:00.636595 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:00.636546 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:01.507874 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:01.507838 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:02.442873 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:02.442831 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:02.697188 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:02.697094 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:03.901993 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:03.901933 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:05.087013 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:05.077639 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg"] Oct 31 13:07:05.747076 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:05.182681 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:05.757007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:05.749354 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg" podUID="9b7bbaef-1b8e-4875-a366-b84f3f19e3f5" containerName="maas-api" containerID="cri-o://ac8e35f4b0e863b0c2983e44483b85b32f814ea4f8c81e2c5108064892d63eab" gracePeriod=30 Oct 31 13:07:05.912560 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:05.912114 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg" podUID="9b7bbaef-1b8e-4875-a366-b84f3f19e3f5" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.134:8443/health\": dial tcp 10.134.0.134:8443: connect: connection refused" Oct 31 13:07:05.976021 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:05.974558 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:05.983026 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:05.977653 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-b64d65f5b-rcxdp"] Oct 31 13:07:05.983026 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:05.977724 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-5d54db4b4c-qxx8v"] Oct 31 13:07:06.127178 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:06.123199 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-5d54db4b4c-qxx8v" podUID="59760ecf-ed86-48bc-92ab-de8527d1fff7" containerName="payload-pre-processing" containerID="cri-o://71ccff3c9184af8f42f8548b7a5dc7a47d26df224fd68e99c0e833032276be26" gracePeriod=30 Oct 31 13:07:06.373017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:06.372847 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-b64d65f5b-rcxdp" podUID="9574b8ac-9faf-47af-a648-8ebf24027496" containerName="payload-processing" containerID="cri-o://b60859163da08a33fe6f623b8431143302aa1f7dcc72ca9053d27bc00218bc77" gracePeriod=30 Oct 31 13:07:06.946008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:06.945179 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:06.957006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:06.954903 6841 generic.go:356] "Generic (PLEG): container finished" podID="59760ecf-ed86-48bc-92ab-de8527d1fff7" containerID="71ccff3c9184af8f42f8548b7a5dc7a47d26df224fd68e99c0e833032276be26" exitCode=0 Oct 31 13:07:06.957006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:06.955138 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-5d54db4b4c-qxx8v" event={"ID":"59760ecf-ed86-48bc-92ab-de8527d1fff7","Type":"ContainerDied","Data":"71ccff3c9184af8f42f8548b7a5dc7a47d26df224fd68e99c0e833032276be26"} Oct 31 13:07:07.143878 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:07:07.142914 6841 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9574b8ac_9faf_47af_a648_8ebf24027496.slice/crio-b60859163da08a33fe6f623b8431143302aa1f7dcc72ca9053d27bc00218bc77.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9574b8ac_9faf_47af_a648_8ebf24027496.slice/crio-conmon-b60859163da08a33fe6f623b8431143302aa1f7dcc72ca9053d27bc00218bc77.scope\": RecentStats: unable to find data in memory cache]" Oct 31 13:07:07.282742 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.282708 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg" Oct 31 13:07:07.325754 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.325725 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-5d54db4b4c-qxx8v" Oct 31 13:07:07.335401 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.335377 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-b64d65f5b-rcxdp" Oct 31 13:07:07.343716 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.343691 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b7bbaef-1b8e-4875-a366-b84f3f19e3f5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b7bbaef-1b8e-4875-a366-b84f3f19e3f5-metrics-certs\") pod \"9b7bbaef-1b8e-4875-a366-b84f3f19e3f5\" (UID: \"9b7bbaef-1b8e-4875-a366-b84f3f19e3f5\") " Oct 31 13:07:07.343856 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.343742 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b7bbaef-1b8e-4875-a366-b84f3f19e3f5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b7bbaef-1b8e-4875-a366-b84f3f19e3f5-maas-api-tls\") pod \"9b7bbaef-1b8e-4875-a366-b84f3f19e3f5\" (UID: \"9b7bbaef-1b8e-4875-a366-b84f3f19e3f5\") " Oct 31 13:07:07.343938 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.343871 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59760ecf-ed86-48bc-92ab-de8527d1fff7-kube-api-access-cl764\" (UniqueName: \"kubernetes.io/projected/59760ecf-ed86-48bc-92ab-de8527d1fff7-kube-api-access-cl764\") pod \"59760ecf-ed86-48bc-92ab-de8527d1fff7\" (UID: \"59760ecf-ed86-48bc-92ab-de8527d1fff7\") " Oct 31 13:07:07.344012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.343998 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59760ecf-ed86-48bc-92ab-de8527d1fff7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59760ecf-ed86-48bc-92ab-de8527d1fff7-plugins-config-volume\") pod \"59760ecf-ed86-48bc-92ab-de8527d1fff7\" (UID: \"59760ecf-ed86-48bc-92ab-de8527d1fff7\") " Oct 31 13:07:07.344084 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.344057 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b7bbaef-1b8e-4875-a366-b84f3f19e3f5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b7bbaef-1b8e-4875-a366-b84f3f19e3f5-openshift-service-ca\") pod \"9b7bbaef-1b8e-4875-a366-b84f3f19e3f5\" (UID: \"9b7bbaef-1b8e-4875-a366-b84f3f19e3f5\") " Oct 31 13:07:07.344186 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.344169 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b7bbaef-1b8e-4875-a366-b84f3f19e3f5-kube-api-access-4dh2v\" (UniqueName: \"kubernetes.io/projected/9b7bbaef-1b8e-4875-a366-b84f3f19e3f5-kube-api-access-4dh2v\") pod \"9b7bbaef-1b8e-4875-a366-b84f3f19e3f5\" (UID: \"9b7bbaef-1b8e-4875-a366-b84f3f19e3f5\") " Oct 31 13:07:07.344814 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.344744 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b7bbaef-1b8e-4875-a366-b84f3f19e3f5-openshift-service-ca" pod "9b7bbaef-1b8e-4875-a366-b84f3f19e3f5" (UID: "9b7bbaef-1b8e-4875-a366-b84f3f19e3f5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:07:07.345635 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.345099 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59760ecf-ed86-48bc-92ab-de8527d1fff7-plugins-config-volume" pod "59760ecf-ed86-48bc-92ab-de8527d1fff7" (UID: "59760ecf-ed86-48bc-92ab-de8527d1fff7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:07:07.351144 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.351118 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b7bbaef-1b8e-4875-a366-b84f3f19e3f5-maas-api-tls" pod "9b7bbaef-1b8e-4875-a366-b84f3f19e3f5" (UID: "9b7bbaef-1b8e-4875-a366-b84f3f19e3f5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:07:07.351144 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.351127 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b7bbaef-1b8e-4875-a366-b84f3f19e3f5-kube-api-access-4dh2v" pod "9b7bbaef-1b8e-4875-a366-b84f3f19e3f5" (UID: "9b7bbaef-1b8e-4875-a366-b84f3f19e3f5"). InnerVolumeSpecName "kube-api-access-4dh2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:07:07.351576 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.351511 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b7bbaef-1b8e-4875-a366-b84f3f19e3f5-metrics-certs" pod "9b7bbaef-1b8e-4875-a366-b84f3f19e3f5" (UID: "9b7bbaef-1b8e-4875-a366-b84f3f19e3f5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:07:07.353480 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.353458 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59760ecf-ed86-48bc-92ab-de8527d1fff7-kube-api-access-cl764" pod "59760ecf-ed86-48bc-92ab-de8527d1fff7" (UID: "59760ecf-ed86-48bc-92ab-de8527d1fff7"). InnerVolumeSpecName "kube-api-access-cl764". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:07:07.446105 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.446077 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9574b8ac-9faf-47af-a648-8ebf24027496-kube-api-access-d8fbq\" (UniqueName: \"kubernetes.io/projected/9574b8ac-9faf-47af-a648-8ebf24027496-kube-api-access-d8fbq\") pod \"9574b8ac-9faf-47af-a648-8ebf24027496\" (UID: \"9574b8ac-9faf-47af-a648-8ebf24027496\") " Oct 31 13:07:07.446236 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.446146 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9574b8ac-9faf-47af-a648-8ebf24027496-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9574b8ac-9faf-47af-a648-8ebf24027496-plugins-config-volume\") pod \"9574b8ac-9faf-47af-a648-8ebf24027496\" (UID: \"9574b8ac-9faf-47af-a648-8ebf24027496\") " Oct 31 13:07:07.446578 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.446555 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9574b8ac-9faf-47af-a648-8ebf24027496-plugins-config-volume" pod "9574b8ac-9faf-47af-a648-8ebf24027496" (UID: "9574b8ac-9faf-47af-a648-8ebf24027496"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:07:07.446703 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.446687 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dh2v\" (UniqueName: \"kubernetes.io/projected/9b7bbaef-1b8e-4875-a366-b84f3f19e3f5-kube-api-access-4dh2v\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:07.446743 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.446707 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b7bbaef-1b8e-4875-a366-b84f3f19e3f5-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:07.446743 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.446716 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b7bbaef-1b8e-4875-a366-b84f3f19e3f5-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:07.446743 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.446724 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9574b8ac-9faf-47af-a648-8ebf24027496-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:07.446743 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.446733 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl764\" (UniqueName: \"kubernetes.io/projected/59760ecf-ed86-48bc-92ab-de8527d1fff7-kube-api-access-cl764\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:07.446743 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.446743 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59760ecf-ed86-48bc-92ab-de8527d1fff7-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:07.446899 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.446752 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b7bbaef-1b8e-4875-a366-b84f3f19e3f5-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:07.448125 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.448103 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9574b8ac-9faf-47af-a648-8ebf24027496-kube-api-access-d8fbq" pod "9574b8ac-9faf-47af-a648-8ebf24027496" (UID: "9574b8ac-9faf-47af-a648-8ebf24027496"). InnerVolumeSpecName "kube-api-access-d8fbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:07:07.537506 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.537472 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:07.547938 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.547907 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8fbq\" (UniqueName: \"kubernetes.io/projected/9574b8ac-9faf-47af-a648-8ebf24027496-kube-api-access-d8fbq\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:07.966384 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.966344 6841 generic.go:356] "Generic (PLEG): container finished" podID="9b7bbaef-1b8e-4875-a366-b84f3f19e3f5" containerID="ac8e35f4b0e863b0c2983e44483b85b32f814ea4f8c81e2c5108064892d63eab" exitCode=0 Oct 31 13:07:07.966622 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.966396 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg" Oct 31 13:07:07.966622 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.966427 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg" event={"ID":"9b7bbaef-1b8e-4875-a366-b84f3f19e3f5","Type":"ContainerDied","Data":"ac8e35f4b0e863b0c2983e44483b85b32f814ea4f8c81e2c5108064892d63eab"} Oct 31 13:07:07.966622 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.966490 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg" event={"ID":"9b7bbaef-1b8e-4875-a366-b84f3f19e3f5","Type":"ContainerDied","Data":"fdbc6732ae019aea4d350fdac61849c4c0e05e17729e50fdab5de16a38de69e7"} Oct 31 13:07:07.966622 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.966523 6841 scope.go:122] "RemoveContainer" containerID="ac8e35f4b0e863b0c2983e44483b85b32f814ea4f8c81e2c5108064892d63eab" Oct 31 13:07:07.971921 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.971896 6841 generic.go:356] "Generic (PLEG): container finished" podID="9574b8ac-9faf-47af-a648-8ebf24027496" containerID="b60859163da08a33fe6f623b8431143302aa1f7dcc72ca9053d27bc00218bc77" exitCode=0 Oct 31 13:07:07.972208 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.971954 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-b64d65f5b-rcxdp" Oct 31 13:07:07.972448 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.971962 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-b64d65f5b-rcxdp" event={"ID":"9574b8ac-9faf-47af-a648-8ebf24027496","Type":"ContainerDied","Data":"b60859163da08a33fe6f623b8431143302aa1f7dcc72ca9053d27bc00218bc77"} Oct 31 13:07:07.972448 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.972382 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-b64d65f5b-rcxdp" event={"ID":"9574b8ac-9faf-47af-a648-8ebf24027496","Type":"ContainerDied","Data":"ae790791b462febe561a396cb636674c131642b592386015688a92e36623f1ec"} Oct 31 13:07:07.975261 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.974720 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-5d54db4b4c-qxx8v" event={"ID":"59760ecf-ed86-48bc-92ab-de8527d1fff7","Type":"ContainerDied","Data":"65af52e935ecb81824670ece334dca2143e36774b463a97d4eff4e2ceccc3594"} Oct 31 13:07:07.975261 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.974805 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-5d54db4b4c-qxx8v" Oct 31 13:07:07.983349 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.983325 6841 scope.go:122] "RemoveContainer" containerID="ac8e35f4b0e863b0c2983e44483b85b32f814ea4f8c81e2c5108064892d63eab" Oct 31 13:07:07.983785 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:07:07.983764 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac8e35f4b0e863b0c2983e44483b85b32f814ea4f8c81e2c5108064892d63eab\": container with ID starting with ac8e35f4b0e863b0c2983e44483b85b32f814ea4f8c81e2c5108064892d63eab not found: ID does not exist" containerID="ac8e35f4b0e863b0c2983e44483b85b32f814ea4f8c81e2c5108064892d63eab" Oct 31 13:07:07.983863 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.983791 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac8e35f4b0e863b0c2983e44483b85b32f814ea4f8c81e2c5108064892d63eab"} err="failed to get container status \"ac8e35f4b0e863b0c2983e44483b85b32f814ea4f8c81e2c5108064892d63eab\": rpc error: code = NotFound desc = could not find container \"ac8e35f4b0e863b0c2983e44483b85b32f814ea4f8c81e2c5108064892d63eab\": container with ID starting with ac8e35f4b0e863b0c2983e44483b85b32f814ea4f8c81e2c5108064892d63eab not found: ID does not exist" Oct 31 13:07:07.983863 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.983807 6841 scope.go:122] "RemoveContainer" containerID="b60859163da08a33fe6f623b8431143302aa1f7dcc72ca9053d27bc00218bc77" Oct 31 13:07:07.994113 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.994087 6841 scope.go:122] "RemoveContainer" containerID="b60859163da08a33fe6f623b8431143302aa1f7dcc72ca9053d27bc00218bc77" Oct 31 13:07:07.998259 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:07:07.998231 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b60859163da08a33fe6f623b8431143302aa1f7dcc72ca9053d27bc00218bc77\": container with ID starting with b60859163da08a33fe6f623b8431143302aa1f7dcc72ca9053d27bc00218bc77 not found: ID does not exist" containerID="b60859163da08a33fe6f623b8431143302aa1f7dcc72ca9053d27bc00218bc77" Oct 31 13:07:07.998353 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.998266 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b60859163da08a33fe6f623b8431143302aa1f7dcc72ca9053d27bc00218bc77"} err="failed to get container status \"b60859163da08a33fe6f623b8431143302aa1f7dcc72ca9053d27bc00218bc77\": rpc error: code = NotFound desc = could not find container \"b60859163da08a33fe6f623b8431143302aa1f7dcc72ca9053d27bc00218bc77\": container with ID starting with b60859163da08a33fe6f623b8431143302aa1f7dcc72ca9053d27bc00218bc77 not found: ID does not exist" Oct 31 13:07:07.998353 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.998291 6841 scope.go:122] "RemoveContainer" containerID="71ccff3c9184af8f42f8548b7a5dc7a47d26df224fd68e99c0e833032276be26" Oct 31 13:07:07.999697 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:07.999681 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg"] Oct 31 13:07:08.005486 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:08.005465 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4cdd9-79d575cbcc-vsfdg"] Oct 31 13:07:08.019138 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:08.019116 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-5d54db4b4c-qxx8v"] Oct 31 13:07:08.022021 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:08.021966 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f4cdd9-5d54db4b4c-qxx8v"] Oct 31 13:07:08.040451 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:08.040424 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-b64d65f5b-rcxdp"] Oct 31 13:07:08.043958 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:08.043934 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f4cdd9-b64d65f5b-rcxdp"] Oct 31 13:07:08.099021 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:08.098904 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:08.449402 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:08.449371 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:09.226802 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:09.226771 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:09.465052 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:09.465016 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:09.643757 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:09.643683 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59760ecf-ed86-48bc-92ab-de8527d1fff7" path="/var/lib/kubelet/pods/59760ecf-ed86-48bc-92ab-de8527d1fff7/volumes" Oct 31 13:07:09.644149 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:09.644136 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9574b8ac-9faf-47af-a648-8ebf24027496" path="/var/lib/kubelet/pods/9574b8ac-9faf-47af-a648-8ebf24027496/volumes" Oct 31 13:07:09.644554 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:09.644541 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b7bbaef-1b8e-4875-a366-b84f3f19e3f5" path="/var/lib/kubelet/pods/9b7bbaef-1b8e-4875-a366-b84f3f19e3f5/volumes" Oct 31 13:07:09.793432 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:09.790579 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:10.023285 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:10.023253 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds"] Oct 31 13:07:10.023579 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:10.023539 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" podUID="e5ae8ae0-da38-4530-856c-c9e70d8484b7" containerName="istio-proxy" containerID="cri-o://d273b24c4de937a2a9366357c5348b01a69d2ac573d0d580a93217e0260c1118" gracePeriod=30 Oct 31 13:07:10.210005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:10.206675 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:10.471676 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:10.471636 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:10.971732 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:10.971680 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:11.215885 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:11.215850 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:12.147670 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:12.147632 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:12.808545 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:12.808186 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:13.046879 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:13.046835 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:13.497365 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:13.497187 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:13.697210 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:13.697181 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:13.752733 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:13.752655 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf"] Oct 31 13:07:13.752900 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:13.752857 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" podUID="916a1b9e-516f-4cde-955a-71a255d469a4" containerName="istio-proxy" containerID="cri-o://70aab0320cc4375536158511d28afcf417caf5ee0d964aa84c322af635cfdbe2" gracePeriod=30 Oct 31 13:07:13.945333 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:13.941863 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:14.538350 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:14.538316 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:15.383460 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:15.383431 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:07:15.400886 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:15.400844 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:15.457999 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:15.457944 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ae8ae0-da38-4530-856c-c9e70d8484b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ae8ae0-da38-4530-856c-c9e70d8484b7-istio-data\") pod \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\" (UID: \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\") " Oct 31 13:07:15.458190 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:15.458141 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ae8ae0-da38-4530-856c-c9e70d8484b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ae8ae0-da38-4530-856c-c9e70d8484b7-credential-socket\") pod \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\" (UID: \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\") " Oct 31 13:07:15.458190 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:15.458177 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ae8ae0-da38-4530-856c-c9e70d8484b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ae8ae0-da38-4530-856c-c9e70d8484b7-workload-socket\") pod \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\" (UID: \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\") " Oct 31 13:07:15.458359 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:15.458215 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5ae8ae0-da38-4530-856c-c9e70d8484b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ae8ae0-da38-4530-856c-c9e70d8484b7-istiod-ca-cert\") pod \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\" (UID: \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\") " Oct 31 13:07:15.458359 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:15.458214 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ae8ae0-da38-4530-856c-c9e70d8484b7-istio-data" pod "e5ae8ae0-da38-4530-856c-c9e70d8484b7" (UID: "e5ae8ae0-da38-4530-856c-c9e70d8484b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:15.458359 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:15.458320 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ae8ae0-da38-4530-856c-c9e70d8484b7-istio-token\" (UniqueName: \"kubernetes.io/projected/e5ae8ae0-da38-4530-856c-c9e70d8484b7-istio-token\") pod \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\" (UID: \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\") " Oct 31 13:07:15.458538 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:15.458396 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ae8ae0-da38-4530-856c-c9e70d8484b7-workload-socket" pod "e5ae8ae0-da38-4530-856c-c9e70d8484b7" (UID: "e5ae8ae0-da38-4530-856c-c9e70d8484b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:15.458538 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:15.458461 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ae8ae0-da38-4530-856c-c9e70d8484b7-kube-api-access-bmvnz\" (UniqueName: \"kubernetes.io/projected/e5ae8ae0-da38-4530-856c-c9e70d8484b7-kube-api-access-bmvnz\") pod \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\" (UID: \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\") " Oct 31 13:07:15.458538 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:15.458518 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5ae8ae0-da38-4530-856c-c9e70d8484b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ae8ae0-da38-4530-856c-c9e70d8484b7-istio-podinfo\") pod \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\" (UID: \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\") " Oct 31 13:07:15.458699 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:15.458560 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ae8ae0-da38-4530-856c-c9e70d8484b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ae8ae0-da38-4530-856c-c9e70d8484b7-istio-envoy\") pod \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\" (UID: \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\") " Oct 31 13:07:15.458699 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:15.458587 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ae8ae0-da38-4530-856c-c9e70d8484b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ae8ae0-da38-4530-856c-c9e70d8484b7-workload-certs\") pod \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\" (UID: \"e5ae8ae0-da38-4530-856c-c9e70d8484b7\") " Oct 31 13:07:15.458699 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:15.458637 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5ae8ae0-da38-4530-856c-c9e70d8484b7-istiod-ca-cert" pod "e5ae8ae0-da38-4530-856c-c9e70d8484b7" (UID: "e5ae8ae0-da38-4530-856c-c9e70d8484b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:07:15.459113 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:15.459086 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ae8ae0-da38-4530-856c-c9e70d8484b7-workload-certs" pod "e5ae8ae0-da38-4530-856c-c9e70d8484b7" (UID: "e5ae8ae0-da38-4530-856c-c9e70d8484b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:15.459234 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:15.459219 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ae8ae0-da38-4530-856c-c9e70d8484b7-credential-socket" pod "e5ae8ae0-da38-4530-856c-c9e70d8484b7" (UID: "e5ae8ae0-da38-4530-856c-c9e70d8484b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:15.459303 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:15.459260 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ae8ae0-da38-4530-856c-c9e70d8484b7-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:15.459303 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:15.459286 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ae8ae0-da38-4530-856c-c9e70d8484b7-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:15.459412 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:15.459300 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ae8ae0-da38-4530-856c-c9e70d8484b7-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:15.459412 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:15.459316 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ae8ae0-da38-4530-856c-c9e70d8484b7-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:15.461384 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:15.461355 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ae8ae0-da38-4530-856c-c9e70d8484b7-istio-envoy" pod "e5ae8ae0-da38-4530-856c-c9e70d8484b7" (UID: "e5ae8ae0-da38-4530-856c-c9e70d8484b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:15.461490 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:15.461429 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ae8ae0-da38-4530-856c-c9e70d8484b7-kube-api-access-bmvnz" pod "e5ae8ae0-da38-4530-856c-c9e70d8484b7" (UID: "e5ae8ae0-da38-4530-856c-c9e70d8484b7"). InnerVolumeSpecName "kube-api-access-bmvnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:07:15.461650 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:15.461623 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ae8ae0-da38-4530-856c-c9e70d8484b7-istio-token" pod "e5ae8ae0-da38-4530-856c-c9e70d8484b7" (UID: "e5ae8ae0-da38-4530-856c-c9e70d8484b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:07:15.461953 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:15.461928 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5ae8ae0-da38-4530-856c-c9e70d8484b7-istio-podinfo" pod "e5ae8ae0-da38-4530-856c-c9e70d8484b7" (UID: "e5ae8ae0-da38-4530-856c-c9e70d8484b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:07:15.560958 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:15.560920 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ae8ae0-da38-4530-856c-c9e70d8484b7-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:15.560958 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:15.560951 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ae8ae0-da38-4530-856c-c9e70d8484b7-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:15.561402 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:15.560962 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmvnz\" (UniqueName: \"kubernetes.io/projected/e5ae8ae0-da38-4530-856c-c9e70d8484b7-kube-api-access-bmvnz\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:15.561402 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:15.561013 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ae8ae0-da38-4530-856c-c9e70d8484b7-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:15.561402 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:15.561022 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ae8ae0-da38-4530-856c-c9e70d8484b7-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:16.047792 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:16.047755 6841 generic.go:356] "Generic (PLEG): container finished" podID="e5ae8ae0-da38-4530-856c-c9e70d8484b7" containerID="d273b24c4de937a2a9366357c5348b01a69d2ac573d0d580a93217e0260c1118" exitCode=0 Oct 31 13:07:16.047973 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:16.047822 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" Oct 31 13:07:16.047973 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:16.047848 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" event={"ID":"e5ae8ae0-da38-4530-856c-c9e70d8484b7","Type":"ContainerDied","Data":"d273b24c4de937a2a9366357c5348b01a69d2ac573d0d580a93217e0260c1118"} Oct 31 13:07:16.047973 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:16.047905 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds" event={"ID":"e5ae8ae0-da38-4530-856c-c9e70d8484b7","Type":"ContainerDied","Data":"5604d38fb7f3fd427e42f4c2ffc005f7b73fe709e12af1b88f0438fa39b78c29"} Oct 31 13:07:16.047973 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:16.047928 6841 scope.go:122] "RemoveContainer" containerID="d273b24c4de937a2a9366357c5348b01a69d2ac573d0d580a93217e0260c1118" Oct 31 13:07:16.058351 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:16.058330 6841 scope.go:122] "RemoveContainer" containerID="26a00c479d203d15b2e834160249c8440bc0518335332f369fc2b2b03393cfd7" Oct 31 13:07:16.067236 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:16.067220 6841 scope.go:122] "RemoveContainer" containerID="d273b24c4de937a2a9366357c5348b01a69d2ac573d0d580a93217e0260c1118" Oct 31 13:07:16.067505 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:07:16.067487 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d273b24c4de937a2a9366357c5348b01a69d2ac573d0d580a93217e0260c1118\": container with ID starting with d273b24c4de937a2a9366357c5348b01a69d2ac573d0d580a93217e0260c1118 not found: ID does not exist" containerID="d273b24c4de937a2a9366357c5348b01a69d2ac573d0d580a93217e0260c1118" Oct 31 13:07:16.067559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:16.067513 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d273b24c4de937a2a9366357c5348b01a69d2ac573d0d580a93217e0260c1118"} err="failed to get container status \"d273b24c4de937a2a9366357c5348b01a69d2ac573d0d580a93217e0260c1118\": rpc error: code = NotFound desc = could not find container \"d273b24c4de937a2a9366357c5348b01a69d2ac573d0d580a93217e0260c1118\": container with ID starting with d273b24c4de937a2a9366357c5348b01a69d2ac573d0d580a93217e0260c1118 not found: ID does not exist" Oct 31 13:07:16.067559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:16.067527 6841 scope.go:122] "RemoveContainer" containerID="26a00c479d203d15b2e834160249c8440bc0518335332f369fc2b2b03393cfd7" Oct 31 13:07:16.067738 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:07:16.067721 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26a00c479d203d15b2e834160249c8440bc0518335332f369fc2b2b03393cfd7\": container with ID starting with 26a00c479d203d15b2e834160249c8440bc0518335332f369fc2b2b03393cfd7 not found: ID does not exist" containerID="26a00c479d203d15b2e834160249c8440bc0518335332f369fc2b2b03393cfd7" Oct 31 13:07:16.067779 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:16.067744 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26a00c479d203d15b2e834160249c8440bc0518335332f369fc2b2b03393cfd7"} err="failed to get container status \"26a00c479d203d15b2e834160249c8440bc0518335332f369fc2b2b03393cfd7\": rpc error: code = NotFound desc = could not find container \"26a00c479d203d15b2e834160249c8440bc0518335332f369fc2b2b03393cfd7\": container with ID starting with 26a00c479d203d15b2e834160249c8440bc0518335332f369fc2b2b03393cfd7 not found: ID does not exist" Oct 31 13:07:16.079317 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:16.079294 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds"] Oct 31 13:07:16.085518 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:16.085496 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-9crds"] Oct 31 13:07:16.451607 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:16.451575 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:17.460922 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:17.460893 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:17.644522 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:17.644491 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5ae8ae0-da38-4530-856c-c9e70d8484b7" path="/var/lib/kubelet/pods/e5ae8ae0-da38-4530-856c-c9e70d8484b7/volumes" Oct 31 13:07:17.995683 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:17.995648 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x"] Oct 31 13:07:17.995909 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:17.995869 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x" podUID="8670d06e-4d46-4e8f-b4f3-532608114855" containerName="maas-api" containerID="cri-o://f1fb0cea8a622a3a3e8502f4ff53d0b82c874ab7fe3d5b6ed798c5c5d776dec7" gracePeriod=30 Oct 31 13:07:18.071478 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:18.071445 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:18.315904 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:18.315662 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:18.438623 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:18.438600 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x" Oct 31 13:07:18.537540 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:18.537510 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8670d06e-4d46-4e8f-b4f3-532608114855-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8670d06e-4d46-4e8f-b4f3-532608114855-openshift-service-ca\") pod \"8670d06e-4d46-4e8f-b4f3-532608114855\" (UID: \"8670d06e-4d46-4e8f-b4f3-532608114855\") " Oct 31 13:07:18.538040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:18.537558 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8670d06e-4d46-4e8f-b4f3-532608114855-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8670d06e-4d46-4e8f-b4f3-532608114855-metrics-certs\") pod \"8670d06e-4d46-4e8f-b4f3-532608114855\" (UID: \"8670d06e-4d46-4e8f-b4f3-532608114855\") " Oct 31 13:07:18.538040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:18.537592 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8670d06e-4d46-4e8f-b4f3-532608114855-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8670d06e-4d46-4e8f-b4f3-532608114855-maas-api-tls\") pod \"8670d06e-4d46-4e8f-b4f3-532608114855\" (UID: \"8670d06e-4d46-4e8f-b4f3-532608114855\") " Oct 31 13:07:18.538040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:18.537776 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8670d06e-4d46-4e8f-b4f3-532608114855-kube-api-access-hzj77\" (UniqueName: \"kubernetes.io/projected/8670d06e-4d46-4e8f-b4f3-532608114855-kube-api-access-hzj77\") pod \"8670d06e-4d46-4e8f-b4f3-532608114855\" (UID: \"8670d06e-4d46-4e8f-b4f3-532608114855\") " Oct 31 13:07:18.538040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:18.537917 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8670d06e-4d46-4e8f-b4f3-532608114855-openshift-service-ca" pod "8670d06e-4d46-4e8f-b4f3-532608114855" (UID: "8670d06e-4d46-4e8f-b4f3-532608114855"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:07:18.538556 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:18.538531 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8670d06e-4d46-4e8f-b4f3-532608114855-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:18.540024 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:18.539968 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8670d06e-4d46-4e8f-b4f3-532608114855-maas-api-tls" pod "8670d06e-4d46-4e8f-b4f3-532608114855" (UID: "8670d06e-4d46-4e8f-b4f3-532608114855"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:07:18.540377 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:18.540034 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8670d06e-4d46-4e8f-b4f3-532608114855-metrics-certs" pod "8670d06e-4d46-4e8f-b4f3-532608114855" (UID: "8670d06e-4d46-4e8f-b4f3-532608114855"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:07:18.540377 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:18.540110 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8670d06e-4d46-4e8f-b4f3-532608114855-kube-api-access-hzj77" pod "8670d06e-4d46-4e8f-b4f3-532608114855" (UID: "8670d06e-4d46-4e8f-b4f3-532608114855"). InnerVolumeSpecName "kube-api-access-hzj77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:07:18.619936 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:18.619863 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:18.643559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:18.642321 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzj77\" (UniqueName: \"kubernetes.io/projected/8670d06e-4d46-4e8f-b4f3-532608114855-kube-api-access-hzj77\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:18.643559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:18.642359 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8670d06e-4d46-4e8f-b4f3-532608114855-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:18.643559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:18.642386 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8670d06e-4d46-4e8f-b4f3-532608114855-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:19.039613 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.039582 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:07:19.048241 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.048218 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/916a1b9e-516f-4cde-955a-71a255d469a4-kube-api-access-mjvg7\" (UniqueName: \"kubernetes.io/projected/916a1b9e-516f-4cde-955a-71a255d469a4-kube-api-access-mjvg7\") pod \"916a1b9e-516f-4cde-955a-71a255d469a4\" (UID: \"916a1b9e-516f-4cde-955a-71a255d469a4\") " Oct 31 13:07:19.048358 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.048291 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/916a1b9e-516f-4cde-955a-71a255d469a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/916a1b9e-516f-4cde-955a-71a255d469a4-credential-socket\") pod \"916a1b9e-516f-4cde-955a-71a255d469a4\" (UID: \"916a1b9e-516f-4cde-955a-71a255d469a4\") " Oct 31 13:07:19.048358 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.048315 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/916a1b9e-516f-4cde-955a-71a255d469a4-istio-token\" (UniqueName: \"kubernetes.io/projected/916a1b9e-516f-4cde-955a-71a255d469a4-istio-token\") pod \"916a1b9e-516f-4cde-955a-71a255d469a4\" (UID: \"916a1b9e-516f-4cde-955a-71a255d469a4\") " Oct 31 13:07:19.048358 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.048351 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/916a1b9e-516f-4cde-955a-71a255d469a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/916a1b9e-516f-4cde-955a-71a255d469a4-istio-data\") pod \"916a1b9e-516f-4cde-955a-71a255d469a4\" (UID: \"916a1b9e-516f-4cde-955a-71a255d469a4\") " Oct 31 13:07:19.048538 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.048414 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/916a1b9e-516f-4cde-955a-71a255d469a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/916a1b9e-516f-4cde-955a-71a255d469a4-istiod-ca-cert\") pod \"916a1b9e-516f-4cde-955a-71a255d469a4\" (UID: \"916a1b9e-516f-4cde-955a-71a255d469a4\") " Oct 31 13:07:19.048538 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.048462 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/916a1b9e-516f-4cde-955a-71a255d469a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/916a1b9e-516f-4cde-955a-71a255d469a4-istio-podinfo\") pod \"916a1b9e-516f-4cde-955a-71a255d469a4\" (UID: \"916a1b9e-516f-4cde-955a-71a255d469a4\") " Oct 31 13:07:19.048538 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.048504 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/916a1b9e-516f-4cde-955a-71a255d469a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/916a1b9e-516f-4cde-955a-71a255d469a4-workload-certs\") pod \"916a1b9e-516f-4cde-955a-71a255d469a4\" (UID: \"916a1b9e-516f-4cde-955a-71a255d469a4\") " Oct 31 13:07:19.048538 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.048536 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/916a1b9e-516f-4cde-955a-71a255d469a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/916a1b9e-516f-4cde-955a-71a255d469a4-istio-envoy\") pod \"916a1b9e-516f-4cde-955a-71a255d469a4\" (UID: \"916a1b9e-516f-4cde-955a-71a255d469a4\") " Oct 31 13:07:19.048744 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.048616 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/916a1b9e-516f-4cde-955a-71a255d469a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/916a1b9e-516f-4cde-955a-71a255d469a4-workload-socket\") pod \"916a1b9e-516f-4cde-955a-71a255d469a4\" (UID: \"916a1b9e-516f-4cde-955a-71a255d469a4\") " Oct 31 13:07:19.048744 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.048685 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916a1b9e-516f-4cde-955a-71a255d469a4-credential-socket" pod "916a1b9e-516f-4cde-955a-71a255d469a4" (UID: "916a1b9e-516f-4cde-955a-71a255d469a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:19.048744 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.048703 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916a1b9e-516f-4cde-955a-71a255d469a4-istio-data" pod "916a1b9e-516f-4cde-955a-71a255d469a4" (UID: "916a1b9e-516f-4cde-955a-71a255d469a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:19.048912 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.048852 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916a1b9e-516f-4cde-955a-71a255d469a4-workload-certs" pod "916a1b9e-516f-4cde-955a-71a255d469a4" (UID: "916a1b9e-516f-4cde-955a-71a255d469a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:19.049039 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.049016 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916a1b9e-516f-4cde-955a-71a255d469a4-workload-socket" pod "916a1b9e-516f-4cde-955a-71a255d469a4" (UID: "916a1b9e-516f-4cde-955a-71a255d469a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:19.049163 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.049139 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/916a1b9e-516f-4cde-955a-71a255d469a4-istiod-ca-cert" pod "916a1b9e-516f-4cde-955a-71a255d469a4" (UID: "916a1b9e-516f-4cde-955a-71a255d469a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:07:19.049349 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.049290 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/916a1b9e-516f-4cde-955a-71a255d469a4-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:19.049511 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.049354 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/916a1b9e-516f-4cde-955a-71a255d469a4-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:19.049511 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.049370 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/916a1b9e-516f-4cde-955a-71a255d469a4-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:19.049511 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.049384 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/916a1b9e-516f-4cde-955a-71a255d469a4-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:19.049511 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.049397 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/916a1b9e-516f-4cde-955a-71a255d469a4-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:19.050937 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.050912 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/916a1b9e-516f-4cde-955a-71a255d469a4-istio-podinfo" pod "916a1b9e-516f-4cde-955a-71a255d469a4" (UID: "916a1b9e-516f-4cde-955a-71a255d469a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:07:19.051060 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.050965 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/916a1b9e-516f-4cde-955a-71a255d469a4-kube-api-access-mjvg7" pod "916a1b9e-516f-4cde-955a-71a255d469a4" (UID: "916a1b9e-516f-4cde-955a-71a255d469a4"). InnerVolumeSpecName "kube-api-access-mjvg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:07:19.051195 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.051169 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/916a1b9e-516f-4cde-955a-71a255d469a4-istio-token" pod "916a1b9e-516f-4cde-955a-71a255d469a4" (UID: "916a1b9e-516f-4cde-955a-71a255d469a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:07:19.051695 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.051675 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916a1b9e-516f-4cde-955a-71a255d469a4-istio-envoy" pod "916a1b9e-516f-4cde-955a-71a255d469a4" (UID: "916a1b9e-516f-4cde-955a-71a255d469a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:19.075464 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.075436 6841 generic.go:356] "Generic (PLEG): container finished" podID="8670d06e-4d46-4e8f-b4f3-532608114855" containerID="f1fb0cea8a622a3a3e8502f4ff53d0b82c874ab7fe3d5b6ed798c5c5d776dec7" exitCode=0 Oct 31 13:07:19.075612 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.075484 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x" Oct 31 13:07:19.075612 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.075505 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x" event={"ID":"8670d06e-4d46-4e8f-b4f3-532608114855","Type":"ContainerDied","Data":"f1fb0cea8a622a3a3e8502f4ff53d0b82c874ab7fe3d5b6ed798c5c5d776dec7"} Oct 31 13:07:19.075612 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.075554 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x" event={"ID":"8670d06e-4d46-4e8f-b4f3-532608114855","Type":"ContainerDied","Data":"5fd0f2fce74b13bb59f264b984d3d5155e448d94d107cba817846c464765c25e"} Oct 31 13:07:19.075612 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.075582 6841 scope.go:122] "RemoveContainer" containerID="f1fb0cea8a622a3a3e8502f4ff53d0b82c874ab7fe3d5b6ed798c5c5d776dec7" Oct 31 13:07:19.078397 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.078372 6841 generic.go:356] "Generic (PLEG): container finished" podID="916a1b9e-516f-4cde-955a-71a255d469a4" containerID="70aab0320cc4375536158511d28afcf417caf5ee0d964aa84c322af635cfdbe2" exitCode=0 Oct 31 13:07:19.078508 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.078472 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" event={"ID":"916a1b9e-516f-4cde-955a-71a255d469a4","Type":"ContainerDied","Data":"70aab0320cc4375536158511d28afcf417caf5ee0d964aa84c322af635cfdbe2"} Oct 31 13:07:19.078563 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.078518 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" event={"ID":"916a1b9e-516f-4cde-955a-71a255d469a4","Type":"ContainerDied","Data":"81c53440a0c1d8f3e78cb835fa3b9495b78249244a87fa3da0129b888ee044e9"} Oct 31 13:07:19.078624 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.078597 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf" Oct 31 13:07:19.101944 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.091282 6841 scope.go:122] "RemoveContainer" containerID="f1fb0cea8a622a3a3e8502f4ff53d0b82c874ab7fe3d5b6ed798c5c5d776dec7" Oct 31 13:07:19.101944 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:07:19.091580 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1fb0cea8a622a3a3e8502f4ff53d0b82c874ab7fe3d5b6ed798c5c5d776dec7\": container with ID starting with f1fb0cea8a622a3a3e8502f4ff53d0b82c874ab7fe3d5b6ed798c5c5d776dec7 not found: ID does not exist" containerID="f1fb0cea8a622a3a3e8502f4ff53d0b82c874ab7fe3d5b6ed798c5c5d776dec7" Oct 31 13:07:19.101944 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.091613 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1fb0cea8a622a3a3e8502f4ff53d0b82c874ab7fe3d5b6ed798c5c5d776dec7"} err="failed to get container status \"f1fb0cea8a622a3a3e8502f4ff53d0b82c874ab7fe3d5b6ed798c5c5d776dec7\": rpc error: code = NotFound desc = could not find container \"f1fb0cea8a622a3a3e8502f4ff53d0b82c874ab7fe3d5b6ed798c5c5d776dec7\": container with ID starting with f1fb0cea8a622a3a3e8502f4ff53d0b82c874ab7fe3d5b6ed798c5c5d776dec7 not found: ID does not exist" Oct 31 13:07:19.101944 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.091637 6841 scope.go:122] "RemoveContainer" containerID="70aab0320cc4375536158511d28afcf417caf5ee0d964aa84c322af635cfdbe2" Oct 31 13:07:19.106051 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.106031 6841 scope.go:122] "RemoveContainer" containerID="054b0b9b69e2378d9f4cb1886539b7e4433b732d211813368c9e24f47fb8874d" Oct 31 13:07:19.106397 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.106381 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x"] Oct 31 13:07:19.111627 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.111605 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e4420f-6555765cfb-m5b5x"] Oct 31 13:07:19.123118 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.123070 6841 scope.go:122] "RemoveContainer" containerID="70aab0320cc4375536158511d28afcf417caf5ee0d964aa84c322af635cfdbe2" Oct 31 13:07:19.123596 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:07:19.123569 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70aab0320cc4375536158511d28afcf417caf5ee0d964aa84c322af635cfdbe2\": container with ID starting with 70aab0320cc4375536158511d28afcf417caf5ee0d964aa84c322af635cfdbe2 not found: ID does not exist" containerID="70aab0320cc4375536158511d28afcf417caf5ee0d964aa84c322af635cfdbe2" Oct 31 13:07:19.123697 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.123622 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70aab0320cc4375536158511d28afcf417caf5ee0d964aa84c322af635cfdbe2"} err="failed to get container status \"70aab0320cc4375536158511d28afcf417caf5ee0d964aa84c322af635cfdbe2\": rpc error: code = NotFound desc = could not find container \"70aab0320cc4375536158511d28afcf417caf5ee0d964aa84c322af635cfdbe2\": container with ID starting with 70aab0320cc4375536158511d28afcf417caf5ee0d964aa84c322af635cfdbe2 not found: ID does not exist" Oct 31 13:07:19.123697 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.123650 6841 scope.go:122] "RemoveContainer" containerID="054b0b9b69e2378d9f4cb1886539b7e4433b732d211813368c9e24f47fb8874d" Oct 31 13:07:19.124473 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:07:19.124443 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"054b0b9b69e2378d9f4cb1886539b7e4433b732d211813368c9e24f47fb8874d\": container with ID starting with 054b0b9b69e2378d9f4cb1886539b7e4433b732d211813368c9e24f47fb8874d not found: ID does not exist" containerID="054b0b9b69e2378d9f4cb1886539b7e4433b732d211813368c9e24f47fb8874d" Oct 31 13:07:19.124563 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.124482 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"054b0b9b69e2378d9f4cb1886539b7e4433b732d211813368c9e24f47fb8874d"} err="failed to get container status \"054b0b9b69e2378d9f4cb1886539b7e4433b732d211813368c9e24f47fb8874d\": rpc error: code = NotFound desc = could not find container \"054b0b9b69e2378d9f4cb1886539b7e4433b732d211813368c9e24f47fb8874d\": container with ID starting with 054b0b9b69e2378d9f4cb1886539b7e4433b732d211813368c9e24f47fb8874d not found: ID does not exist" Oct 31 13:07:19.126058 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.126037 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf"] Oct 31 13:07:19.135887 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.135853 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f4cdd9-openshift-default-6dd6b4bcb9-rmdqf"] Oct 31 13:07:19.150708 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.150681 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjvg7\" (UniqueName: \"kubernetes.io/projected/916a1b9e-516f-4cde-955a-71a255d469a4-kube-api-access-mjvg7\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:19.150708 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.150709 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/916a1b9e-516f-4cde-955a-71a255d469a4-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:19.150850 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.150719 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/916a1b9e-516f-4cde-955a-71a255d469a4-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:19.150850 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.150729 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/916a1b9e-516f-4cde-955a-71a255d469a4-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:19.440532 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.440446 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:19.642944 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.642908 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8670d06e-4d46-4e8f-b4f3-532608114855" path="/var/lib/kubelet/pods/8670d06e-4d46-4e8f-b4f3-532608114855/volumes" Oct 31 13:07:19.643486 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.643470 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="916a1b9e-516f-4cde-955a-71a255d469a4" path="/var/lib/kubelet/pods/916a1b9e-516f-4cde-955a-71a255d469a4/volumes" Oct 31 13:07:19.657308 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:19.657271 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:20.034702 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:07:20.034667 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:07:20.367059 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:20.366479 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:21.234112 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:21.231666 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:22.015307 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:22.015266 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:22.924339 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:22.924306 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:25.118007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.116702 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:25.378179 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.378082 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:25.545008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.544920 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg"] Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.545906 6841 cpu_manager.go:414] "containerMap: removing container" podUID="59760ecf-ed86-48bc-92ab-de8527d1fff7" containerName="payload-pre-processing" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.545933 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59760ecf-ed86-48bc-92ab-de8527d1fff7" containerName="payload-pre-processing" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.545960 6841 cpu_manager.go:414] "containerMap: removing container" podUID="8670d06e-4d46-4e8f-b4f3-532608114855" containerName="maas-api" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.545969 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8670d06e-4d46-4e8f-b4f3-532608114855" containerName="maas-api" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546012 6841 cpu_manager.go:414] "containerMap: removing container" podUID="e5ae8ae0-da38-4530-856c-c9e70d8484b7" containerName="istio-proxy" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546022 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5ae8ae0-da38-4530-856c-c9e70d8484b7" containerName="istio-proxy" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546163 6841 cpu_manager.go:414] "containerMap: removing container" podUID="e5ae8ae0-da38-4530-856c-c9e70d8484b7" containerName="istio-proxy" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546171 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5ae8ae0-da38-4530-856c-c9e70d8484b7" containerName="istio-proxy" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546194 6841 cpu_manager.go:414] "containerMap: removing container" podUID="e024b9a0-c5a3-4850-914a-fe60e0c6f50e" containerName="payload-processing" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546201 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e024b9a0-c5a3-4850-914a-fe60e0c6f50e" containerName="payload-processing" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546222 6841 cpu_manager.go:414] "containerMap: removing container" podUID="c548ef73-9218-4068-bcd2-101717e5295f" containerName="maas-api" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546230 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c548ef73-9218-4068-bcd2-101717e5295f" containerName="maas-api" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546245 6841 cpu_manager.go:414] "containerMap: removing container" podUID="5ebddec1-8fab-4709-84de-f5d99a9112d0" containerName="istio-proxy" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546254 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ebddec1-8fab-4709-84de-f5d99a9112d0" containerName="istio-proxy" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546264 6841 cpu_manager.go:414] "containerMap: removing container" podUID="84a47faf-ed37-4499-abcb-cd5fec47dc13" containerName="istio-proxy" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546272 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a47faf-ed37-4499-abcb-cd5fec47dc13" containerName="istio-proxy" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546299 6841 cpu_manager.go:414] "containerMap: removing container" podUID="688d5b2f-0756-480e-83f2-66e05c48527b" containerName="main" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546307 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="688d5b2f-0756-480e-83f2-66e05c48527b" containerName="main" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546320 6841 cpu_manager.go:414] "containerMap: removing container" podUID="d03bab9c-ade2-43da-8bb1-37fc53fd93f9" containerName="payload-pre-processing" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546329 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d03bab9c-ade2-43da-8bb1-37fc53fd93f9" containerName="payload-pre-processing" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546349 6841 cpu_manager.go:414] "containerMap: removing container" podUID="9b7bbaef-1b8e-4875-a366-b84f3f19e3f5" containerName="maas-api" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546357 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b7bbaef-1b8e-4875-a366-b84f3f19e3f5" containerName="maas-api" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546371 6841 cpu_manager.go:414] "containerMap: removing container" podUID="916a1b9e-516f-4cde-955a-71a255d469a4" containerName="istio-proxy" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546379 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="916a1b9e-516f-4cde-955a-71a255d469a4" containerName="istio-proxy" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546388 6841 cpu_manager.go:414] "containerMap: removing container" podUID="9574b8ac-9faf-47af-a648-8ebf24027496" containerName="payload-processing" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546396 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9574b8ac-9faf-47af-a648-8ebf24027496" containerName="payload-processing" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546407 6841 cpu_manager.go:414] "containerMap: removing container" podUID="916a1b9e-516f-4cde-955a-71a255d469a4" containerName="istio-proxy" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546414 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="916a1b9e-516f-4cde-955a-71a255d469a4" containerName="istio-proxy" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546446 6841 cpu_manager.go:414] "containerMap: removing container" podUID="5ebddec1-8fab-4709-84de-f5d99a9112d0" containerName="istio-proxy" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546455 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ebddec1-8fab-4709-84de-f5d99a9112d0" containerName="istio-proxy" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546468 6841 cpu_manager.go:414] "containerMap: removing container" podUID="129ee68e-30b6-4e0c-97e8-85f780b2cab6" containerName="maas-api" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546475 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="129ee68e-30b6-4e0c-97e8-85f780b2cab6" containerName="maas-api" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546620 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b7bbaef-1b8e-4875-a366-b84f3f19e3f5" containerName="maas-api" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546633 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="916a1b9e-516f-4cde-955a-71a255d469a4" containerName="istio-proxy" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546643 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="9574b8ac-9faf-47af-a648-8ebf24027496" containerName="payload-processing" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546655 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="916a1b9e-516f-4cde-955a-71a255d469a4" containerName="istio-proxy" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546669 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ebddec1-8fab-4709-84de-f5d99a9112d0" containerName="istio-proxy" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546679 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="129ee68e-30b6-4e0c-97e8-85f780b2cab6" containerName="maas-api" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546687 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="59760ecf-ed86-48bc-92ab-de8527d1fff7" containerName="payload-pre-processing" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546698 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="8670d06e-4d46-4e8f-b4f3-532608114855" containerName="maas-api" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546709 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5ae8ae0-da38-4530-856c-c9e70d8484b7" containerName="istio-proxy" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546719 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5ae8ae0-da38-4530-856c-c9e70d8484b7" containerName="istio-proxy" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546728 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="e024b9a0-c5a3-4850-914a-fe60e0c6f50e" containerName="payload-processing" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546741 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="c548ef73-9218-4068-bcd2-101717e5295f" containerName="maas-api" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546750 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a47faf-ed37-4499-abcb-cd5fec47dc13" containerName="istio-proxy" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546760 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="688d5b2f-0756-480e-83f2-66e05c48527b" containerName="main" Oct 31 13:07:25.549006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.546768 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="d03bab9c-ade2-43da-8bb1-37fc53fd93f9" containerName="payload-pre-processing" Oct 31 13:07:25.559763 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.559735 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:07:25.596145 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.594594 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg"] Oct 31 13:07:25.598095 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.598064 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-7cdec6-openshift-default-dockercfg-85mvx\"" Oct 31 13:07:25.658896 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.658809 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:25.744041 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.721871 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe9fa73-5f1d-47ea-b303-869b3392ac75-workload-socket\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg\" (UID: \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:07:25.744041 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.721999 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fe9fa73-5f1d-47ea-b303-869b3392ac75-istio-envoy\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg\" (UID: \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:07:25.787029 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.744400 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fe9fa73-5f1d-47ea-b303-869b3392ac75-istiod-ca-cert\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg\" (UID: \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:07:25.787029 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.744491 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fe9fa73-5f1d-47ea-b303-869b3392ac75-istio-token\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg\" (UID: \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:07:25.787029 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.744539 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fe9fa73-5f1d-47ea-b303-869b3392ac75-workload-certs\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg\" (UID: \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:07:25.787029 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.744599 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnc27\" (UniqueName: \"kubernetes.io/projected/0fe9fa73-5f1d-47ea-b303-869b3392ac75-kube-api-access-nnc27\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg\" (UID: \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:07:25.787029 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.744648 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fe9fa73-5f1d-47ea-b303-869b3392ac75-istio-data\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg\" (UID: \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:07:25.787029 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.744704 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe9fa73-5f1d-47ea-b303-869b3392ac75-credential-socket\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg\" (UID: \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:07:25.787029 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.744812 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fe9fa73-5f1d-47ea-b303-869b3392ac75-istio-podinfo\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg\" (UID: \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:07:25.858323 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.846203 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fe9fa73-5f1d-47ea-b303-869b3392ac75-istio-podinfo\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg\" (UID: \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:07:25.875606 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.875571 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe9fa73-5f1d-47ea-b303-869b3392ac75-workload-socket\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg\" (UID: \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:07:25.876578 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.876556 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fe9fa73-5f1d-47ea-b303-869b3392ac75-istio-envoy\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg\" (UID: \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:07:25.877224 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.877203 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fe9fa73-5f1d-47ea-b303-869b3392ac75-istiod-ca-cert\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg\" (UID: \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:07:25.877384 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.877369 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fe9fa73-5f1d-47ea-b303-869b3392ac75-istio-token\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg\" (UID: \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:07:25.877508 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.877495 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fe9fa73-5f1d-47ea-b303-869b3392ac75-workload-certs\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg\" (UID: \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:07:25.877645 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.877630 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnc27\" (UniqueName: \"kubernetes.io/projected/0fe9fa73-5f1d-47ea-b303-869b3392ac75-kube-api-access-nnc27\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg\" (UID: \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:07:25.877767 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.877753 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fe9fa73-5f1d-47ea-b303-869b3392ac75-istio-data\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg\" (UID: \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:07:25.882043 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.882017 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe9fa73-5f1d-47ea-b303-869b3392ac75-credential-socket\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg\" (UID: \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:07:25.889205 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.879611 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fe9fa73-5f1d-47ea-b303-869b3392ac75-workload-certs\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg\" (UID: \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:07:25.896656 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.880389 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fe9fa73-5f1d-47ea-b303-869b3392ac75-istio-data\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg\" (UID: \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:07:25.932062 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.876463 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe9fa73-5f1d-47ea-b303-869b3392ac75-workload-socket\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg\" (UID: \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:07:25.932062 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.898402 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe9fa73-5f1d-47ea-b303-869b3392ac75-credential-socket\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg\" (UID: \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:07:25.932062 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.889957 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fe9fa73-5f1d-47ea-b303-869b3392ac75-istiod-ca-cert\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg\" (UID: \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:07:25.933211 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.933152 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnc27\" (UniqueName: \"kubernetes.io/projected/0fe9fa73-5f1d-47ea-b303-869b3392ac75-kube-api-access-nnc27\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg\" (UID: \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:07:25.942045 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.934384 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:25.942045 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.934947 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fe9fa73-5f1d-47ea-b303-869b3392ac75-istio-token\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg\" (UID: \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:07:25.942250 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.942134 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fe9fa73-5f1d-47ea-b303-869b3392ac75-istio-podinfo\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg\" (UID: \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:07:25.944956 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:25.944936 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fe9fa73-5f1d-47ea-b303-869b3392ac75-istio-envoy\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg\" (UID: \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:07:26.089116 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.089077 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:07:26.200714 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.200687 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx"] Oct 31 13:07:26.202299 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.202167 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:26.202528 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.202453 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ebddec1-8fab-4709-84de-f5d99a9112d0" containerName="istio-proxy" Oct 31 13:07:26.207946 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.207923 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:07:26.231382 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.231350 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx"] Oct 31 13:07:26.240839 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.238388 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-6cef79-openshift-default-dockercfg-4zrsr\"" Oct 31 13:07:26.294389 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.294362 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26wxb\" (UniqueName: \"kubernetes.io/projected/32580ef3-c0ea-46bc-8b62-c2b1362e998c-kube-api-access-26wxb\") pod \"e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx\" (UID: \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\") " pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:07:26.294541 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.294433 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32580ef3-c0ea-46bc-8b62-c2b1362e998c-istio-envoy\") pod \"e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx\" (UID: \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\") " pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:07:26.294541 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.294535 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32580ef3-c0ea-46bc-8b62-c2b1362e998c-credential-socket\") pod \"e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx\" (UID: \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\") " pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:07:26.294638 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.294593 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32580ef3-c0ea-46bc-8b62-c2b1362e998c-istio-podinfo\") pod \"e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx\" (UID: \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\") " pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:07:26.294674 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.294637 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32580ef3-c0ea-46bc-8b62-c2b1362e998c-workload-socket\") pod \"e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx\" (UID: \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\") " pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:07:26.294714 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.294653 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32580ef3-c0ea-46bc-8b62-c2b1362e998c-istio-data\") pod \"e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx\" (UID: \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\") " pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:07:26.294714 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.294691 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32580ef3-c0ea-46bc-8b62-c2b1362e998c-istio-token\") pod \"e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx\" (UID: \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\") " pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:07:26.294782 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.294770 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32580ef3-c0ea-46bc-8b62-c2b1362e998c-istiod-ca-cert\") pod \"e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx\" (UID: \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\") " pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:07:26.294837 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.294825 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32580ef3-c0ea-46bc-8b62-c2b1362e998c-workload-certs\") pod \"e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx\" (UID: \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\") " pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:07:26.428007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.402797 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26wxb\" (UniqueName: \"kubernetes.io/projected/32580ef3-c0ea-46bc-8b62-c2b1362e998c-kube-api-access-26wxb\") pod \"e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx\" (UID: \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\") " pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:07:26.428007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.426464 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32580ef3-c0ea-46bc-8b62-c2b1362e998c-istio-envoy\") pod \"e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx\" (UID: \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\") " pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:07:26.428007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.426552 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32580ef3-c0ea-46bc-8b62-c2b1362e998c-credential-socket\") pod \"e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx\" (UID: \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\") " pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:07:26.428007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.426622 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32580ef3-c0ea-46bc-8b62-c2b1362e998c-istio-podinfo\") pod \"e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx\" (UID: \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\") " pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:07:26.428007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.426659 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32580ef3-c0ea-46bc-8b62-c2b1362e998c-workload-socket\") pod \"e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx\" (UID: \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\") " pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:07:26.428007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.426687 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32580ef3-c0ea-46bc-8b62-c2b1362e998c-istio-data\") pod \"e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx\" (UID: \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\") " pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:07:26.428007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.426712 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32580ef3-c0ea-46bc-8b62-c2b1362e998c-istio-token\") pod \"e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx\" (UID: \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\") " pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:07:26.428007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.426806 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32580ef3-c0ea-46bc-8b62-c2b1362e998c-istiod-ca-cert\") pod \"e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx\" (UID: \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\") " pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:07:26.428007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.426903 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32580ef3-c0ea-46bc-8b62-c2b1362e998c-workload-certs\") pod \"e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx\" (UID: \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\") " pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:07:26.428007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.427433 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32580ef3-c0ea-46bc-8b62-c2b1362e998c-workload-certs\") pod \"e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx\" (UID: \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\") " pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:07:26.428007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.427974 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32580ef3-c0ea-46bc-8b62-c2b1362e998c-workload-socket\") pod \"e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx\" (UID: \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\") " pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:07:26.428745 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.428249 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32580ef3-c0ea-46bc-8b62-c2b1362e998c-credential-socket\") pod \"e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx\" (UID: \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\") " pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:07:26.438016 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.431589 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32580ef3-c0ea-46bc-8b62-c2b1362e998c-istio-data\") pod \"e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx\" (UID: \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\") " pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:07:26.438016 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.432217 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32580ef3-c0ea-46bc-8b62-c2b1362e998c-istiod-ca-cert\") pod \"e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx\" (UID: \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\") " pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:07:26.463088 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.440274 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32580ef3-c0ea-46bc-8b62-c2b1362e998c-istio-envoy\") pod \"e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx\" (UID: \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\") " pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:07:26.463088 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.449099 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26wxb\" (UniqueName: \"kubernetes.io/projected/32580ef3-c0ea-46bc-8b62-c2b1362e998c-kube-api-access-26wxb\") pod \"e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx\" (UID: \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\") " pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:07:26.463088 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.457549 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32580ef3-c0ea-46bc-8b62-c2b1362e998c-istio-podinfo\") pod \"e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx\" (UID: \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\") " pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:07:26.494645 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.494551 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32580ef3-c0ea-46bc-8b62-c2b1362e998c-istio-token\") pod \"e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx\" (UID: \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\") " pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:07:26.550006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.531662 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:07:26.550006 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:07:26.532149 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe9fa73_5f1d_47ea_b303_869b3392ac75.slice/crio-7bfd179b550088fdd872d1e00b447d5732b71beaf80e9e8a8341ecabadb3f609 WatchSource:0}: Error finding container 7bfd179b550088fdd872d1e00b447d5732b71beaf80e9e8a8341ecabadb3f609: Status 404 returned error can't find the container with id 7bfd179b550088fdd872d1e00b447d5732b71beaf80e9e8a8341ecabadb3f609 Oct 31 13:07:26.569201 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.551028 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:26.569201 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.551962 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg"] Oct 31 13:07:26.569201 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.554604 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:07:26.569201 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.562414 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:07:26.569201 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.562479 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:07:26.803060 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.789635 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:26.923483 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.923434 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx"] Oct 31 13:07:26.944892 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.944838 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:07:26.945048 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.944944 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:07:26.945133 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.945041 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:07:26.994910 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:26.994870 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:27.206435 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:27.206403 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:27.281833 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:27.281794 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" event={"ID":"0fe9fa73-5f1d-47ea-b303-869b3392ac75","Type":"ContainerStarted","Data":"e5ad7fee3046d5d4b15fd5814289d3b69f57aba3304c813816f26f282ea2c55d"} Oct 31 13:07:27.282065 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:27.281860 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" event={"ID":"0fe9fa73-5f1d-47ea-b303-869b3392ac75","Type":"ContainerStarted","Data":"7bfd179b550088fdd872d1e00b447d5732b71beaf80e9e8a8341ecabadb3f609"} Oct 31 13:07:27.283747 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:27.283719 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" event={"ID":"32580ef3-c0ea-46bc-8b62-c2b1362e998c","Type":"ContainerStarted","Data":"b3f2b12f6db769d6cda6d6e171cd458e27e99461a775f49ee91314795bf2d136"} Oct 31 13:07:27.283864 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:27.283771 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" event={"ID":"32580ef3-c0ea-46bc-8b62-c2b1362e998c","Type":"ContainerStarted","Data":"c7c5ccbc499e228d29a5f75f3de236bc0461bc831a3212099218ddbf0f7573c3"} Oct 31 13:07:27.308276 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:27.308226 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" podStartSLOduration=2.308213035 podStartE2EDuration="2.308213035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:07:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:07:27.3067549 +0000 UTC m=+2093.789877805" watchObservedRunningTime="2026-10-31 13:07:27.308213035 +0000 UTC m=+2093.791335935" Oct 31 13:07:27.333107 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:27.333060 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" podStartSLOduration=1.333046892 podStartE2EDuration="1.333046892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:07:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:07:27.329299574 +0000 UTC m=+2093.812422476" watchObservedRunningTime="2026-10-31 13:07:27.333046892 +0000 UTC m=+2093.816169792" Oct 31 13:07:27.536738 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:27.535914 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:07:27.757170 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:27.757136 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:27.966133 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:27.966093 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:28.068138 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.068097 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt"] Oct 31 13:07:28.072726 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.072701 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:07:28.083814 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.083788 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-45bcad-openshift-default-dockercfg-gfb47\"" Oct 31 13:07:28.089307 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.089286 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:07:28.096040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.096012 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:07:28.098310 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.098285 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt"] Oct 31 13:07:28.217089 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.216108 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c85bee9f-4267-44ec-a03f-0b97c6b95d24-workload-socket\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt\" (UID: \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:07:28.217089 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.216188 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c85bee9f-4267-44ec-a03f-0b97c6b95d24-istio-envoy\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt\" (UID: \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:07:28.217089 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.216227 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c85bee9f-4267-44ec-a03f-0b97c6b95d24-workload-certs\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt\" (UID: \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:07:28.217089 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.216394 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c85bee9f-4267-44ec-a03f-0b97c6b95d24-istio-data\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt\" (UID: \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:07:28.217089 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.216544 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c85bee9f-4267-44ec-a03f-0b97c6b95d24-istio-podinfo\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt\" (UID: \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:07:28.217089 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.216625 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zrtf\" (UniqueName: \"kubernetes.io/projected/c85bee9f-4267-44ec-a03f-0b97c6b95d24-kube-api-access-7zrtf\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt\" (UID: \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:07:28.217089 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.216680 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c85bee9f-4267-44ec-a03f-0b97c6b95d24-istiod-ca-cert\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt\" (UID: \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:07:28.217089 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.216709 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c85bee9f-4267-44ec-a03f-0b97c6b95d24-istio-token\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt\" (UID: \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:07:28.217089 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.216752 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c85bee9f-4267-44ec-a03f-0b97c6b95d24-credential-socket\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt\" (UID: \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:07:28.260006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.255260 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:28.331007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.330765 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c85bee9f-4267-44ec-a03f-0b97c6b95d24-istio-data\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt\" (UID: \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:07:28.331007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.330897 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c85bee9f-4267-44ec-a03f-0b97c6b95d24-istio-podinfo\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt\" (UID: \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:07:28.331007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.330944 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zrtf\" (UniqueName: \"kubernetes.io/projected/c85bee9f-4267-44ec-a03f-0b97c6b95d24-kube-api-access-7zrtf\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt\" (UID: \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:07:28.331007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.331007 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c85bee9f-4267-44ec-a03f-0b97c6b95d24-istiod-ca-cert\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt\" (UID: \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:07:28.331493 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.331046 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c85bee9f-4267-44ec-a03f-0b97c6b95d24-istio-token\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt\" (UID: \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:07:28.331493 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.331107 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c85bee9f-4267-44ec-a03f-0b97c6b95d24-credential-socket\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt\" (UID: \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:07:28.331493 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.331203 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c85bee9f-4267-44ec-a03f-0b97c6b95d24-workload-socket\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt\" (UID: \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:07:28.331493 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.331263 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c85bee9f-4267-44ec-a03f-0b97c6b95d24-istio-envoy\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt\" (UID: \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:07:28.331493 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.331297 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c85bee9f-4267-44ec-a03f-0b97c6b95d24-workload-certs\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt\" (UID: \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:07:28.338829 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.331842 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c85bee9f-4267-44ec-a03f-0b97c6b95d24-workload-certs\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt\" (UID: \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:07:28.338829 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.332491 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c85bee9f-4267-44ec-a03f-0b97c6b95d24-credential-socket\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt\" (UID: \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:07:28.338829 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.332864 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c85bee9f-4267-44ec-a03f-0b97c6b95d24-workload-socket\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt\" (UID: \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:07:28.338829 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.333668 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c85bee9f-4267-44ec-a03f-0b97c6b95d24-istio-data\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt\" (UID: \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:07:28.364739 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.339976 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:07:28.364739 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.355376 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c85bee9f-4267-44ec-a03f-0b97c6b95d24-istio-envoy\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt\" (UID: \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:07:28.364739 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.364541 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:07:28.366037 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.365925 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c85bee9f-4267-44ec-a03f-0b97c6b95d24-istiod-ca-cert\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt\" (UID: \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:07:28.372675 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.371581 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c85bee9f-4267-44ec-a03f-0b97c6b95d24-istio-podinfo\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt\" (UID: \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:07:28.386407 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.386340 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c85bee9f-4267-44ec-a03f-0b97c6b95d24-istio-token\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt\" (UID: \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:07:28.397775 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.397738 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zrtf\" (UniqueName: \"kubernetes.io/projected/c85bee9f-4267-44ec-a03f-0b97c6b95d24-kube-api-access-7zrtf\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt\" (UID: \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:07:28.510878 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.510722 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:28.541667 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.541638 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:07:28.689543 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:28.689506 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:07:29.062459 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:29.062431 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt"] Oct 31 13:07:29.063540 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:07:29.063511 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc85bee9f_4267_44ec_a03f_0b97c6b95d24.slice/crio-da69a7f4ee393af06fc3cd53158e23f03b92a40bbf1d196fe88866a3ccf36f04 WatchSource:0}: Error finding container da69a7f4ee393af06fc3cd53158e23f03b92a40bbf1d196fe88866a3ccf36f04: Status 404 returned error can't find the container with id da69a7f4ee393af06fc3cd53158e23f03b92a40bbf1d196fe88866a3ccf36f04 Oct 31 13:07:29.066100 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:29.066071 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:07:29.066165 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:29.066133 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:07:29.066206 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:29.066162 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:07:29.079757 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:29.079724 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:29.349453 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:29.349367 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" event={"ID":"c85bee9f-4267-44ec-a03f-0b97c6b95d24","Type":"ContainerStarted","Data":"1b1b3d24f4bcadf276d1dcf4ebedd9018d267f346abb926a4c64ac2f352d023f"} Oct 31 13:07:29.349453 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:29.349432 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" event={"ID":"c85bee9f-4267-44ec-a03f-0b97c6b95d24","Type":"ContainerStarted","Data":"da69a7f4ee393af06fc3cd53158e23f03b92a40bbf1d196fe88866a3ccf36f04"} Oct 31 13:07:29.349958 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:29.349870 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:07:29.350946 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:29.350925 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:07:29.381159 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:29.381107 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" podStartSLOduration=1.381090076 podStartE2EDuration="1.381090076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:07:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:07:29.376440871 +0000 UTC m=+2095.859563774" watchObservedRunningTime="2026-10-31 13:07:29.381090076 +0000 UTC m=+2095.864212979" Oct 31 13:07:29.690379 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:29.690295 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:07:29.707068 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:29.707017 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:29.938588 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:29.938550 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:30.288115 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:30.288082 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:30.696206 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:30.696179 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:07:31.108772 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:31.108686 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:31.124084 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:31.124055 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-7cdec6-5c4d8b7565-4lv4n"] Oct 31 13:07:31.128368 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:31.128348 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-7cdec6-5c4d8b7565-4lv4n" Oct 31 13:07:31.136221 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:31.136197 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-7cdec6-dockercfg-cvxp6\"" Oct 31 13:07:31.137944 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:31.136534 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-7cdec6\"" Oct 31 13:07:31.141407 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:31.141381 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-7cdec6-5c4d8b7565-4lv4n"] Oct 31 13:07:31.201379 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:31.201346 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eee4ba24-3680-41c1-adbf-c7432d2d20fc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-7cdec6-5c4d8b7565-4lv4n\" (UID: \"eee4ba24-3680-41c1-adbf-c7432d2d20fc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-7cdec6-5c4d8b7565-4lv4n" Oct 31 13:07:31.201595 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:31.201579 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjlwq\" (UniqueName: \"kubernetes.io/projected/eee4ba24-3680-41c1-adbf-c7432d2d20fc-kube-api-access-wjlwq\") pod \"payload-processing-e2e-shared-a-w0-7cdec6-5c4d8b7565-4lv4n\" (UID: \"eee4ba24-3680-41c1-adbf-c7432d2d20fc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-7cdec6-5c4d8b7565-4lv4n" Oct 31 13:07:31.334009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:31.333139 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eee4ba24-3680-41c1-adbf-c7432d2d20fc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-7cdec6-5c4d8b7565-4lv4n\" (UID: \"eee4ba24-3680-41c1-adbf-c7432d2d20fc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-7cdec6-5c4d8b7565-4lv4n" Oct 31 13:07:31.334009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:31.333498 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjlwq\" (UniqueName: \"kubernetes.io/projected/eee4ba24-3680-41c1-adbf-c7432d2d20fc-kube-api-access-wjlwq\") pod \"payload-processing-e2e-shared-a-w0-7cdec6-5c4d8b7565-4lv4n\" (UID: \"eee4ba24-3680-41c1-adbf-c7432d2d20fc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-7cdec6-5c4d8b7565-4lv4n" Oct 31 13:07:31.368065 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:31.348061 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:31.394007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:31.391931 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:07:31.427007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:31.425687 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:07:31.427007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:31.426003 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eee4ba24-3680-41c1-adbf-c7432d2d20fc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-7cdec6-5c4d8b7565-4lv4n\" (UID: \"eee4ba24-3680-41c1-adbf-c7432d2d20fc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-7cdec6-5c4d8b7565-4lv4n" Oct 31 13:07:31.491652 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:31.491616 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjlwq\" (UniqueName: \"kubernetes.io/projected/eee4ba24-3680-41c1-adbf-c7432d2d20fc-kube-api-access-wjlwq\") pod \"payload-processing-e2e-shared-a-w0-7cdec6-5c4d8b7565-4lv4n\" (UID: \"eee4ba24-3680-41c1-adbf-c7432d2d20fc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-7cdec6-5c4d8b7565-4lv4n" Oct 31 13:07:31.677946 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:31.677842 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:31.745426 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:31.745397 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-7cdec6-5c4d8b7565-4lv4n" Oct 31 13:07:31.952737 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:31.952704 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz"] Oct 31 13:07:31.957837 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:31.957805 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:07:31.986375 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:31.983471 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:31.991732 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:31.991703 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz"] Oct 31 13:07:32.005088 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.005059 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-4574ec-openshift-default-dockercfg-kg8h4\"" Oct 31 13:07:32.053067 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.053032 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f2bc576-3d6a-44d0-b30a-3eae4f627961-workload-socket\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz\" (UID: \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:07:32.053211 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.053088 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f2bc576-3d6a-44d0-b30a-3eae4f627961-workload-certs\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz\" (UID: \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:07:32.053211 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.053172 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f2bc576-3d6a-44d0-b30a-3eae4f627961-istio-envoy\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz\" (UID: \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:07:32.053298 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.053209 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f2bc576-3d6a-44d0-b30a-3eae4f627961-istio-data\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz\" (UID: \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:07:32.053337 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.053289 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f2bc576-3d6a-44d0-b30a-3eae4f627961-istio-podinfo\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz\" (UID: \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:07:32.053374 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.053345 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htk49\" (UniqueName: \"kubernetes.io/projected/4f2bc576-3d6a-44d0-b30a-3eae4f627961-kube-api-access-htk49\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz\" (UID: \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:07:32.053454 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.053417 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f2bc576-3d6a-44d0-b30a-3eae4f627961-istio-token\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz\" (UID: \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:07:32.053454 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.053447 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f2bc576-3d6a-44d0-b30a-3eae4f627961-istiod-ca-cert\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz\" (UID: \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:07:32.053521 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.053471 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f2bc576-3d6a-44d0-b30a-3eae4f627961-credential-socket\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz\" (UID: \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:07:32.230187 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.143239 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-7cdec6-5c4d8b7565-4lv4n"] Oct 31 13:07:32.230345 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.163300 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f2bc576-3d6a-44d0-b30a-3eae4f627961-workload-certs\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz\" (UID: \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:07:32.230345 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.230307 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f2bc576-3d6a-44d0-b30a-3eae4f627961-istio-envoy\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz\" (UID: \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:07:32.230345 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.163686 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f2bc576-3d6a-44d0-b30a-3eae4f627961-workload-certs\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz\" (UID: \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:07:32.230534 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.230357 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f2bc576-3d6a-44d0-b30a-3eae4f627961-istio-data\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz\" (UID: \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:07:32.230534 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.230431 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f2bc576-3d6a-44d0-b30a-3eae4f627961-istio-podinfo\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz\" (UID: \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:07:32.230642 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.230599 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htk49\" (UniqueName: \"kubernetes.io/projected/4f2bc576-3d6a-44d0-b30a-3eae4f627961-kube-api-access-htk49\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz\" (UID: \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:07:32.230697 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.230674 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f2bc576-3d6a-44d0-b30a-3eae4f627961-istio-token\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz\" (UID: \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:07:32.230761 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.230734 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f2bc576-3d6a-44d0-b30a-3eae4f627961-istiod-ca-cert\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz\" (UID: \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:07:32.230818 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.230778 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f2bc576-3d6a-44d0-b30a-3eae4f627961-credential-socket\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz\" (UID: \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:07:32.230939 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.230913 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f2bc576-3d6a-44d0-b30a-3eae4f627961-workload-socket\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz\" (UID: \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:07:32.231548 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.231510 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f2bc576-3d6a-44d0-b30a-3eae4f627961-workload-socket\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz\" (UID: \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:07:32.239782 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.239705 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:32.253262 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.249421 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f2bc576-3d6a-44d0-b30a-3eae4f627961-istio-data\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz\" (UID: \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:07:32.254648 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.254397 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f2bc576-3d6a-44d0-b30a-3eae4f627961-credential-socket\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz\" (UID: \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:07:32.255576 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.255010 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f2bc576-3d6a-44d0-b30a-3eae4f627961-istio-podinfo\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz\" (UID: \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:07:32.270068 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.263662 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f2bc576-3d6a-44d0-b30a-3eae4f627961-istiod-ca-cert\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz\" (UID: \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:07:32.303003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.299393 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f2bc576-3d6a-44d0-b30a-3eae4f627961-istio-token\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz\" (UID: \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:07:32.303003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.302732 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htk49\" (UniqueName: \"kubernetes.io/projected/4f2bc576-3d6a-44d0-b30a-3eae4f627961-kube-api-access-htk49\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz\" (UID: \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:07:32.401057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.398311 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f2bc576-3d6a-44d0-b30a-3eae4f627961-istio-envoy\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz\" (UID: \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:07:32.531959 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.531869 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:32.563553 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.563518 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-7cdec6-5c4d8b7565-4lv4n" event={"ID":"eee4ba24-3680-41c1-adbf-c7432d2d20fc","Type":"ContainerStarted","Data":"6d77a123fb2b16b93b2640844954f10a0f0ee41c5c3a6ed69455e9cf6cf4e932"} Oct 31 13:07:32.616665 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.615886 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:07:32.907807 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.907775 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz"] Oct 31 13:07:32.908946 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:07:32.908914 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f2bc576_3d6a_44d0_b30a_3eae4f627961.slice/crio-8dd2c2e63e5f791419d317560898f31fb97987cdb2aa3a63ad03c964df704b21 WatchSource:0}: Error finding container 8dd2c2e63e5f791419d317560898f31fb97987cdb2aa3a63ad03c964df704b21: Status 404 returned error can't find the container with id 8dd2c2e63e5f791419d317560898f31fb97987cdb2aa3a63ad03c964df704b21 Oct 31 13:07:32.911556 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.911521 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:07:32.911655 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.911586 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:07:32.911655 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:32.911615 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:07:33.165113 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:33.165022 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:33.308282 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:33.308242 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-45bcad-6f4486b69f-qtdsk"] Oct 31 13:07:33.313091 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:33.313060 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-45bcad-6f4486b69f-qtdsk" Oct 31 13:07:33.330056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:33.330028 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-45bcad\"" Oct 31 13:07:33.330056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:33.330028 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-45bcad-dockercfg-tgmct\"" Oct 31 13:07:33.345531 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:33.345460 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-45bcad-6f4486b69f-qtdsk"] Oct 31 13:07:33.369023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:33.368946 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz8x4\" (UniqueName: \"kubernetes.io/projected/68db5d02-47c3-4605-8967-30a007c782a7-kube-api-access-dz8x4\") pod \"payload-pre-processing-e2e-shared-a-w2-45bcad-6f4486b69f-qtdsk\" (UID: \"68db5d02-47c3-4605-8967-30a007c782a7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-45bcad-6f4486b69f-qtdsk" Oct 31 13:07:33.369216 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:33.369086 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68db5d02-47c3-4605-8967-30a007c782a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-45bcad-6f4486b69f-qtdsk\" (UID: \"68db5d02-47c3-4605-8967-30a007c782a7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-45bcad-6f4486b69f-qtdsk" Oct 31 13:07:33.522007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:33.521591 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz8x4\" (UniqueName: \"kubernetes.io/projected/68db5d02-47c3-4605-8967-30a007c782a7-kube-api-access-dz8x4\") pod \"payload-pre-processing-e2e-shared-a-w2-45bcad-6f4486b69f-qtdsk\" (UID: \"68db5d02-47c3-4605-8967-30a007c782a7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-45bcad-6f4486b69f-qtdsk" Oct 31 13:07:33.522007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:33.521686 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68db5d02-47c3-4605-8967-30a007c782a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-45bcad-6f4486b69f-qtdsk\" (UID: \"68db5d02-47c3-4605-8967-30a007c782a7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-45bcad-6f4486b69f-qtdsk" Oct 31 13:07:33.548266 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:33.531010 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68db5d02-47c3-4605-8967-30a007c782a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-45bcad-6f4486b69f-qtdsk\" (UID: \"68db5d02-47c3-4605-8967-30a007c782a7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-45bcad-6f4486b69f-qtdsk" Oct 31 13:07:33.623696 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:33.623654 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:33.688084 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:33.685896 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz8x4\" (UniqueName: \"kubernetes.io/projected/68db5d02-47c3-4605-8967-30a007c782a7-kube-api-access-dz8x4\") pod \"payload-pre-processing-e2e-shared-a-w2-45bcad-6f4486b69f-qtdsk\" (UID: \"68db5d02-47c3-4605-8967-30a007c782a7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-45bcad-6f4486b69f-qtdsk" Oct 31 13:07:33.695456 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:33.695427 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-7cdec6-5c4d8b7565-4lv4n" Oct 31 13:07:33.695628 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:33.695490 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" event={"ID":"4f2bc576-3d6a-44d0-b30a-3eae4f627961","Type":"ContainerStarted","Data":"87e55479dcc685d7818ca1d579dbd3fa589e7cccc2fc22ec0f440007157d5a66"} Oct 31 13:07:33.695628 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:33.695522 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" event={"ID":"4f2bc576-3d6a-44d0-b30a-3eae4f627961","Type":"ContainerStarted","Data":"8dd2c2e63e5f791419d317560898f31fb97987cdb2aa3a63ad03c964df704b21"} Oct 31 13:07:33.695628 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:33.695547 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-7cdec6-5c4d8b7565-4lv4n" event={"ID":"eee4ba24-3680-41c1-adbf-c7432d2d20fc","Type":"ContainerStarted","Data":"2595865c9ab28f267007edee0ac245cceab442b6c01d6f18b33f6a378406bf67"} Oct 31 13:07:33.707919 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:33.707862 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" podStartSLOduration=2.7078441 podStartE2EDuration="2.7078441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:07:33.706194428 +0000 UTC m=+2100.189317329" watchObservedRunningTime="2026-10-31 13:07:33.7078441 +0000 UTC m=+2100.190967003" Oct 31 13:07:33.745774 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:33.745719 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-7cdec6-5c4d8b7565-4lv4n" podStartSLOduration=2.745702604 podStartE2EDuration="2.745702604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:07:33.742954975 +0000 UTC m=+2100.226077877" watchObservedRunningTime="2026-10-31 13:07:33.745702604 +0000 UTC m=+2100.228825506" Oct 31 13:07:33.941059 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:33.940901 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-45bcad-dockercfg-tgmct\"" Oct 31 13:07:33.949939 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:33.949912 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-45bcad-6f4486b69f-qtdsk" Oct 31 13:07:34.266294 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:34.266261 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:34.268273 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:34.268246 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-45bcad-6f4486b69f-qtdsk"] Oct 31 13:07:34.271150 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:07:34.271119 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68db5d02_47c3_4605_8967_30a007c782a7.slice/crio-13b0e62220ad3a4c07d5503d0200e4c6592ccdf7e30e723bedd40e98e658a340 WatchSource:0}: Error finding container 13b0e62220ad3a4c07d5503d0200e4c6592ccdf7e30e723bedd40e98e658a340: Status 404 returned error can't find the container with id 13b0e62220ad3a4c07d5503d0200e4c6592ccdf7e30e723bedd40e98e658a340 Oct 31 13:07:34.617716 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:34.617633 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:07:34.622790 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:34.622763 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:07:34.692871 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:34.692837 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-45bcad-6f4486b69f-qtdsk" event={"ID":"68db5d02-47c3-4605-8967-30a007c782a7","Type":"ContainerStarted","Data":"f4f2ab140e6c994376056efa1b15143f9f69db4fba640bc7aa2b8e20047a7453"} Oct 31 13:07:34.693081 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:34.692905 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-45bcad-6f4486b69f-qtdsk" event={"ID":"68db5d02-47c3-4605-8967-30a007c782a7","Type":"ContainerStarted","Data":"13b0e62220ad3a4c07d5503d0200e4c6592ccdf7e30e723bedd40e98e658a340"} Oct 31 13:07:34.693081 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:34.692937 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-45bcad-6f4486b69f-qtdsk" Oct 31 13:07:34.693373 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:34.693354 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:07:34.694239 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:34.694218 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:07:34.712647 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:34.712604 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-45bcad-6f4486b69f-qtdsk" podStartSLOduration=1.712592632 podStartE2EDuration="1.712592632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:07:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:07:34.71069257 +0000 UTC m=+2101.193815495" watchObservedRunningTime="2026-10-31 13:07:34.712592632 +0000 UTC m=+2101.195715533" Oct 31 13:07:35.087329 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:35.087294 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:35.864348 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:35.864307 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c"] Oct 31 13:07:35.868819 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:35.868799 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c" Oct 31 13:07:35.876785 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:35.876761 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-7cdec6-serving-cert\"" Oct 31 13:07:35.879278 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:35.879256 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c"] Oct 31 13:07:35.886513 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:35.886483 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/682058fc-61a5-4c39-a42f-ada4dc83a0cc-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c\" (UID: \"682058fc-61a5-4c39-a42f-ada4dc83a0cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c" Oct 31 13:07:35.886627 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:35.886545 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hsvq\" (UniqueName: \"kubernetes.io/projected/682058fc-61a5-4c39-a42f-ada4dc83a0cc-kube-api-access-5hsvq\") pod \"maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c\" (UID: \"682058fc-61a5-4c39-a42f-ada4dc83a0cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c" Oct 31 13:07:35.886746 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:35.886715 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/682058fc-61a5-4c39-a42f-ada4dc83a0cc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c\" (UID: \"682058fc-61a5-4c39-a42f-ada4dc83a0cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c" Oct 31 13:07:35.886896 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:35.886779 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/682058fc-61a5-4c39-a42f-ada4dc83a0cc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c\" (UID: \"682058fc-61a5-4c39-a42f-ada4dc83a0cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c" Oct 31 13:07:36.001006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:35.997002 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/682058fc-61a5-4c39-a42f-ada4dc83a0cc-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c\" (UID: \"682058fc-61a5-4c39-a42f-ada4dc83a0cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c" Oct 31 13:07:36.001006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:35.997079 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hsvq\" (UniqueName: \"kubernetes.io/projected/682058fc-61a5-4c39-a42f-ada4dc83a0cc-kube-api-access-5hsvq\") pod \"maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c\" (UID: \"682058fc-61a5-4c39-a42f-ada4dc83a0cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c" Oct 31 13:07:36.001006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:35.997213 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/682058fc-61a5-4c39-a42f-ada4dc83a0cc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c\" (UID: \"682058fc-61a5-4c39-a42f-ada4dc83a0cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c" Oct 31 13:07:36.001006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:35.997271 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/682058fc-61a5-4c39-a42f-ada4dc83a0cc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c\" (UID: \"682058fc-61a5-4c39-a42f-ada4dc83a0cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c" Oct 31 13:07:36.001006 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:07:35.997566 6841 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-7cdec6-serving-cert: secret "maas-api-e2e-shared-a-w0-7cdec6-serving-cert" not found Oct 31 13:07:36.001006 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:07:35.997642 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/682058fc-61a5-4c39-a42f-ada4dc83a0cc-maas-api-tls podName:682058fc-61a5-4c39-a42f-ada4dc83a0cc nodeName:}" failed. No retries permitted until 2026-10-31 13:07:36.497621803 +0000 UTC m=+2102.980744682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/682058fc-61a5-4c39-a42f-ada4dc83a0cc-maas-api-tls") pod "maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c" (UID: "682058fc-61a5-4c39-a42f-ada4dc83a0cc") : secret "maas-api-e2e-shared-a-w0-7cdec6-serving-cert" not found Oct 31 13:07:36.001006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:35.999009 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/682058fc-61a5-4c39-a42f-ada4dc83a0cc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c\" (UID: \"682058fc-61a5-4c39-a42f-ada4dc83a0cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c" Oct 31 13:07:36.002633 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:36.002606 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/682058fc-61a5-4c39-a42f-ada4dc83a0cc-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c\" (UID: \"682058fc-61a5-4c39-a42f-ada4dc83a0cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c" Oct 31 13:07:36.029491 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:36.024781 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:36.029491 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:36.025622 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hsvq\" (UniqueName: \"kubernetes.io/projected/682058fc-61a5-4c39-a42f-ada4dc83a0cc-kube-api-access-5hsvq\") pod \"maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c\" (UID: \"682058fc-61a5-4c39-a42f-ada4dc83a0cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c" Oct 31 13:07:36.312057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:36.312012 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:36.506142 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:36.506107 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/682058fc-61a5-4c39-a42f-ada4dc83a0cc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c\" (UID: \"682058fc-61a5-4c39-a42f-ada4dc83a0cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c" Oct 31 13:07:36.508709 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:36.508683 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/682058fc-61a5-4c39-a42f-ada4dc83a0cc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c\" (UID: \"682058fc-61a5-4c39-a42f-ada4dc83a0cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c" Oct 31 13:07:36.787315 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:36.787267 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c" Oct 31 13:07:37.133945 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:37.133862 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn"] Oct 31 13:07:37.139452 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:37.139422 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn" Oct 31 13:07:37.168619 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:37.168592 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn"] Oct 31 13:07:37.173355 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:37.173327 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-6cef79-serving-cert\"" Oct 31 13:07:37.173694 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:37.173667 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c"] Oct 31 13:07:37.217250 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:37.217215 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d996d76a-0e68-45da-855d-c59a99bb2395-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn\" (UID: \"d996d76a-0e68-45da-855d-c59a99bb2395\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn" Oct 31 13:07:37.217407 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:37.217383 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d996d76a-0e68-45da-855d-c59a99bb2395-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn\" (UID: \"d996d76a-0e68-45da-855d-c59a99bb2395\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn" Oct 31 13:07:37.217501 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:37.217479 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn4q4\" (UniqueName: \"kubernetes.io/projected/d996d76a-0e68-45da-855d-c59a99bb2395-kube-api-access-jn4q4\") pod \"maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn\" (UID: \"d996d76a-0e68-45da-855d-c59a99bb2395\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn" Oct 31 13:07:37.217545 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:37.217519 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d996d76a-0e68-45da-855d-c59a99bb2395-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn\" (UID: \"d996d76a-0e68-45da-855d-c59a99bb2395\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn" Oct 31 13:07:37.311612 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:37.311578 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:37.318954 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:37.318927 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d996d76a-0e68-45da-855d-c59a99bb2395-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn\" (UID: \"d996d76a-0e68-45da-855d-c59a99bb2395\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn" Oct 31 13:07:37.319412 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:37.319142 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d996d76a-0e68-45da-855d-c59a99bb2395-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn\" (UID: \"d996d76a-0e68-45da-855d-c59a99bb2395\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn" Oct 31 13:07:37.319412 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:37.319294 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d996d76a-0e68-45da-855d-c59a99bb2395-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn\" (UID: \"d996d76a-0e68-45da-855d-c59a99bb2395\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn" Oct 31 13:07:37.319412 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:37.319342 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn4q4\" (UniqueName: \"kubernetes.io/projected/d996d76a-0e68-45da-855d-c59a99bb2395-kube-api-access-jn4q4\") pod \"maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn\" (UID: \"d996d76a-0e68-45da-855d-c59a99bb2395\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn" Oct 31 13:07:37.319931 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:37.319903 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d996d76a-0e68-45da-855d-c59a99bb2395-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn\" (UID: \"d996d76a-0e68-45da-855d-c59a99bb2395\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn" Oct 31 13:07:37.321796 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:37.321776 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d996d76a-0e68-45da-855d-c59a99bb2395-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn\" (UID: \"d996d76a-0e68-45da-855d-c59a99bb2395\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn" Oct 31 13:07:37.321912 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:37.321882 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d996d76a-0e68-45da-855d-c59a99bb2395-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn\" (UID: \"d996d76a-0e68-45da-855d-c59a99bb2395\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn" Oct 31 13:07:37.331963 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:37.331939 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn4q4\" (UniqueName: \"kubernetes.io/projected/d996d76a-0e68-45da-855d-c59a99bb2395-kube-api-access-jn4q4\") pod \"maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn\" (UID: \"d996d76a-0e68-45da-855d-c59a99bb2395\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn" Oct 31 13:07:37.461597 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:37.461282 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn" Oct 31 13:07:37.624174 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:37.624141 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:37.727119 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:37.727093 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c" event={"ID":"682058fc-61a5-4c39-a42f-ada4dc83a0cc","Type":"ContainerStarted","Data":"cf1c1bd7982b858dd9ce4c06cd40cc7681c3c340a0a19fca45a7b85ceb086520"} Oct 31 13:07:37.769081 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:37.769052 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn"] Oct 31 13:07:37.769384 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:07:37.769351 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd996d76a_0e68_45da_855d_c59a99bb2395.slice/crio-c15ac616349c17d9b7f21423ddadbd86bbc136049cfd1d13fcb66ada3905a8d3 WatchSource:0}: Error finding container c15ac616349c17d9b7f21423ddadbd86bbc136049cfd1d13fcb66ada3905a8d3: Status 404 returned error can't find the container with id c15ac616349c17d9b7f21423ddadbd86bbc136049cfd1d13fcb66ada3905a8d3 Oct 31 13:07:37.965245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:37.965208 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:38.195058 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.187592 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:38.332677 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.332640 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh"] Oct 31 13:07:38.338281 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.338249 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" Oct 31 13:07:38.341604 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.341577 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh"] Oct 31 13:07:38.358172 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.358147 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz"] Oct 31 13:07:38.362225 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.362202 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" Oct 31 13:07:38.387693 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.387660 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz"] Oct 31 13:07:38.458227 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.458200 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6fdf066-cd83-408d-9c1c-6c97f9189f33-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh\" (UID: \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" Oct 31 13:07:38.458369 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.458242 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q2nv\" (UniqueName: \"kubernetes.io/projected/d6fdf066-cd83-408d-9c1c-6c97f9189f33-kube-api-access-5q2nv\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh\" (UID: \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" Oct 31 13:07:38.458369 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.458323 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6fdf066-cd83-408d-9c1c-6c97f9189f33-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh\" (UID: \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" Oct 31 13:07:38.458617 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.458458 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6fdf066-cd83-408d-9c1c-6c97f9189f33-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh\" (UID: \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" Oct 31 13:07:38.458617 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.458505 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6fdf066-cd83-408d-9c1c-6c97f9189f33-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh\" (UID: \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" Oct 31 13:07:38.458617 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.458541 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6fdf066-cd83-408d-9c1c-6c97f9189f33-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh\" (UID: \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" Oct 31 13:07:38.458617 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.458615 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6fdf066-cd83-408d-9c1c-6c97f9189f33-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh\" (UID: \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" Oct 31 13:07:38.458848 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.458637 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6fdf066-cd83-408d-9c1c-6c97f9189f33-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh\" (UID: \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" Oct 31 13:07:38.458848 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.458698 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6fdf066-cd83-408d-9c1c-6c97f9189f33-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh\" (UID: \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" Oct 31 13:07:38.560526 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.560488 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6fdf066-cd83-408d-9c1c-6c97f9189f33-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh\" (UID: \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" Oct 31 13:07:38.560696 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.560540 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6fdf066-cd83-408d-9c1c-6c97f9189f33-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh\" (UID: \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" Oct 31 13:07:38.560696 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.560577 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6fdf066-cd83-408d-9c1c-6c97f9189f33-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh\" (UID: \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" Oct 31 13:07:38.560696 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.560609 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88130f73-b9a3-4bdb-aea9-8557803120cd-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz\" (UID: \"88130f73-b9a3-4bdb-aea9-8557803120cd\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" Oct 31 13:07:38.560696 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.560646 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6fdf066-cd83-408d-9c1c-6c97f9189f33-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh\" (UID: \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" Oct 31 13:07:38.560866 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.560729 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6fdf066-cd83-408d-9c1c-6c97f9189f33-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh\" (UID: \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" Oct 31 13:07:38.560866 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.560803 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6fdf066-cd83-408d-9c1c-6c97f9189f33-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh\" (UID: \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" Oct 31 13:07:38.560973 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.560863 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j8bf\" (UniqueName: \"kubernetes.io/projected/88130f73-b9a3-4bdb-aea9-8557803120cd-kube-api-access-5j8bf\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz\" (UID: \"88130f73-b9a3-4bdb-aea9-8557803120cd\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" Oct 31 13:07:38.560973 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.560947 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88130f73-b9a3-4bdb-aea9-8557803120cd-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz\" (UID: \"88130f73-b9a3-4bdb-aea9-8557803120cd\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" Oct 31 13:07:38.561123 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.561083 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6fdf066-cd83-408d-9c1c-6c97f9189f33-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh\" (UID: \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" Oct 31 13:07:38.561187 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.561151 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6fdf066-cd83-408d-9c1c-6c97f9189f33-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh\" (UID: \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" Oct 31 13:07:38.561242 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.561200 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6fdf066-cd83-408d-9c1c-6c97f9189f33-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh\" (UID: \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" Oct 31 13:07:38.561379 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.561238 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88130f73-b9a3-4bdb-aea9-8557803120cd-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz\" (UID: \"88130f73-b9a3-4bdb-aea9-8557803120cd\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" Oct 31 13:07:38.561379 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.561204 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6fdf066-cd83-408d-9c1c-6c97f9189f33-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh\" (UID: \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" Oct 31 13:07:38.561379 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.561305 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q2nv\" (UniqueName: \"kubernetes.io/projected/d6fdf066-cd83-408d-9c1c-6c97f9189f33-kube-api-access-5q2nv\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh\" (UID: \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" Oct 31 13:07:38.561379 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.561301 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6fdf066-cd83-408d-9c1c-6c97f9189f33-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh\" (UID: \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" Oct 31 13:07:38.561379 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.561348 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6fdf066-cd83-408d-9c1c-6c97f9189f33-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh\" (UID: \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" Oct 31 13:07:38.561637 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.561394 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88130f73-b9a3-4bdb-aea9-8557803120cd-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz\" (UID: \"88130f73-b9a3-4bdb-aea9-8557803120cd\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" Oct 31 13:07:38.561637 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.561474 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88130f73-b9a3-4bdb-aea9-8557803120cd-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz\" (UID: \"88130f73-b9a3-4bdb-aea9-8557803120cd\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" Oct 31 13:07:38.561637 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.561508 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6fdf066-cd83-408d-9c1c-6c97f9189f33-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh\" (UID: \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" Oct 31 13:07:38.561637 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.561529 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88130f73-b9a3-4bdb-aea9-8557803120cd-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz\" (UID: \"88130f73-b9a3-4bdb-aea9-8557803120cd\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" Oct 31 13:07:38.561637 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.561571 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88130f73-b9a3-4bdb-aea9-8557803120cd-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz\" (UID: \"88130f73-b9a3-4bdb-aea9-8557803120cd\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" Oct 31 13:07:38.561637 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.561616 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88130f73-b9a3-4bdb-aea9-8557803120cd-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz\" (UID: \"88130f73-b9a3-4bdb-aea9-8557803120cd\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" Oct 31 13:07:38.563741 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.563713 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6fdf066-cd83-408d-9c1c-6c97f9189f33-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh\" (UID: \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" Oct 31 13:07:38.563909 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.563885 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6fdf066-cd83-408d-9c1c-6c97f9189f33-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh\" (UID: \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" Oct 31 13:07:38.633288 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.633123 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q2nv\" (UniqueName: \"kubernetes.io/projected/d6fdf066-cd83-408d-9c1c-6c97f9189f33-kube-api-access-5q2nv\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh\" (UID: \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" Oct 31 13:07:38.633629 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.633601 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6fdf066-cd83-408d-9c1c-6c97f9189f33-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh\" (UID: \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" Oct 31 13:07:38.664017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.663206 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88130f73-b9a3-4bdb-aea9-8557803120cd-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz\" (UID: \"88130f73-b9a3-4bdb-aea9-8557803120cd\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" Oct 31 13:07:38.664017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.663339 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j8bf\" (UniqueName: \"kubernetes.io/projected/88130f73-b9a3-4bdb-aea9-8557803120cd-kube-api-access-5j8bf\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz\" (UID: \"88130f73-b9a3-4bdb-aea9-8557803120cd\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" Oct 31 13:07:38.664017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.663406 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88130f73-b9a3-4bdb-aea9-8557803120cd-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz\" (UID: \"88130f73-b9a3-4bdb-aea9-8557803120cd\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" Oct 31 13:07:38.664017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.663491 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88130f73-b9a3-4bdb-aea9-8557803120cd-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz\" (UID: \"88130f73-b9a3-4bdb-aea9-8557803120cd\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" Oct 31 13:07:38.664017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.663566 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88130f73-b9a3-4bdb-aea9-8557803120cd-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz\" (UID: \"88130f73-b9a3-4bdb-aea9-8557803120cd\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" Oct 31 13:07:38.664017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.663635 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88130f73-b9a3-4bdb-aea9-8557803120cd-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz\" (UID: \"88130f73-b9a3-4bdb-aea9-8557803120cd\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" Oct 31 13:07:38.664017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.663662 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88130f73-b9a3-4bdb-aea9-8557803120cd-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz\" (UID: \"88130f73-b9a3-4bdb-aea9-8557803120cd\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" Oct 31 13:07:38.664017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.663688 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88130f73-b9a3-4bdb-aea9-8557803120cd-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz\" (UID: \"88130f73-b9a3-4bdb-aea9-8557803120cd\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" Oct 31 13:07:38.664017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.663715 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88130f73-b9a3-4bdb-aea9-8557803120cd-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz\" (UID: \"88130f73-b9a3-4bdb-aea9-8557803120cd\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" Oct 31 13:07:38.679081 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.677504 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88130f73-b9a3-4bdb-aea9-8557803120cd-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz\" (UID: \"88130f73-b9a3-4bdb-aea9-8557803120cd\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" Oct 31 13:07:38.679081 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.677808 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88130f73-b9a3-4bdb-aea9-8557803120cd-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz\" (UID: \"88130f73-b9a3-4bdb-aea9-8557803120cd\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" Oct 31 13:07:38.679081 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.678068 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88130f73-b9a3-4bdb-aea9-8557803120cd-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz\" (UID: \"88130f73-b9a3-4bdb-aea9-8557803120cd\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" Oct 31 13:07:38.679081 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.679036 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88130f73-b9a3-4bdb-aea9-8557803120cd-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz\" (UID: \"88130f73-b9a3-4bdb-aea9-8557803120cd\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" Oct 31 13:07:38.683803 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.682363 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88130f73-b9a3-4bdb-aea9-8557803120cd-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz\" (UID: \"88130f73-b9a3-4bdb-aea9-8557803120cd\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" Oct 31 13:07:38.683803 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.682739 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88130f73-b9a3-4bdb-aea9-8557803120cd-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz\" (UID: \"88130f73-b9a3-4bdb-aea9-8557803120cd\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" Oct 31 13:07:38.683803 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.683393 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" Oct 31 13:07:38.694712 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.694687 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88130f73-b9a3-4bdb-aea9-8557803120cd-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz\" (UID: \"88130f73-b9a3-4bdb-aea9-8557803120cd\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" Oct 31 13:07:38.709449 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.709422 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88130f73-b9a3-4bdb-aea9-8557803120cd-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz\" (UID: \"88130f73-b9a3-4bdb-aea9-8557803120cd\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" Oct 31 13:07:38.709663 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.709643 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j8bf\" (UniqueName: \"kubernetes.io/projected/88130f73-b9a3-4bdb-aea9-8557803120cd-kube-api-access-5j8bf\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz\" (UID: \"88130f73-b9a3-4bdb-aea9-8557803120cd\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" Oct 31 13:07:38.735733 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.735697 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c" event={"ID":"682058fc-61a5-4c39-a42f-ada4dc83a0cc","Type":"ContainerStarted","Data":"cf01ccae404ac6996c33cc779525af4ba63f384def38d06074cf763bcfb3f44c"} Oct 31 13:07:38.735920 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.735810 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c" Oct 31 13:07:38.737608 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.737576 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn" event={"ID":"d996d76a-0e68-45da-855d-c59a99bb2395","Type":"ContainerStarted","Data":"28bff7ddfa58b2a00a60e357e305d9303666f9b8362945e8457e3d5f0e885429"} Oct 31 13:07:38.737739 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.737629 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn" Oct 31 13:07:38.737739 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.737649 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn" event={"ID":"d996d76a-0e68-45da-855d-c59a99bb2395","Type":"ContainerStarted","Data":"c15ac616349c17d9b7f21423ddadbd86bbc136049cfd1d13fcb66ada3905a8d3"} Oct 31 13:07:38.767034 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.766080 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:38.772914 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.772851 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c" podStartSLOduration=3.235001969 podStartE2EDuration="3.772831915s" totalImagesPullingTime="537.829946ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:07:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:07:38.770575659 +0000 UTC m=+2105.253698560" watchObservedRunningTime="2026-10-31 13:07:38.772831915 +0000 UTC m=+2105.255954816" Oct 31 13:07:38.809629 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.809570 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn" podStartSLOduration=1.184750863 podStartE2EDuration="1.809550858s" totalImagesPullingTime="624.799995ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:07:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:07:38.808963423 +0000 UTC m=+2105.292086324" watchObservedRunningTime="2026-10-31 13:07:38.809550858 +0000 UTC m=+2105.292673764" Oct 31 13:07:38.990084 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:38.986304 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" Oct 31 13:07:39.042463 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:39.042434 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh"] Oct 31 13:07:39.044064 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:07:39.044035 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6fdf066_cd83_408d_9c1c_6c97f9189f33.slice/crio-dec44256f518e7139bd768fd3237e7629058d6db5877c89c727220c3deed5f56 WatchSource:0}: Error finding container dec44256f518e7139bd768fd3237e7629058d6db5877c89c727220c3deed5f56: Status 404 returned error can't find the container with id dec44256f518e7139bd768fd3237e7629058d6db5877c89c727220c3deed5f56 Oct 31 13:07:39.047045 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:39.047010 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:07:39.047132 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:39.047080 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:07:39.047132 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:39.047115 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:07:39.309517 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:39.309487 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz"] Oct 31 13:07:39.310397 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:07:39.310368 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88130f73_b9a3_4bdb_aea9_8557803120cd.slice/crio-6e0ac44c7785581911206d8e5665881110cc03668915506247d646a97c6efe1b WatchSource:0}: Error finding container 6e0ac44c7785581911206d8e5665881110cc03668915506247d646a97c6efe1b: Status 404 returned error can't find the container with id 6e0ac44c7785581911206d8e5665881110cc03668915506247d646a97c6efe1b Oct 31 13:07:39.313474 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:39.313436 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:07:39.313611 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:39.313522 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:07:39.313611 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:39.313551 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:07:39.711297 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:39.711189 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:39.746851 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:39.746817 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" event={"ID":"d6fdf066-cd83-408d-9c1c-6c97f9189f33","Type":"ContainerStarted","Data":"a978080ec4f8e1250f3f8f4caa420a97949bd4955f0d1632ea4208f690ccc40c"} Oct 31 13:07:39.747041 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:39.746876 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" event={"ID":"d6fdf066-cd83-408d-9c1c-6c97f9189f33","Type":"ContainerStarted","Data":"dec44256f518e7139bd768fd3237e7629058d6db5877c89c727220c3deed5f56"} Oct 31 13:07:39.749050 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:39.749020 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" event={"ID":"88130f73-b9a3-4bdb-aea9-8557803120cd","Type":"ContainerStarted","Data":"6f64a098a6c9c98c5415c07a9dd0056fa668064189f876792c8718448e12c60b"} Oct 31 13:07:39.749174 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:39.749070 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" event={"ID":"88130f73-b9a3-4bdb-aea9-8557803120cd","Type":"ContainerStarted","Data":"6e0ac44c7785581911206d8e5665881110cc03668915506247d646a97c6efe1b"} Oct 31 13:07:39.778876 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:39.778828 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" podStartSLOduration=1.778814745 podStartE2EDuration="1.778814745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:07:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:07:39.773242229 +0000 UTC m=+2106.256365134" watchObservedRunningTime="2026-10-31 13:07:39.778814745 +0000 UTC m=+2106.261937645" Oct 31 13:07:39.799424 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:39.799365 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" podStartSLOduration=1.799347568 podStartE2EDuration="1.799347568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:07:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:07:39.797963568 +0000 UTC m=+2106.281086473" watchObservedRunningTime="2026-10-31 13:07:39.799347568 +0000 UTC m=+2106.282470472" Oct 31 13:07:39.986810 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:39.986778 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" Oct 31 13:07:40.273496 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:40.273357 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:40.483035 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:40.479702 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:40.690169 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:40.689204 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" Oct 31 13:07:40.690169 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:40.689367 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 31 13:07:40.690169 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:40.689415 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" podUID="d6fdf066-cd83-408d-9c1c-6c97f9189f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 31 13:07:40.810524 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:40.810497 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:41.045724 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:41.042543 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 31 13:07:41.045724 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:41.042609 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" podUID="88130f73-b9a3-4bdb-aea9-8557803120cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 31 13:07:41.323332 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:41.316685 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:41.688643 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:41.688541 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 31 13:07:41.688934 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:41.688891 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" podUID="d6fdf066-cd83-408d-9c1c-6c97f9189f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 31 13:07:41.998359 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:41.995245 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 31 13:07:41.998359 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:41.995319 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" podUID="88130f73-b9a3-4bdb-aea9-8557803120cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 31 13:07:42.091692 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:42.087673 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:42.689005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:42.688439 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 31 13:07:42.689005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:42.688506 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" podUID="d6fdf066-cd83-408d-9c1c-6c97f9189f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 31 13:07:42.808005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:42.807643 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:42.988017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:42.987701 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 31 13:07:42.988017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:42.987759 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" podUID="88130f73-b9a3-4bdb-aea9-8557803120cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 31 13:07:43.410904 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:43.410817 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps"] Oct 31 13:07:43.417363 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:43.417097 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps"] Oct 31 13:07:43.417363 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:43.417207 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps" Oct 31 13:07:43.422473 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:43.422449 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-4574ec-serving-cert\"" Oct 31 13:07:43.503432 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:43.503380 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1faea580-200c-4d2e-8ead-ef3d6cdae95b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps\" (UID: \"1faea580-200c-4d2e-8ead-ef3d6cdae95b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps" Oct 31 13:07:43.503628 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:43.503512 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1faea580-200c-4d2e-8ead-ef3d6cdae95b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps\" (UID: \"1faea580-200c-4d2e-8ead-ef3d6cdae95b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps" Oct 31 13:07:43.503628 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:43.503582 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1faea580-200c-4d2e-8ead-ef3d6cdae95b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps\" (UID: \"1faea580-200c-4d2e-8ead-ef3d6cdae95b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps" Oct 31 13:07:43.503798 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:43.503775 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7frw\" (UniqueName: \"kubernetes.io/projected/1faea580-200c-4d2e-8ead-ef3d6cdae95b-kube-api-access-n7frw\") pod \"maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps\" (UID: \"1faea580-200c-4d2e-8ead-ef3d6cdae95b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps" Oct 31 13:07:43.543006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:43.541953 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:43.607005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:43.605966 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7frw\" (UniqueName: \"kubernetes.io/projected/1faea580-200c-4d2e-8ead-ef3d6cdae95b-kube-api-access-n7frw\") pod \"maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps\" (UID: \"1faea580-200c-4d2e-8ead-ef3d6cdae95b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps" Oct 31 13:07:43.607005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:43.606121 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1faea580-200c-4d2e-8ead-ef3d6cdae95b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps\" (UID: \"1faea580-200c-4d2e-8ead-ef3d6cdae95b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps" Oct 31 13:07:43.607005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:43.606218 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1faea580-200c-4d2e-8ead-ef3d6cdae95b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps\" (UID: \"1faea580-200c-4d2e-8ead-ef3d6cdae95b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps" Oct 31 13:07:43.607005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:43.606287 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1faea580-200c-4d2e-8ead-ef3d6cdae95b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps\" (UID: \"1faea580-200c-4d2e-8ead-ef3d6cdae95b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps" Oct 31 13:07:43.607385 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:43.607282 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1faea580-200c-4d2e-8ead-ef3d6cdae95b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps\" (UID: \"1faea580-200c-4d2e-8ead-ef3d6cdae95b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps" Oct 31 13:07:43.619015 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:07:43.608108 6841 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4574ec-serving-cert: secret "maas-api-e2e-shared-a-w3-4574ec-serving-cert" not found Oct 31 13:07:43.619015 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:07:43.608186 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1faea580-200c-4d2e-8ead-ef3d6cdae95b-maas-api-tls podName:1faea580-200c-4d2e-8ead-ef3d6cdae95b nodeName:}" failed. No retries permitted until 2026-10-31 13:07:44.108165567 +0000 UTC m=+2110.591288453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1faea580-200c-4d2e-8ead-ef3d6cdae95b-maas-api-tls") pod "maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps" (UID: "1faea580-200c-4d2e-8ead-ef3d6cdae95b") : secret "maas-api-e2e-shared-a-w3-4574ec-serving-cert" not found Oct 31 13:07:43.640013 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:43.631639 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1faea580-200c-4d2e-8ead-ef3d6cdae95b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps\" (UID: \"1faea580-200c-4d2e-8ead-ef3d6cdae95b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps" Oct 31 13:07:43.647256 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:43.644749 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7frw\" (UniqueName: \"kubernetes.io/projected/1faea580-200c-4d2e-8ead-ef3d6cdae95b-kube-api-access-n7frw\") pod \"maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps\" (UID: \"1faea580-200c-4d2e-8ead-ef3d6cdae95b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps" Oct 31 13:07:43.688060 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:43.687100 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 31 13:07:43.688060 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:43.687175 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" podUID="d6fdf066-cd83-408d-9c1c-6c97f9189f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 31 13:07:43.789663 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:43.789630 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:43.992007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:43.991017 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 31 13:07:43.992007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:43.991078 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" podUID="88130f73-b9a3-4bdb-aea9-8557803120cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 31 13:07:44.009253 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:44.009157 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:44.185646 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:44.135149 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1faea580-200c-4d2e-8ead-ef3d6cdae95b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps\" (UID: \"1faea580-200c-4d2e-8ead-ef3d6cdae95b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps" Oct 31 13:07:44.348689 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:44.348600 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1faea580-200c-4d2e-8ead-ef3d6cdae95b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps\" (UID: \"1faea580-200c-4d2e-8ead-ef3d6cdae95b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps" Oct 31 13:07:44.349185 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:44.349158 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps" Oct 31 13:07:44.439087 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:44.437998 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:44.684072 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:44.674515 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:44.700739 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:44.700298 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 31 13:07:44.700739 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:44.700352 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" podUID="d6fdf066-cd83-408d-9c1c-6c97f9189f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 31 13:07:44.706821 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:44.706097 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-7cdec6-5c4d8b7565-4lv4n" Oct 31 13:07:44.802491 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:44.801656 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c" Oct 31 13:07:44.810895 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:44.810509 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn" Oct 31 13:07:44.935675 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:44.935650 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps"] Oct 31 13:07:44.936345 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:44.936312 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps" event={"ID":"1faea580-200c-4d2e-8ead-ef3d6cdae95b","Type":"ContainerStarted","Data":"70c557c860b5a828771cbda805de72861f0be609abb485de4e927efaa84b2c45"} Oct 31 13:07:45.014054 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:44.995475 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 31 13:07:45.014054 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:44.995538 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" podUID="88130f73-b9a3-4bdb-aea9-8557803120cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 31 13:07:45.379005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:45.378024 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:45.685073 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:45.684512 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 31 13:07:45.685073 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:45.684572 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" podUID="d6fdf066-cd83-408d-9c1c-6c97f9189f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 31 13:07:45.740007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:45.739181 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-45bcad-6f4486b69f-qtdsk" Oct 31 13:07:46.025005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:46.012756 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 31 13:07:46.025005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:46.012820 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" podUID="88130f73-b9a3-4bdb-aea9-8557803120cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 31 13:07:46.043936 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:46.042107 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps" event={"ID":"1faea580-200c-4d2e-8ead-ef3d6cdae95b","Type":"ContainerStarted","Data":"6b6f47a8c3f664651cafb448332fe42267e92a688534bd170e85fb36243195d4"} Oct 31 13:07:46.043936 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:46.043122 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps" Oct 31 13:07:46.694456 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:46.694417 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 31 13:07:46.695157 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:46.695120 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" podUID="d6fdf066-cd83-408d-9c1c-6c97f9189f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 31 13:07:46.998047 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:46.997938 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 31 13:07:46.998340 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:46.998310 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" podUID="88130f73-b9a3-4bdb-aea9-8557803120cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 31 13:07:47.311616 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:47.311519 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:47.700260 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:47.684466 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 31 13:07:47.700260 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:47.684532 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" podUID="d6fdf066-cd83-408d-9c1c-6c97f9189f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 31 13:07:47.988056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:47.987428 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 31 13:07:47.988056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:47.987490 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" podUID="88130f73-b9a3-4bdb-aea9-8557803120cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 31 13:07:48.684584 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:48.684545 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 31 13:07:48.684871 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:48.684838 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" podUID="d6fdf066-cd83-408d-9c1c-6c97f9189f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 31 13:07:48.868022 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:48.860275 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:48.998797 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:48.998348 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 31 13:07:48.998797 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:48.998409 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" podUID="88130f73-b9a3-4bdb-aea9-8557803120cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 31 13:07:49.732691 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:49.732652 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:49.963791 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:49.963753 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:50.054938 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:07:50.054849 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:07:50.172610 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.172572 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:50.206291 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.206256 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" Oct 31 13:07:50.207615 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.207594 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" Oct 31 13:07:50.208916 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.208848 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" Oct 31 13:07:50.241174 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.241121 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps" podStartSLOduration=6.66865054 podStartE2EDuration="7.241104708s" totalImagesPullingTime="572.454168ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:07:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:07:46.101936439 +0000 UTC m=+2112.585059344" watchObservedRunningTime="2026-10-31 13:07:50.241104708 +0000 UTC m=+2116.724227602" Oct 31 13:07:50.399244 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.399197 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:50.474735 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.474432 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk"] Oct 31 13:07:50.481851 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.481826 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:07:50.486153 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.486127 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-6106a3-openshift-default-dockercfg-gvbfm\"" Oct 31 13:07:50.510595 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.510556 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk"] Oct 31 13:07:50.557166 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.556339 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" Oct 31 13:07:50.558452 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.558426 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" Oct 31 13:07:50.560448 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.560426 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" Oct 31 13:07:50.576397 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.576357 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/913b94e9-bc62-42b9-ad68-2fab683a9593-istiod-ca-cert\") pod \"e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk\" (UID: \"913b94e9-bc62-42b9-ad68-2fab683a9593\") " pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:07:50.576554 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.576502 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/913b94e9-bc62-42b9-ad68-2fab683a9593-istio-data\") pod \"e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk\" (UID: \"913b94e9-bc62-42b9-ad68-2fab683a9593\") " pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:07:50.576626 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.576566 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/913b94e9-bc62-42b9-ad68-2fab683a9593-workload-certs\") pod \"e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk\" (UID: \"913b94e9-bc62-42b9-ad68-2fab683a9593\") " pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:07:50.576723 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.576699 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlktc\" (UniqueName: \"kubernetes.io/projected/913b94e9-bc62-42b9-ad68-2fab683a9593-kube-api-access-qlktc\") pod \"e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk\" (UID: \"913b94e9-bc62-42b9-ad68-2fab683a9593\") " pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:07:50.576796 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.576772 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/913b94e9-bc62-42b9-ad68-2fab683a9593-istio-envoy\") pod \"e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk\" (UID: \"913b94e9-bc62-42b9-ad68-2fab683a9593\") " pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:07:50.576956 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.576936 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/913b94e9-bc62-42b9-ad68-2fab683a9593-credential-socket\") pod \"e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk\" (UID: \"913b94e9-bc62-42b9-ad68-2fab683a9593\") " pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:07:50.577065 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.576976 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/913b94e9-bc62-42b9-ad68-2fab683a9593-workload-socket\") pod \"e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk\" (UID: \"913b94e9-bc62-42b9-ad68-2fab683a9593\") " pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:07:50.577126 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.577115 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/913b94e9-bc62-42b9-ad68-2fab683a9593-istio-token\") pod \"e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk\" (UID: \"913b94e9-bc62-42b9-ad68-2fab683a9593\") " pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:07:50.577183 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.577142 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/913b94e9-bc62-42b9-ad68-2fab683a9593-istio-podinfo\") pod \"e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk\" (UID: \"913b94e9-bc62-42b9-ad68-2fab683a9593\") " pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:07:50.657060 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.654699 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:50.698012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.682843 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/913b94e9-bc62-42b9-ad68-2fab683a9593-istio-token\") pod \"e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk\" (UID: \"913b94e9-bc62-42b9-ad68-2fab683a9593\") " pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:07:50.698012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.682904 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/913b94e9-bc62-42b9-ad68-2fab683a9593-istio-podinfo\") pod \"e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk\" (UID: \"913b94e9-bc62-42b9-ad68-2fab683a9593\") " pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:07:50.698012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.682968 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/913b94e9-bc62-42b9-ad68-2fab683a9593-istiod-ca-cert\") pod \"e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk\" (UID: \"913b94e9-bc62-42b9-ad68-2fab683a9593\") " pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:07:50.698012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.683101 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/913b94e9-bc62-42b9-ad68-2fab683a9593-istio-data\") pod \"e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk\" (UID: \"913b94e9-bc62-42b9-ad68-2fab683a9593\") " pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:07:50.698012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.683172 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/913b94e9-bc62-42b9-ad68-2fab683a9593-workload-certs\") pod \"e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk\" (UID: \"913b94e9-bc62-42b9-ad68-2fab683a9593\") " pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:07:50.698012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.683245 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlktc\" (UniqueName: \"kubernetes.io/projected/913b94e9-bc62-42b9-ad68-2fab683a9593-kube-api-access-qlktc\") pod \"e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk\" (UID: \"913b94e9-bc62-42b9-ad68-2fab683a9593\") " pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:07:50.698012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.683288 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/913b94e9-bc62-42b9-ad68-2fab683a9593-istio-envoy\") pod \"e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk\" (UID: \"913b94e9-bc62-42b9-ad68-2fab683a9593\") " pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:07:50.698012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.683464 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/913b94e9-bc62-42b9-ad68-2fab683a9593-credential-socket\") pod \"e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk\" (UID: \"913b94e9-bc62-42b9-ad68-2fab683a9593\") " pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:07:50.698012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.683511 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/913b94e9-bc62-42b9-ad68-2fab683a9593-workload-socket\") pod \"e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk\" (UID: \"913b94e9-bc62-42b9-ad68-2fab683a9593\") " pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:07:50.705789 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.698968 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/913b94e9-bc62-42b9-ad68-2fab683a9593-credential-socket\") pod \"e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk\" (UID: \"913b94e9-bc62-42b9-ad68-2fab683a9593\") " pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:07:50.705789 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.700720 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/913b94e9-bc62-42b9-ad68-2fab683a9593-istiod-ca-cert\") pod \"e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk\" (UID: \"913b94e9-bc62-42b9-ad68-2fab683a9593\") " pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:07:50.705789 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.701093 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/913b94e9-bc62-42b9-ad68-2fab683a9593-istio-data\") pod \"e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk\" (UID: \"913b94e9-bc62-42b9-ad68-2fab683a9593\") " pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:07:50.705789 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.701388 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/913b94e9-bc62-42b9-ad68-2fab683a9593-workload-certs\") pod \"e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk\" (UID: \"913b94e9-bc62-42b9-ad68-2fab683a9593\") " pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:07:50.705789 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.701625 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/913b94e9-bc62-42b9-ad68-2fab683a9593-workload-socket\") pod \"e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk\" (UID: \"913b94e9-bc62-42b9-ad68-2fab683a9593\") " pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:07:50.762007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.739506 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/913b94e9-bc62-42b9-ad68-2fab683a9593-istio-envoy\") pod \"e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk\" (UID: \"913b94e9-bc62-42b9-ad68-2fab683a9593\") " pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:07:50.762007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.738590 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/913b94e9-bc62-42b9-ad68-2fab683a9593-istio-podinfo\") pod \"e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk\" (UID: \"913b94e9-bc62-42b9-ad68-2fab683a9593\") " pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:07:50.911726 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.911446 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:50.967004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.962767 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlktc\" (UniqueName: \"kubernetes.io/projected/913b94e9-bc62-42b9-ad68-2fab683a9593-kube-api-access-qlktc\") pod \"e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk\" (UID: \"913b94e9-bc62-42b9-ad68-2fab683a9593\") " pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:07:50.967004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:50.963246 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/913b94e9-bc62-42b9-ad68-2fab683a9593-istio-token\") pod \"e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk\" (UID: \"913b94e9-bc62-42b9-ad68-2fab683a9593\") " pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:07:51.171136 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:51.159569 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:07:51.378145 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:51.357455 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:51.630792 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:51.622909 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:51.970547 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:51.970503 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:51.991817 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:51.991777 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk"] Oct 31 13:07:52.033689 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:52.033639 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:07:52.034483 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:52.034452 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:07:52.034727 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:52.034701 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:07:52.316024 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:52.315844 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:52.590169 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:52.590071 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" event={"ID":"913b94e9-bc62-42b9-ad68-2fab683a9593","Type":"ContainerStarted","Data":"a6a0b04c2bd005394a9dd11838d16ed7be93d59a2e75836d9a9ac3a8d690f495"} Oct 31 13:07:52.590511 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:52.590489 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" event={"ID":"913b94e9-bc62-42b9-ad68-2fab683a9593","Type":"ContainerStarted","Data":"ef22627763281e4ab8cd1a60ef92b0cbc1bc81e3147e4c9f7a634de8fc315bfc"} Oct 31 13:07:52.667007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:52.659926 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" podStartSLOduration=2.6599050650000002 podStartE2EDuration="2.659905065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:07:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:07:52.644715468 +0000 UTC m=+2119.127838373" watchObservedRunningTime="2026-10-31 13:07:52.659905065 +0000 UTC m=+2119.143027967" Oct 31 13:07:52.928051 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:52.927916 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:53.167004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:53.166481 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:07:53.177579 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:53.177540 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:07:53.496538 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:53.496511 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps" Oct 31 13:07:53.596382 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:53.596354 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:07:53.597236 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:53.597213 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:07:53.708816 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:53.708745 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:54.772343 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:54.772299 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:55.046772 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:55.046674 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:55.277330 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:55.277300 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:55.516571 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:55.516532 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:55.778692 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:55.778596 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:56.266418 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:56.266375 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:56.821773 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:56.821733 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:57.106693 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:57.106591 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:57.320833 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:57.320798 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:57.536221 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:57.536151 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:57.739371 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:57.739322 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:57.867484 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:57.867403 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df"] Oct 31 13:07:57.873879 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:57.873848 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:07:57.884737 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:57.884711 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-569075-openshift-default-dockercfg-lwqkn\"" Oct 31 13:07:57.889593 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:57.889565 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df"] Oct 31 13:07:58.003180 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:58.003134 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4e95366-875b-4fda-8e1b-6e441df2f5b2-istio-data\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df\" (UID: \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:07:58.003602 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:58.003571 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8t5z\" (UniqueName: \"kubernetes.io/projected/e4e95366-875b-4fda-8e1b-6e441df2f5b2-kube-api-access-n8t5z\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df\" (UID: \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:07:58.003775 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:58.003750 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4e95366-875b-4fda-8e1b-6e441df2f5b2-istiod-ca-cert\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df\" (UID: \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:07:58.003923 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:58.003819 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4e95366-875b-4fda-8e1b-6e441df2f5b2-istio-podinfo\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df\" (UID: \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:07:58.003923 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:58.003868 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e95366-875b-4fda-8e1b-6e441df2f5b2-credential-socket\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df\" (UID: \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:07:58.004086 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:58.004022 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4e95366-875b-4fda-8e1b-6e441df2f5b2-istio-envoy\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df\" (UID: \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:07:58.004153 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:58.004081 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e95366-875b-4fda-8e1b-6e441df2f5b2-workload-certs\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df\" (UID: \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:07:58.004153 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:58.004114 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e95366-875b-4fda-8e1b-6e441df2f5b2-workload-socket\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df\" (UID: \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:07:58.004253 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:58.004232 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4e95366-875b-4fda-8e1b-6e441df2f5b2-istio-token\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df\" (UID: \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:07:58.032665 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:58.032631 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:58.105660 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:58.105626 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8t5z\" (UniqueName: \"kubernetes.io/projected/e4e95366-875b-4fda-8e1b-6e441df2f5b2-kube-api-access-n8t5z\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df\" (UID: \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:07:58.105660 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:58.105663 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4e95366-875b-4fda-8e1b-6e441df2f5b2-istiod-ca-cert\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df\" (UID: \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:07:58.105967 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:58.105942 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4e95366-875b-4fda-8e1b-6e441df2f5b2-istio-podinfo\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df\" (UID: \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:07:58.106057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:58.106047 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e95366-875b-4fda-8e1b-6e441df2f5b2-credential-socket\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df\" (UID: \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:07:58.106136 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:58.106118 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4e95366-875b-4fda-8e1b-6e441df2f5b2-istio-envoy\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df\" (UID: \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:07:58.106216 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:58.106160 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e95366-875b-4fda-8e1b-6e441df2f5b2-workload-certs\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df\" (UID: \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:07:58.106216 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:58.106184 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e95366-875b-4fda-8e1b-6e441df2f5b2-workload-socket\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df\" (UID: \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:07:58.106323 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:58.106273 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4e95366-875b-4fda-8e1b-6e441df2f5b2-istio-token\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df\" (UID: \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:07:58.106323 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:58.106297 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4e95366-875b-4fda-8e1b-6e441df2f5b2-istio-data\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df\" (UID: \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:07:58.106489 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:58.106462 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4e95366-875b-4fda-8e1b-6e441df2f5b2-istiod-ca-cert\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df\" (UID: \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:07:58.106727 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:58.106703 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4e95366-875b-4fda-8e1b-6e441df2f5b2-istio-data\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df\" (UID: \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:07:58.106856 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:58.106731 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e95366-875b-4fda-8e1b-6e441df2f5b2-workload-certs\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df\" (UID: \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:07:58.106938 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:58.106885 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e95366-875b-4fda-8e1b-6e441df2f5b2-credential-socket\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df\" (UID: \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:07:58.107285 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:58.107263 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e95366-875b-4fda-8e1b-6e441df2f5b2-workload-socket\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df\" (UID: \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:07:58.109338 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:58.109310 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4e95366-875b-4fda-8e1b-6e441df2f5b2-istio-envoy\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df\" (UID: \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:07:58.109765 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:58.109738 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4e95366-875b-4fda-8e1b-6e441df2f5b2-istio-podinfo\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df\" (UID: \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:07:58.132452 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:58.132375 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4e95366-875b-4fda-8e1b-6e441df2f5b2-istio-token\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df\" (UID: \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:07:58.136446 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:58.134687 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8t5z\" (UniqueName: \"kubernetes.io/projected/e4e95366-875b-4fda-8e1b-6e441df2f5b2-kube-api-access-n8t5z\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df\" (UID: \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:07:58.190919 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:58.190888 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:07:58.252015 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:58.250962 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:58.655518 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:58.655480 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:58.765865 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:07:58.765816 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e95366_875b_4fda_8e1b_6e441df2f5b2.slice/crio-98673cac34ee61d6c099d9d31a70ac0a53159c36a9796da74a3aeabbabcc40e1 WatchSource:0}: Error finding container 98673cac34ee61d6c099d9d31a70ac0a53159c36a9796da74a3aeabbabcc40e1: Status 404 returned error can't find the container with id 98673cac34ee61d6c099d9d31a70ac0a53159c36a9796da74a3aeabbabcc40e1 Oct 31 13:07:58.766561 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:58.766536 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df"] Oct 31 13:07:58.769925 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:58.769892 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:07:58.770053 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:58.769958 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:07:58.770053 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:58.770027 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:07:58.914381 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:58.914282 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:59.548521 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:59.548487 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:07:59.650135 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:59.650096 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" event={"ID":"e4e95366-875b-4fda-8e1b-6e441df2f5b2","Type":"ContainerStarted","Data":"292291e58a2f3d3a1b8136feffc27ea8b3fc67d29f274879803b92e63bd231b4"} Oct 31 13:07:59.650284 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:59.650151 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" event={"ID":"e4e95366-875b-4fda-8e1b-6e441df2f5b2","Type":"ContainerStarted","Data":"98673cac34ee61d6c099d9d31a70ac0a53159c36a9796da74a3aeabbabcc40e1"} Oct 31 13:07:59.702011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:07:59.695559 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" podStartSLOduration=2.695535997 podStartE2EDuration="2.695535997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:07:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:07:59.680687595 +0000 UTC m=+2126.163810520" watchObservedRunningTime="2026-10-31 13:07:59.695535997 +0000 UTC m=+2126.178658900" Oct 31 13:08:00.210010 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:00.206251 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:08:00.227640 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:00.227606 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:08:00.382697 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:00.382660 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:00.658106 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:00.658025 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:08:00.658969 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:00.658947 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:08:01.312488 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:01.309706 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:01.518483 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:01.518435 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:02.081221 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:02.081183 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:02.370244 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:02.370156 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:02.664087 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:02.659601 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:03.358311 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:03.356704 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:03.599561 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:03.599523 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:03.879941 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:03.877269 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:04.316124 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:04.316073 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9"] Oct 31 13:08:04.330623 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:04.330594 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9" Oct 31 13:08:04.344219 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:04.344166 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a4747b7-ff5a-4c37-b3f9-5db97fce5d12-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9\" (UID: \"3a4747b7-ff5a-4c37-b3f9-5db97fce5d12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9" Oct 31 13:08:04.344549 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:04.344492 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a4747b7-ff5a-4c37-b3f9-5db97fce5d12-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9\" (UID: \"3a4747b7-ff5a-4c37-b3f9-5db97fce5d12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9" Oct 31 13:08:04.344788 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:04.344759 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a4747b7-ff5a-4c37-b3f9-5db97fce5d12-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9\" (UID: \"3a4747b7-ff5a-4c37-b3f9-5db97fce5d12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9" Oct 31 13:08:04.344908 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:04.344822 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6679b\" (UniqueName: \"kubernetes.io/projected/3a4747b7-ff5a-4c37-b3f9-5db97fce5d12-kube-api-access-6679b\") pod \"maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9\" (UID: \"3a4747b7-ff5a-4c37-b3f9-5db97fce5d12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9" Oct 31 13:08:04.350472 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:04.350300 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9"] Oct 31 13:08:04.354162 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:04.354137 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-377f60-serving-cert\"" Oct 31 13:08:04.473043 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:04.468409 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a4747b7-ff5a-4c37-b3f9-5db97fce5d12-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9\" (UID: \"3a4747b7-ff5a-4c37-b3f9-5db97fce5d12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9" Oct 31 13:08:04.473043 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:04.468487 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6679b\" (UniqueName: \"kubernetes.io/projected/3a4747b7-ff5a-4c37-b3f9-5db97fce5d12-kube-api-access-6679b\") pod \"maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9\" (UID: \"3a4747b7-ff5a-4c37-b3f9-5db97fce5d12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9" Oct 31 13:08:04.473043 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:04.468735 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a4747b7-ff5a-4c37-b3f9-5db97fce5d12-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9\" (UID: \"3a4747b7-ff5a-4c37-b3f9-5db97fce5d12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9" Oct 31 13:08:04.473043 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:04.468803 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a4747b7-ff5a-4c37-b3f9-5db97fce5d12-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9\" (UID: \"3a4747b7-ff5a-4c37-b3f9-5db97fce5d12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9" Oct 31 13:08:04.483045 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:04.474744 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a4747b7-ff5a-4c37-b3f9-5db97fce5d12-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9\" (UID: \"3a4747b7-ff5a-4c37-b3f9-5db97fce5d12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9" Oct 31 13:08:04.483383 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:04.478251 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a4747b7-ff5a-4c37-b3f9-5db97fce5d12-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9\" (UID: \"3a4747b7-ff5a-4c37-b3f9-5db97fce5d12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9" Oct 31 13:08:04.502171 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:08:04.478701 6841 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-377f60-serving-cert: secret "maas-api-e2e-shared-b-w0-377f60-serving-cert" not found Oct 31 13:08:04.502552 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:08:04.502521 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a4747b7-ff5a-4c37-b3f9-5db97fce5d12-maas-api-tls podName:3a4747b7-ff5a-4c37-b3f9-5db97fce5d12 nodeName:}" failed. No retries permitted until 2026-10-31 13:08:05.002488473 +0000 UTC m=+2131.485611365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3a4747b7-ff5a-4c37-b3f9-5db97fce5d12-maas-api-tls") pod "maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9" (UID: "3a4747b7-ff5a-4c37-b3f9-5db97fce5d12") : secret "maas-api-e2e-shared-b-w0-377f60-serving-cert" not found Oct 31 13:08:04.502724 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:04.492933 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6679b\" (UniqueName: \"kubernetes.io/projected/3a4747b7-ff5a-4c37-b3f9-5db97fce5d12-kube-api-access-6679b\") pod \"maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9\" (UID: \"3a4747b7-ff5a-4c37-b3f9-5db97fce5d12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9" Oct 31 13:08:04.510640 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:04.510603 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:04.769353 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:04.769322 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:05.079145 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:05.079050 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a4747b7-ff5a-4c37-b3f9-5db97fce5d12-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9\" (UID: \"3a4747b7-ff5a-4c37-b3f9-5db97fce5d12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9" Oct 31 13:08:05.082736 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:05.082706 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a4747b7-ff5a-4c37-b3f9-5db97fce5d12-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9\" (UID: \"3a4747b7-ff5a-4c37-b3f9-5db97fce5d12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9" Oct 31 13:08:05.261224 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:05.261190 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9" Oct 31 13:08:05.551689 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:05.551660 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:05.566622 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:05.566595 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9"] Oct 31 13:08:05.567304 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:08:05.567252 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a4747b7_ff5a_4c37_b3f9_5db97fce5d12.slice/crio-13e45277c82cc2eb4d8c14f60cb2cbb4b085a864c7eb264cd69a88eb6bbb6281 WatchSource:0}: Error finding container 13e45277c82cc2eb4d8c14f60cb2cbb4b085a864c7eb264cd69a88eb6bbb6281: Status 404 returned error can't find the container with id 13e45277c82cc2eb4d8c14f60cb2cbb4b085a864c7eb264cd69a88eb6bbb6281 Oct 31 13:08:05.568912 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:05.568891 6841 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 13:08:05.802809 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:05.802726 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9" event={"ID":"3a4747b7-ff5a-4c37-b3f9-5db97fce5d12","Type":"ContainerStarted","Data":"13e45277c82cc2eb4d8c14f60cb2cbb4b085a864c7eb264cd69a88eb6bbb6281"} Oct 31 13:08:06.500422 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:06.498806 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:06.743726 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:06.743687 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:06.812398 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:06.812314 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9" event={"ID":"3a4747b7-ff5a-4c37-b3f9-5db97fce5d12","Type":"ContainerStarted","Data":"92fe698505839cf89b84aea619f132e229e1109998ea790c9b94669e298f0445"} Oct 31 13:08:06.812761 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:06.812401 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9" Oct 31 13:08:06.839408 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:06.839361 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9" podStartSLOduration=2.3881508289999998 podStartE2EDuration="2.839348743s" totalImagesPullingTime="451.197914ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:08:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:08:06.835243865 +0000 UTC m=+2133.318366766" watchObservedRunningTime="2026-10-31 13:08:06.839348743 +0000 UTC m=+2133.322471643" Oct 31 13:08:07.597871 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:07.597827 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf"] Oct 31 13:08:07.604225 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:07.604197 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf" Oct 31 13:08:07.618944 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:07.618875 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf"] Oct 31 13:08:07.660686 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:07.660659 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-5c24e1-serving-cert\"" Oct 31 13:08:07.721936 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:07.721898 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/202fa689-242a-4b4d-8ed6-d3992b0883ff-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf\" (UID: \"202fa689-242a-4b4d-8ed6-d3992b0883ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf" Oct 31 13:08:07.722175 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:07.722152 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l8dw\" (UniqueName: \"kubernetes.io/projected/202fa689-242a-4b4d-8ed6-d3992b0883ff-kube-api-access-8l8dw\") pod \"maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf\" (UID: \"202fa689-242a-4b4d-8ed6-d3992b0883ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf" Oct 31 13:08:07.722260 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:07.722245 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/202fa689-242a-4b4d-8ed6-d3992b0883ff-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf\" (UID: \"202fa689-242a-4b4d-8ed6-d3992b0883ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf" Oct 31 13:08:07.722306 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:07.722283 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/202fa689-242a-4b4d-8ed6-d3992b0883ff-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf\" (UID: \"202fa689-242a-4b4d-8ed6-d3992b0883ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf" Oct 31 13:08:07.766416 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:07.766384 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:07.831005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:07.827075 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l8dw\" (UniqueName: \"kubernetes.io/projected/202fa689-242a-4b4d-8ed6-d3992b0883ff-kube-api-access-8l8dw\") pod \"maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf\" (UID: \"202fa689-242a-4b4d-8ed6-d3992b0883ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf" Oct 31 13:08:07.831005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:07.827170 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/202fa689-242a-4b4d-8ed6-d3992b0883ff-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf\" (UID: \"202fa689-242a-4b4d-8ed6-d3992b0883ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf" Oct 31 13:08:07.879290 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:07.841812 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/202fa689-242a-4b4d-8ed6-d3992b0883ff-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf\" (UID: \"202fa689-242a-4b4d-8ed6-d3992b0883ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf" Oct 31 13:08:07.879290 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:07.841912 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/202fa689-242a-4b4d-8ed6-d3992b0883ff-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf\" (UID: \"202fa689-242a-4b4d-8ed6-d3992b0883ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf" Oct 31 13:08:07.879290 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:07.842161 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/202fa689-242a-4b4d-8ed6-d3992b0883ff-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf\" (UID: \"202fa689-242a-4b4d-8ed6-d3992b0883ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf" Oct 31 13:08:07.879290 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:07.846888 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/202fa689-242a-4b4d-8ed6-d3992b0883ff-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf\" (UID: \"202fa689-242a-4b4d-8ed6-d3992b0883ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf" Oct 31 13:08:07.890814 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:07.890783 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/202fa689-242a-4b4d-8ed6-d3992b0883ff-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf\" (UID: \"202fa689-242a-4b4d-8ed6-d3992b0883ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf" Oct 31 13:08:07.920898 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:07.920804 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l8dw\" (UniqueName: \"kubernetes.io/projected/202fa689-242a-4b4d-8ed6-d3992b0883ff-kube-api-access-8l8dw\") pod \"maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf\" (UID: \"202fa689-242a-4b4d-8ed6-d3992b0883ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf" Oct 31 13:08:07.931237 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:07.931206 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf" Oct 31 13:08:08.163275 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:08.131524 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:08.444348 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:08.444319 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf"] Oct 31 13:08:08.446433 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:08:08.446400 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod202fa689_242a_4b4d_8ed6_d3992b0883ff.slice/crio-e58ab4c6434d0ab3cbddb5b537a3b5e04c9f2c19f5a5b57297f6eba59244bc3b WatchSource:0}: Error finding container e58ab4c6434d0ab3cbddb5b537a3b5e04c9f2c19f5a5b57297f6eba59244bc3b: Status 404 returned error can't find the container with id e58ab4c6434d0ab3cbddb5b537a3b5e04c9f2c19f5a5b57297f6eba59244bc3b Oct 31 13:08:08.567956 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:08.567883 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:08.809962 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:08.809878 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:08.941647 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:08.941595 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf" event={"ID":"202fa689-242a-4b4d-8ed6-d3992b0883ff","Type":"ContainerStarted","Data":"e58ab4c6434d0ab3cbddb5b537a3b5e04c9f2c19f5a5b57297f6eba59244bc3b"} Oct 31 13:08:09.182037 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:09.179590 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:10.016511 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:10.015914 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf" event={"ID":"202fa689-242a-4b4d-8ed6-d3992b0883ff","Type":"ContainerStarted","Data":"f88dfae730a90c1d0d269b46ae0631a2adac34b6578bdce63abaad2d3c9239e9"} Oct 31 13:08:10.016511 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:10.016443 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf" Oct 31 13:08:10.083007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:10.082209 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf" podStartSLOduration=2.501371277 podStartE2EDuration="3.082183911s" totalImagesPullingTime="580.812634ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:08:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:08:10.03915291 +0000 UTC m=+2136.522275807" watchObservedRunningTime="2026-10-31 13:08:10.082183911 +0000 UTC m=+2136.565306842" Oct 31 13:08:10.544265 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:10.544222 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:11.432306 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:11.432257 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:12.155829 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:12.154358 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:12.937005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:12.933767 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:12.972022 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:12.969149 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9" Oct 31 13:08:13.722257 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:13.722210 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:14.585007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:14.576664 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:14.785518 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:14.785482 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:15.515446 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:15.515409 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:16.365291 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:16.365246 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf" Oct 31 13:08:16.423143 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:16.423110 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:17.227306 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:17.226790 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:18.076005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:18.072157 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:18.801762 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:18.801726 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq"] Oct 31 13:08:18.841473 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:18.841435 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq"] Oct 31 13:08:18.841784 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:18.841539 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq" Oct 31 13:08:18.847892 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:18.847872 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-5452fc-serving-cert\"" Oct 31 13:08:18.956745 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:18.956708 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:18.958436 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:18.957532 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a2efb2-e354-4258-811e-d9ec46ef48a0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq\" (UID: \"35a2efb2-e354-4258-811e-d9ec46ef48a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq" Oct 31 13:08:18.958436 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:18.957975 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlmxl\" (UniqueName: \"kubernetes.io/projected/35a2efb2-e354-4258-811e-d9ec46ef48a0-kube-api-access-dlmxl\") pod \"maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq\" (UID: \"35a2efb2-e354-4258-811e-d9ec46ef48a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq" Oct 31 13:08:18.958436 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:18.958060 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35a2efb2-e354-4258-811e-d9ec46ef48a0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq\" (UID: \"35a2efb2-e354-4258-811e-d9ec46ef48a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq" Oct 31 13:08:18.958436 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:18.958139 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35a2efb2-e354-4258-811e-d9ec46ef48a0-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq\" (UID: \"35a2efb2-e354-4258-811e-d9ec46ef48a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq" Oct 31 13:08:19.059873 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:19.059777 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlmxl\" (UniqueName: \"kubernetes.io/projected/35a2efb2-e354-4258-811e-d9ec46ef48a0-kube-api-access-dlmxl\") pod \"maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq\" (UID: \"35a2efb2-e354-4258-811e-d9ec46ef48a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq" Oct 31 13:08:19.059873 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:19.059847 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35a2efb2-e354-4258-811e-d9ec46ef48a0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq\" (UID: \"35a2efb2-e354-4258-811e-d9ec46ef48a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq" Oct 31 13:08:19.060129 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:19.059915 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35a2efb2-e354-4258-811e-d9ec46ef48a0-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq\" (UID: \"35a2efb2-e354-4258-811e-d9ec46ef48a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq" Oct 31 13:08:19.060129 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:08:19.059956 6841 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5452fc-serving-cert: secret "maas-api-e2e-shared-b-w3-5452fc-serving-cert" not found Oct 31 13:08:19.060129 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:19.060004 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a2efb2-e354-4258-811e-d9ec46ef48a0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq\" (UID: \"35a2efb2-e354-4258-811e-d9ec46ef48a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq" Oct 31 13:08:19.060129 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:08:19.060095 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35a2efb2-e354-4258-811e-d9ec46ef48a0-maas-api-tls podName:35a2efb2-e354-4258-811e-d9ec46ef48a0 nodeName:}" failed. No retries permitted until 2026-10-31 13:08:19.56005132 +0000 UTC m=+2146.043174221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/35a2efb2-e354-4258-811e-d9ec46ef48a0-maas-api-tls") pod "maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq" (UID: "35a2efb2-e354-4258-811e-d9ec46ef48a0") : secret "maas-api-e2e-shared-b-w3-5452fc-serving-cert" not found Oct 31 13:08:19.060737 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:19.060704 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a2efb2-e354-4258-811e-d9ec46ef48a0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq\" (UID: \"35a2efb2-e354-4258-811e-d9ec46ef48a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq" Oct 31 13:08:19.063398 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:19.063375 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35a2efb2-e354-4258-811e-d9ec46ef48a0-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq\" (UID: \"35a2efb2-e354-4258-811e-d9ec46ef48a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq" Oct 31 13:08:19.073184 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:19.073150 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlmxl\" (UniqueName: \"kubernetes.io/projected/35a2efb2-e354-4258-811e-d9ec46ef48a0-kube-api-access-dlmxl\") pod \"maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq\" (UID: \"35a2efb2-e354-4258-811e-d9ec46ef48a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq" Oct 31 13:08:19.195220 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:19.195180 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:19.407839 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:19.407759 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:19.575084 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:19.568658 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35a2efb2-e354-4258-811e-d9ec46ef48a0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq\" (UID: \"35a2efb2-e354-4258-811e-d9ec46ef48a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq" Oct 31 13:08:19.596374 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:19.595880 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35a2efb2-e354-4258-811e-d9ec46ef48a0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq\" (UID: \"35a2efb2-e354-4258-811e-d9ec46ef48a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq" Oct 31 13:08:19.760939 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:19.760906 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq" Oct 31 13:08:20.029446 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:20.029364 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:20.058496 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:20.058464 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq"] Oct 31 13:08:20.059949 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:08:20.059918 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a2efb2_e354_4258_811e_d9ec46ef48a0.slice/crio-c5ac30b5ba5f086aea6adaa9fab32322240f153bd1f2f3642adbae3a02f6dd75 WatchSource:0}: Error finding container c5ac30b5ba5f086aea6adaa9fab32322240f153bd1f2f3642adbae3a02f6dd75: Status 404 returned error can't find the container with id c5ac30b5ba5f086aea6adaa9fab32322240f153bd1f2f3642adbae3a02f6dd75 Oct 31 13:08:20.066440 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:08:20.066412 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:08:20.253181 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:20.253140 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:20.468759 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:20.468717 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq" event={"ID":"35a2efb2-e354-4258-811e-d9ec46ef48a0","Type":"ContainerStarted","Data":"c5ac30b5ba5f086aea6adaa9fab32322240f153bd1f2f3642adbae3a02f6dd75"} Oct 31 13:08:21.098071 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:21.096423 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:21.477874 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:21.477844 6841 generic.go:356] "Generic (PLEG): container finished" podID="32580ef3-c0ea-46bc-8b62-c2b1362e998c" containerID="b3f2b12f6db769d6cda6d6e171cd458e27e99461a775f49ee91314795bf2d136" exitCode=0 Oct 31 13:08:21.478397 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:21.477920 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" event={"ID":"32580ef3-c0ea-46bc-8b62-c2b1362e998c","Type":"ContainerDied","Data":"b3f2b12f6db769d6cda6d6e171cd458e27e99461a775f49ee91314795bf2d136"} Oct 31 13:08:21.478507 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:21.478488 6841 scope.go:122] "RemoveContainer" containerID="b3f2b12f6db769d6cda6d6e171cd458e27e99461a775f49ee91314795bf2d136" Oct 31 13:08:21.480024 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:21.479964 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq" event={"ID":"35a2efb2-e354-4258-811e-d9ec46ef48a0","Type":"ContainerStarted","Data":"dd3fcaccc55b4c1fffc4e855cf45156c46dbcc03ef080913d96d56117259dedc"} Oct 31 13:08:21.480152 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:21.480074 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq" Oct 31 13:08:21.480338 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:21.480310 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:08:21.480427 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:21.480381 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:08:21.480427 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:21.480420 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:08:21.543240 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:21.543194 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq" podStartSLOduration=3.08852875 podStartE2EDuration="3.543178399s" totalImagesPullingTime="454.649649ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:08:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:08:21.542459031 +0000 UTC m=+2148.025581933" watchObservedRunningTime="2026-10-31 13:08:21.543178399 +0000 UTC m=+2148.026301300" Oct 31 13:08:22.164666 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:22.164625 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:22.463031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:22.457967 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:22.490226 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:22.490188 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" event={"ID":"32580ef3-c0ea-46bc-8b62-c2b1362e998c","Type":"ContainerStarted","Data":"858fa07c513333442e79ffe82f049ea19edccc4977872f916e9bf92d6105d76b"} Oct 31 13:08:22.535969 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:22.535935 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:08:22.536210 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:22.536148 6841 patch_prober.go:29] interesting pod/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 31 13:08:22.536266 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:22.536217 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" podUID="32580ef3-c0ea-46bc-8b62-c2b1362e998c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 31 13:08:23.321667 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:23.320060 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:23.536537 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:23.536503 6841 patch_prober.go:29] interesting pod/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 31 13:08:23.536939 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:23.536557 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" podUID="32580ef3-c0ea-46bc-8b62-c2b1362e998c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 31 13:08:24.219556 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:24.219514 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:24.536224 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:24.536140 6841 patch_prober.go:29] interesting pod/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 31 13:08:24.536224 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:24.536195 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" podUID="32580ef3-c0ea-46bc-8b62-c2b1362e998c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 31 13:08:25.034109 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:25.034073 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:25.545005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:25.537429 6841 patch_prober.go:29] interesting pod/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 31 13:08:25.545005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:25.537494 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" podUID="32580ef3-c0ea-46bc-8b62-c2b1362e998c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 31 13:08:25.567006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:25.553479 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:25.886295 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:25.886204 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:26.363008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:26.362585 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:26.535849 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:26.535811 6841 patch_prober.go:29] interesting pod/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 31 13:08:26.536085 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:26.535870 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" podUID="32580ef3-c0ea-46bc-8b62-c2b1362e998c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 31 13:08:27.291848 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:27.291821 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:27.495918 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:27.495890 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq" Oct 31 13:08:27.536671 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:27.536633 6841 patch_prober.go:29] interesting pod/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 31 13:08:27.536832 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:27.536701 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" podUID="32580ef3-c0ea-46bc-8b62-c2b1362e998c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 31 13:08:28.253081 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:28.250664 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:28.540600 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:28.540521 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:08:28.541014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:28.540856 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:08:28.541628 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:28.541606 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:08:29.253787 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:29.253751 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:30.236074 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:30.235360 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:31.216008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:31.212486 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:32.534935 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:32.534902 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:32.793423 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:32.793312 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:33.845750 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:33.845706 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:34.783052 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:34.782932 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:35.769006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:35.764558 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:36.782805 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:36.782723 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:36.924631 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:36.924599 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh"] Oct 31 13:08:36.933259 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:36.932321 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh"] Oct 31 13:08:36.933259 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:36.932424 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh" Oct 31 13:08:36.938050 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:36.937429 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-6106a3\"/\"openshift-service-ca.crt\"" Oct 31 13:08:36.939362 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:36.939323 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-6106a3\"/\"default-dockercfg-lmxvd\"" Oct 31 13:08:36.939715 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:36.939696 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-6106a3\"/\"explicit-model-6106a3-kserve-self-signed-certs\"" Oct 31 13:08:36.940020 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:36.940003 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-6106a3\"/\"kube-root-ca.crt\"" Oct 31 13:08:37.035344 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:37.035182 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:37.164007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:37.161322 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-model-cache\") pod \"explicit-model-6106a3-kserve-79849d6c89-6mcrh\" (UID: \"c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b\") " pod="ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh" Oct 31 13:08:37.164007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:37.161455 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-tls-certs\") pod \"explicit-model-6106a3-kserve-79849d6c89-6mcrh\" (UID: \"c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b\") " pod="ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh" Oct 31 13:08:37.164007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:37.161635 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scgz9\" (UniqueName: \"kubernetes.io/projected/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-kube-api-access-scgz9\") pod \"explicit-model-6106a3-kserve-79849d6c89-6mcrh\" (UID: \"c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b\") " pod="ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh" Oct 31 13:08:37.164007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:37.161679 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-tmp-dir\") pod \"explicit-model-6106a3-kserve-79849d6c89-6mcrh\" (UID: \"c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b\") " pod="ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh" Oct 31 13:08:37.164007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:37.161740 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-home\") pod \"explicit-model-6106a3-kserve-79849d6c89-6mcrh\" (UID: \"c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b\") " pod="ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh" Oct 31 13:08:37.164007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:37.161771 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-dshm\") pod \"explicit-model-6106a3-kserve-79849d6c89-6mcrh\" (UID: \"c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b\") " pod="ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh" Oct 31 13:08:37.264328 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:37.264278 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:37.277803 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:37.266136 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-home\") pod \"explicit-model-6106a3-kserve-79849d6c89-6mcrh\" (UID: \"c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b\") " pod="ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh" Oct 31 13:08:37.277803 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:37.276282 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-dshm\") pod \"explicit-model-6106a3-kserve-79849d6c89-6mcrh\" (UID: \"c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b\") " pod="ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh" Oct 31 13:08:37.277803 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:37.276561 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-model-cache\") pod \"explicit-model-6106a3-kserve-79849d6c89-6mcrh\" (UID: \"c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b\") " pod="ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh" Oct 31 13:08:37.277803 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:37.276668 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-tls-certs\") pod \"explicit-model-6106a3-kserve-79849d6c89-6mcrh\" (UID: \"c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b\") " pod="ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh" Oct 31 13:08:37.277803 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:37.276861 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scgz9\" (UniqueName: \"kubernetes.io/projected/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-kube-api-access-scgz9\") pod \"explicit-model-6106a3-kserve-79849d6c89-6mcrh\" (UID: \"c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b\") " pod="ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh" Oct 31 13:08:37.277803 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:37.276907 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-tmp-dir\") pod \"explicit-model-6106a3-kserve-79849d6c89-6mcrh\" (UID: \"c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b\") " pod="ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh" Oct 31 13:08:37.277803 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:37.277424 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-tmp-dir\") pod \"explicit-model-6106a3-kserve-79849d6c89-6mcrh\" (UID: \"c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b\") " pod="ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh" Oct 31 13:08:37.277803 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:37.266678 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-home\") pod \"explicit-model-6106a3-kserve-79849d6c89-6mcrh\" (UID: \"c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b\") " pod="ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh" Oct 31 13:08:37.287392 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:37.282929 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-model-cache\") pod \"explicit-model-6106a3-kserve-79849d6c89-6mcrh\" (UID: \"c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b\") " pod="ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh" Oct 31 13:08:37.305768 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:37.296621 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-tls-certs\") pod \"explicit-model-6106a3-kserve-79849d6c89-6mcrh\" (UID: \"c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b\") " pod="ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh" Oct 31 13:08:37.305768 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:37.305267 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-dshm\") pod \"explicit-model-6106a3-kserve-79849d6c89-6mcrh\" (UID: \"c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b\") " pod="ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh" Oct 31 13:08:37.325664 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:37.324807 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scgz9\" (UniqueName: \"kubernetes.io/projected/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-kube-api-access-scgz9\") pod \"explicit-model-6106a3-kserve-79849d6c89-6mcrh\" (UID: \"c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b\") " pod="ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh" Oct 31 13:08:37.325664 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:37.325363 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh" Oct 31 13:08:37.598092 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:37.597187 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:37.794224 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:37.779941 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh"] Oct 31 13:08:37.796729 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:08:37.796697 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc07ffb4e_e21f_4a07_a3a8_2f9f1289e81b.slice/crio-e418762cf3659a14414857f87b8548399873fc751959ab91e7e3066b3e5d1665 WatchSource:0}: Error finding container e418762cf3659a14414857f87b8548399873fc751959ab91e7e3066b3e5d1665: Status 404 returned error can't find the container with id e418762cf3659a14414857f87b8548399873fc751959ab91e7e3066b3e5d1665 Oct 31 13:08:37.921063 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:37.920639 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:38.773224 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:38.773187 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:38.811202 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:38.811161 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh" event={"ID":"c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b","Type":"ContainerStarted","Data":"ad22cbce53281c373fbb5e7aaa7cc22dd103bb8ac26394f89481e4ffd38038c4"} Oct 31 13:08:38.811681 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:38.811215 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh" event={"ID":"c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b","Type":"ContainerStarted","Data":"e418762cf3659a14414857f87b8548399873fc751959ab91e7e3066b3e5d1665"} Oct 31 13:08:38.832429 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:38.832381 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh" podStartSLOduration=2.832366929 podStartE2EDuration="2.832366929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:08:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:08:38.830685964 +0000 UTC m=+2165.313808877" watchObservedRunningTime="2026-10-31 13:08:38.832366929 +0000 UTC m=+2165.315489830" Oct 31 13:08:39.326643 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:39.326604 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh" Oct 31 13:08:39.604313 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:39.601583 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:40.449008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:40.438522 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:41.219319 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:41.217925 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:41.661405 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:41.661322 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:42.124662 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:42.124626 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:42.999910 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:42.982745 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:43.725653 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:43.725545 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:43.885919 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:43.885888 6841 generic.go:356] "Generic (PLEG): container finished" podID="913b94e9-bc62-42b9-ad68-2fab683a9593" containerID="a6a0b04c2bd005394a9dd11838d16ed7be93d59a2e75836d9a9ac3a8d690f495" exitCode=0 Oct 31 13:08:43.886121 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:43.885964 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" event={"ID":"913b94e9-bc62-42b9-ad68-2fab683a9593","Type":"ContainerDied","Data":"a6a0b04c2bd005394a9dd11838d16ed7be93d59a2e75836d9a9ac3a8d690f495"} Oct 31 13:08:43.886531 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:43.886510 6841 scope.go:122] "RemoveContainer" containerID="a6a0b04c2bd005394a9dd11838d16ed7be93d59a2e75836d9a9ac3a8d690f495" Oct 31 13:08:43.887890 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:43.887870 6841 generic.go:356] "Generic (PLEG): container finished" podID="e4e95366-875b-4fda-8e1b-6e441df2f5b2" containerID="292291e58a2f3d3a1b8136feffc27ea8b3fc67d29f274879803b92e63bd231b4" exitCode=0 Oct 31 13:08:43.888017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:43.887939 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" event={"ID":"e4e95366-875b-4fda-8e1b-6e441df2f5b2","Type":"ContainerDied","Data":"292291e58a2f3d3a1b8136feffc27ea8b3fc67d29f274879803b92e63bd231b4"} Oct 31 13:08:43.888330 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:43.888316 6841 scope.go:122] "RemoveContainer" containerID="292291e58a2f3d3a1b8136feffc27ea8b3fc67d29f274879803b92e63bd231b4" Oct 31 13:08:43.888378 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:43.888337 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:08:43.888414 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:43.888400 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:08:43.888449 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:43.888429 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:08:43.889812 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:43.889787 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:08:43.889882 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:43.889850 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:08:43.889927 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:43.889882 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:08:44.666011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:44.661411 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:44.905680 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:44.905644 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" event={"ID":"913b94e9-bc62-42b9-ad68-2fab683a9593","Type":"ContainerStarted","Data":"0ff971f5a7c643814dbb2f222e94de703925055e2a202eb210ace7ecb09502c2"} Oct 31 13:08:44.907609 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:44.907584 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" event={"ID":"e4e95366-875b-4fda-8e1b-6e441df2f5b2","Type":"ContainerStarted","Data":"a23137f1b1b265f1cf0325beaa79d15d14a373a04245b16622e8c6d98aec9cf1"} Oct 31 13:08:45.178022 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:45.168615 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:08:45.178022 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:45.168793 6841 patch_prober.go:29] interesting pod/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 31 13:08:45.178022 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:45.168842 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" podUID="913b94e9-bc62-42b9-ad68-2fab683a9593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 31 13:08:45.193372 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:45.192310 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:08:45.193687 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:45.193662 6841 patch_prober.go:29] interesting pod/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 31 13:08:45.193841 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:45.193818 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" podUID="e4e95366-875b-4fda-8e1b-6e441df2f5b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 31 13:08:45.472159 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:45.472127 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:45.689530 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:45.689499 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:08:46.165387 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:46.165349 6841 patch_prober.go:29] interesting pod/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 31 13:08:46.165696 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:46.165666 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" podUID="913b94e9-bc62-42b9-ad68-2fab683a9593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 31 13:08:46.193491 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:46.191930 6841 patch_prober.go:29] interesting pod/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 31 13:08:46.193491 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:46.192010 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" podUID="e4e95366-875b-4fda-8e1b-6e441df2f5b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 31 13:08:46.293005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:46.282826 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:46.505538 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:46.505504 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:47.175011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:47.171378 6841 patch_prober.go:29] interesting pod/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 31 13:08:47.175011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:47.171440 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" podUID="913b94e9-bc62-42b9-ad68-2fab683a9593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 31 13:08:47.193529 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:47.193491 6841 patch_prober.go:29] interesting pod/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 31 13:08:47.193826 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:47.193794 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" podUID="e4e95366-875b-4fda-8e1b-6e441df2f5b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 31 13:08:47.321003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:47.319663 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:47.340045 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:47.330239 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh" Oct 31 13:08:47.440311 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:47.440231 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh" Oct 31 13:08:47.945008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:47.941142 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh" Oct 31 13:08:48.180009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:48.171427 6841 patch_prober.go:29] interesting pod/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 31 13:08:48.180009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:48.171491 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" podUID="913b94e9-bc62-42b9-ad68-2fab683a9593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 31 13:08:48.196084 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:48.193444 6841 patch_prober.go:29] interesting pod/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 31 13:08:48.196084 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:48.193515 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" podUID="e4e95366-875b-4fda-8e1b-6e441df2f5b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 31 13:08:48.219012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:48.214171 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:49.165197 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:49.165163 6841 patch_prober.go:29] interesting pod/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 31 13:08:49.165719 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:49.165666 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" podUID="913b94e9-bc62-42b9-ad68-2fab683a9593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 31 13:08:49.167464 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:49.167440 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:49.194481 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:49.194441 6841 patch_prober.go:29] interesting pod/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 31 13:08:49.195152 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:49.195115 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" podUID="e4e95366-875b-4fda-8e1b-6e441df2f5b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 31 13:08:50.010006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:50.005937 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:50.140961 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:08:50.140921 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:08:50.178124 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:50.170524 6841 patch_prober.go:29] interesting pod/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 31 13:08:50.178124 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:50.170604 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" podUID="913b94e9-bc62-42b9-ad68-2fab683a9593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 31 13:08:50.193586 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:50.193545 6841 patch_prober.go:29] interesting pod/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 31 13:08:50.193913 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:50.193880 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" podUID="e4e95366-875b-4fda-8e1b-6e441df2f5b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 31 13:08:50.232066 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:50.232033 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:50.440858 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:50.440820 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:50.921808 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:50.921764 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:51.170357 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:51.170326 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:08:51.170838 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:51.170815 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:08:51.171579 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:51.171562 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:08:51.196648 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:51.196626 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:08:51.197487 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:51.197467 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:08:51.707085 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:51.707046 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:52.478915 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:52.478880 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:52.948760 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:52.948723 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh"] Oct 31 13:08:53.034657 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:53.034617 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:53.370409 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:53.370324 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm"] Oct 31 13:08:53.377439 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:53.377410 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm" Oct 31 13:08:53.385713 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:53.385687 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm"] Oct 31 13:08:53.393732 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:53.393706 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-377f60\"/\"auth-test-model-377f60-kserve-self-signed-certs\"" Oct 31 13:08:53.394122 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:53.394100 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-377f60\"/\"default-dockercfg-9sbgv\"" Oct 31 13:08:53.394259 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:53.394224 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-377f60\"/\"kube-root-ca.crt\"" Oct 31 13:08:53.394336 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:53.394288 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-377f60\"/\"openshift-service-ca.crt\"" Oct 31 13:08:53.451401 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:53.451349 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:53.472430 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:53.472401 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-dshm\") pod \"auth-test-model-377f60-kserve-5db5968cb9-2n2lm\" (UID: \"09e7a6fc-9d99-47f3-aae0-310859bcc6bb\") " pod="ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm" Oct 31 13:08:53.472595 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:53.472529 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-tls-certs\") pod \"auth-test-model-377f60-kserve-5db5968cb9-2n2lm\" (UID: \"09e7a6fc-9d99-47f3-aae0-310859bcc6bb\") " pod="ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm" Oct 31 13:08:53.472595 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:53.472564 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-tmp-dir\") pod \"auth-test-model-377f60-kserve-5db5968cb9-2n2lm\" (UID: \"09e7a6fc-9d99-47f3-aae0-310859bcc6bb\") " pod="ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm" Oct 31 13:08:53.472742 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:53.472726 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-home\") pod \"auth-test-model-377f60-kserve-5db5968cb9-2n2lm\" (UID: \"09e7a6fc-9d99-47f3-aae0-310859bcc6bb\") " pod="ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm" Oct 31 13:08:53.472928 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:53.472911 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sf75\" (UniqueName: \"kubernetes.io/projected/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-kube-api-access-2sf75\") pod \"auth-test-model-377f60-kserve-5db5968cb9-2n2lm\" (UID: \"09e7a6fc-9d99-47f3-aae0-310859bcc6bb\") " pod="ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm" Oct 31 13:08:53.473051 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:53.473031 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-model-cache\") pod \"auth-test-model-377f60-kserve-5db5968cb9-2n2lm\" (UID: \"09e7a6fc-9d99-47f3-aae0-310859bcc6bb\") " pod="ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm" Oct 31 13:08:53.580366 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:53.578069 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-dshm\") pod \"auth-test-model-377f60-kserve-5db5968cb9-2n2lm\" (UID: \"09e7a6fc-9d99-47f3-aae0-310859bcc6bb\") " pod="ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm" Oct 31 13:08:53.580366 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:53.578160 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-tls-certs\") pod \"auth-test-model-377f60-kserve-5db5968cb9-2n2lm\" (UID: \"09e7a6fc-9d99-47f3-aae0-310859bcc6bb\") " pod="ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm" Oct 31 13:08:53.580366 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:53.578194 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-tmp-dir\") pod \"auth-test-model-377f60-kserve-5db5968cb9-2n2lm\" (UID: \"09e7a6fc-9d99-47f3-aae0-310859bcc6bb\") " pod="ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm" Oct 31 13:08:53.580366 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:53.578315 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-home\") pod \"auth-test-model-377f60-kserve-5db5968cb9-2n2lm\" (UID: \"09e7a6fc-9d99-47f3-aae0-310859bcc6bb\") " pod="ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm" Oct 31 13:08:53.580366 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:53.578510 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sf75\" (UniqueName: \"kubernetes.io/projected/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-kube-api-access-2sf75\") pod \"auth-test-model-377f60-kserve-5db5968cb9-2n2lm\" (UID: \"09e7a6fc-9d99-47f3-aae0-310859bcc6bb\") " pod="ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm" Oct 31 13:08:53.580366 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:53.578595 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-model-cache\") pod \"auth-test-model-377f60-kserve-5db5968cb9-2n2lm\" (UID: \"09e7a6fc-9d99-47f3-aae0-310859bcc6bb\") " pod="ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm" Oct 31 13:08:53.580366 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:53.579176 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-model-cache\") pod \"auth-test-model-377f60-kserve-5db5968cb9-2n2lm\" (UID: \"09e7a6fc-9d99-47f3-aae0-310859bcc6bb\") " pod="ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm" Oct 31 13:08:53.603433 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:53.603387 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-tmp-dir\") pod \"auth-test-model-377f60-kserve-5db5968cb9-2n2lm\" (UID: \"09e7a6fc-9d99-47f3-aae0-310859bcc6bb\") " pod="ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm" Oct 31 13:08:53.613004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:53.604349 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-home\") pod \"auth-test-model-377f60-kserve-5db5968cb9-2n2lm\" (UID: \"09e7a6fc-9d99-47f3-aae0-310859bcc6bb\") " pod="ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm" Oct 31 13:08:53.613650 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:53.613619 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-tls-certs\") pod \"auth-test-model-377f60-kserve-5db5968cb9-2n2lm\" (UID: \"09e7a6fc-9d99-47f3-aae0-310859bcc6bb\") " pod="ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm" Oct 31 13:08:53.668055 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:53.642869 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sf75\" (UniqueName: \"kubernetes.io/projected/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-kube-api-access-2sf75\") pod \"auth-test-model-377f60-kserve-5db5968cb9-2n2lm\" (UID: \"09e7a6fc-9d99-47f3-aae0-310859bcc6bb\") " pod="ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm" Oct 31 13:08:53.668055 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:53.645654 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-dshm\") pod \"auth-test-model-377f60-kserve-5db5968cb9-2n2lm\" (UID: \"09e7a6fc-9d99-47f3-aae0-310859bcc6bb\") " pod="ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm" Oct 31 13:08:53.668055 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:53.665192 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:53.706006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:53.695264 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm" Oct 31 13:08:54.036057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:54.036030 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm"] Oct 31 13:08:54.037683 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:08:54.037653 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09e7a6fc_9d99_47f3_aae0_310859bcc6bb.slice/crio-5fb704739815bb82b1c7a08364f780d2d9835b192c0fbef614704f09fa6f901d WatchSource:0}: Error finding container 5fb704739815bb82b1c7a08364f780d2d9835b192c0fbef614704f09fa6f901d: Status 404 returned error can't find the container with id 5fb704739815bb82b1c7a08364f780d2d9835b192c0fbef614704f09fa6f901d Oct 31 13:08:54.128014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:54.125390 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm" event={"ID":"09e7a6fc-9d99-47f3-aae0-310859bcc6bb","Type":"ContainerStarted","Data":"5e0f3a4abb4229f77cd7b0e4e9acb534be79e792f0596b42d8d6bc4c1f8bdb22"} Oct 31 13:08:54.128014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:54.125444 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm" event={"ID":"09e7a6fc-9d99-47f3-aae0-310859bcc6bb","Type":"ContainerStarted","Data":"5fb704739815bb82b1c7a08364f780d2d9835b192c0fbef614704f09fa6f901d"} Oct 31 13:08:54.150156 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:54.150106 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm" podStartSLOduration=1.150091187 podStartE2EDuration="1.150091187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:08:54.147431864 +0000 UTC m=+2180.630554766" watchObservedRunningTime="2026-10-31 13:08:54.150091187 +0000 UTC m=+2180.633214089" Oct 31 13:08:54.535004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:54.531512 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:54.706361 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:54.706328 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm" Oct 31 13:08:55.327062 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:55.323422 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:55.585683 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:55.585588 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:56.190225 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:56.190183 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:56.971038 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:56.970977 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:57.716028 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:57.715969 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:57.948654 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:57.948610 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:58.442612 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:58.442577 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:58.643970 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:58.643921 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:08:59.602333 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:08:59.600807 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:00.762708 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:00.762668 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:01.734883 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:01.734847 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:02.625769 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:02.625732 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:03.518005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:03.517573 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:03.706138 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:03.706090 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm" Oct 31 13:09:03.719141 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:03.719115 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm" Oct 31 13:09:04.361197 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:04.353936 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:04.505486 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:04.505456 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm" Oct 31 13:09:05.131718 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:05.131648 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:06.089399 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:06.089357 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:07.037624 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:07.037584 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:07.852363 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:07.852323 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:07.977032 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:07.976953 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh" podUID="c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b" containerName="main" containerID="cri-o://ad22cbce53281c373fbb5e7aaa7cc22dd103bb8ac26394f89481e4ffd38038c4" gracePeriod=45 Oct 31 13:09:08.294373 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:08.294343 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh" Oct 31 13:09:08.429764 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:08.429679 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-kube-api-access-scgz9\" (UniqueName: \"kubernetes.io/projected/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-kube-api-access-scgz9\") pod \"c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b\" (UID: \"c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b\") " Oct 31 13:09:08.429920 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:08.429770 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-dshm\") pod \"c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b\" (UID: \"c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b\") " Oct 31 13:09:08.429920 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:08.429810 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-model-cache\") pod \"c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b\" (UID: \"c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b\") " Oct 31 13:09:08.429920 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:08.429895 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-home\" (UniqueName: \"kubernetes.io/empty-dir/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-home\") pod \"c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b\" (UID: \"c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b\") " Oct 31 13:09:08.430080 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:08.429996 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-tmp-dir\") pod \"c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b\" (UID: \"c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b\") " Oct 31 13:09:08.430080 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:08.430073 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-tls-certs\" (UniqueName: \"kubernetes.io/secret/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-tls-certs\") pod \"c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b\" (UID: \"c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b\") " Oct 31 13:09:08.431784 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:08.431492 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-model-cache" pod "c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b" (UID: "c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:09:08.431940 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:08.431794 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-tmp-dir" pod "c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b" (UID: "c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:09:08.431940 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:08.431851 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-home" pod "c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b" (UID: "c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:09:08.432092 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:08.431939 6841 generic.go:356] "Generic (PLEG): container finished" podID="c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b" containerID="ad22cbce53281c373fbb5e7aaa7cc22dd103bb8ac26394f89481e4ffd38038c4" exitCode=0 Oct 31 13:09:08.432092 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:08.432062 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh" event={"ID":"c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b","Type":"ContainerDied","Data":"ad22cbce53281c373fbb5e7aaa7cc22dd103bb8ac26394f89481e4ffd38038c4"} Oct 31 13:09:08.432203 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:08.432079 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh" Oct 31 13:09:08.432203 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:08.432111 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh" event={"ID":"c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b","Type":"ContainerDied","Data":"e418762cf3659a14414857f87b8548399873fc751959ab91e7e3066b3e5d1665"} Oct 31 13:09:08.432203 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:08.432137 6841 scope.go:122] "RemoveContainer" containerID="ad22cbce53281c373fbb5e7aaa7cc22dd103bb8ac26394f89481e4ffd38038c4" Oct 31 13:09:08.434678 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:08.434635 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-kube-api-access-scgz9" pod "c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b" (UID: "c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b"). InnerVolumeSpecName "kube-api-access-scgz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:09:08.438327 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:08.438289 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-dshm" pod "c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b" (UID: "c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:09:08.438494 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:08.438337 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-tls-certs" pod "c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b" (UID: "c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:09:08.478299 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:08.478269 6841 scope.go:122] "RemoveContainer" containerID="ad22cbce53281c373fbb5e7aaa7cc22dd103bb8ac26394f89481e4ffd38038c4" Oct 31 13:09:08.478742 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:09:08.478710 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad22cbce53281c373fbb5e7aaa7cc22dd103bb8ac26394f89481e4ffd38038c4\": container with ID starting with ad22cbce53281c373fbb5e7aaa7cc22dd103bb8ac26394f89481e4ffd38038c4 not found: ID does not exist" containerID="ad22cbce53281c373fbb5e7aaa7cc22dd103bb8ac26394f89481e4ffd38038c4" Oct 31 13:09:08.478868 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:08.478758 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad22cbce53281c373fbb5e7aaa7cc22dd103bb8ac26394f89481e4ffd38038c4"} err="failed to get container status \"ad22cbce53281c373fbb5e7aaa7cc22dd103bb8ac26394f89481e4ffd38038c4\": rpc error: code = NotFound desc = could not find container \"ad22cbce53281c373fbb5e7aaa7cc22dd103bb8ac26394f89481e4ffd38038c4\": container with ID starting with ad22cbce53281c373fbb5e7aaa7cc22dd103bb8ac26394f89481e4ffd38038c4 not found: ID does not exist" Oct 31 13:09:08.533351 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:08.533302 6841 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-tls-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:08.533351 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:08.533347 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scgz9\" (UniqueName: \"kubernetes.io/projected/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-kube-api-access-scgz9\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:08.533616 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:08.533365 6841 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-dshm\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:08.533616 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:08.533382 6841 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-model-cache\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:08.533616 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:08.533396 6841 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-home\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:08.533616 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:08.533413 6841 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b-tmp-dir\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:08.664260 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:08.664160 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:08.767563 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:08.767526 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh"] Oct 31 13:09:08.771207 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:08.771185 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-6106a3/explicit-model-6106a3-kserve-79849d6c89-6mcrh"] Oct 31 13:09:09.366307 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:09.366269 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:09.645911 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:09.645828 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b" path="/var/lib/kubelet/pods/c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b/volumes" Oct 31 13:09:10.253165 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:10.252508 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:10.655732 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:10.655638 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh"] Oct 31 13:09:10.656393 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:10.656375 6841 cpu_manager.go:414] "containerMap: removing container" podUID="c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b" containerName="main" Oct 31 13:09:10.656393 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:10.656392 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b" containerName="main" Oct 31 13:09:10.656532 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:10.656522 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="c07ffb4e-e21f-4a07-a3a8-2f9f1289e81b" containerName="main" Oct 31 13:09:10.717680 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:10.717637 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh"] Oct 31 13:09:10.717879 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:10.717761 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh" Oct 31 13:09:10.723704 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:10.723610 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-5c24e1\"/\"auth-test-model-5c24e1-kserve-self-signed-certs\"" Oct 31 13:09:10.724649 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:10.724622 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-5c24e1\"/\"default-dockercfg-4w66c\"" Oct 31 13:09:10.724834 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:10.724630 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-5c24e1\"/\"openshift-service-ca.crt\"" Oct 31 13:09:10.724834 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:10.724753 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-5c24e1\"/\"kube-root-ca.crt\"" Oct 31 13:09:10.790935 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:10.790880 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a2e95ce-2103-46b2-867b-de098db55e96-tls-certs\") pod \"auth-test-model-5c24e1-kserve-85b876497c-2nwbh\" (UID: \"4a2e95ce-2103-46b2-867b-de098db55e96\") " pod="ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh" Oct 31 13:09:10.791202 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:10.791044 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a2e95ce-2103-46b2-867b-de098db55e96-dshm\") pod \"auth-test-model-5c24e1-kserve-85b876497c-2nwbh\" (UID: \"4a2e95ce-2103-46b2-867b-de098db55e96\") " pod="ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh" Oct 31 13:09:10.791202 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:10.791118 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a2e95ce-2103-46b2-867b-de098db55e96-model-cache\") pod \"auth-test-model-5c24e1-kserve-85b876497c-2nwbh\" (UID: \"4a2e95ce-2103-46b2-867b-de098db55e96\") " pod="ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh" Oct 31 13:09:10.791356 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:10.791202 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a2e95ce-2103-46b2-867b-de098db55e96-home\") pod \"auth-test-model-5c24e1-kserve-85b876497c-2nwbh\" (UID: \"4a2e95ce-2103-46b2-867b-de098db55e96\") " pod="ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh" Oct 31 13:09:10.791356 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:10.791266 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:10.791356 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:10.791287 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm8pv\" (UniqueName: \"kubernetes.io/projected/4a2e95ce-2103-46b2-867b-de098db55e96-kube-api-access-fm8pv\") pod \"auth-test-model-5c24e1-kserve-85b876497c-2nwbh\" (UID: \"4a2e95ce-2103-46b2-867b-de098db55e96\") " pod="ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh" Oct 31 13:09:10.791540 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:10.791473 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a2e95ce-2103-46b2-867b-de098db55e96-tmp-dir\") pod \"auth-test-model-5c24e1-kserve-85b876497c-2nwbh\" (UID: \"4a2e95ce-2103-46b2-867b-de098db55e96\") " pod="ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh" Oct 31 13:09:11.082673 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:11.082630 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a2e95ce-2103-46b2-867b-de098db55e96-home\") pod \"auth-test-model-5c24e1-kserve-85b876497c-2nwbh\" (UID: \"4a2e95ce-2103-46b2-867b-de098db55e96\") " pod="ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh" Oct 31 13:09:11.083095 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:11.083057 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm8pv\" (UniqueName: \"kubernetes.io/projected/4a2e95ce-2103-46b2-867b-de098db55e96-kube-api-access-fm8pv\") pod \"auth-test-model-5c24e1-kserve-85b876497c-2nwbh\" (UID: \"4a2e95ce-2103-46b2-867b-de098db55e96\") " pod="ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh" Oct 31 13:09:11.083377 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:11.083351 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a2e95ce-2103-46b2-867b-de098db55e96-tmp-dir\") pod \"auth-test-model-5c24e1-kserve-85b876497c-2nwbh\" (UID: \"4a2e95ce-2103-46b2-867b-de098db55e96\") " pod="ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh" Oct 31 13:09:11.083537 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:11.083510 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a2e95ce-2103-46b2-867b-de098db55e96-tls-certs\") pod \"auth-test-model-5c24e1-kserve-85b876497c-2nwbh\" (UID: \"4a2e95ce-2103-46b2-867b-de098db55e96\") " pod="ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh" Oct 31 13:09:11.083630 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:11.083568 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a2e95ce-2103-46b2-867b-de098db55e96-dshm\") pod \"auth-test-model-5c24e1-kserve-85b876497c-2nwbh\" (UID: \"4a2e95ce-2103-46b2-867b-de098db55e96\") " pod="ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh" Oct 31 13:09:11.083692 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:11.083635 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a2e95ce-2103-46b2-867b-de098db55e96-model-cache\") pod \"auth-test-model-5c24e1-kserve-85b876497c-2nwbh\" (UID: \"4a2e95ce-2103-46b2-867b-de098db55e96\") " pod="ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh" Oct 31 13:09:11.084717 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:11.084111 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a2e95ce-2103-46b2-867b-de098db55e96-tmp-dir\") pod \"auth-test-model-5c24e1-kserve-85b876497c-2nwbh\" (UID: \"4a2e95ce-2103-46b2-867b-de098db55e96\") " pod="ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh" Oct 31 13:09:11.084717 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:11.084132 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a2e95ce-2103-46b2-867b-de098db55e96-model-cache\") pod \"auth-test-model-5c24e1-kserve-85b876497c-2nwbh\" (UID: \"4a2e95ce-2103-46b2-867b-de098db55e96\") " pod="ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh" Oct 31 13:09:11.084717 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:11.084697 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a2e95ce-2103-46b2-867b-de098db55e96-home\") pod \"auth-test-model-5c24e1-kserve-85b876497c-2nwbh\" (UID: \"4a2e95ce-2103-46b2-867b-de098db55e96\") " pod="ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh" Oct 31 13:09:11.088645 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:11.088249 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:11.116025 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:11.115937 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a2e95ce-2103-46b2-867b-de098db55e96-tls-certs\") pod \"auth-test-model-5c24e1-kserve-85b876497c-2nwbh\" (UID: \"4a2e95ce-2103-46b2-867b-de098db55e96\") " pod="ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh" Oct 31 13:09:11.183196 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:11.181733 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a2e95ce-2103-46b2-867b-de098db55e96-dshm\") pod \"auth-test-model-5c24e1-kserve-85b876497c-2nwbh\" (UID: \"4a2e95ce-2103-46b2-867b-de098db55e96\") " pod="ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh" Oct 31 13:09:11.183196 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:11.182517 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm8pv\" (UniqueName: \"kubernetes.io/projected/4a2e95ce-2103-46b2-867b-de098db55e96-kube-api-access-fm8pv\") pod \"auth-test-model-5c24e1-kserve-85b876497c-2nwbh\" (UID: \"4a2e95ce-2103-46b2-867b-de098db55e96\") " pod="ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh" Oct 31 13:09:11.350591 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:11.350506 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh" Oct 31 13:09:12.199778 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:11.679074 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh"] Oct 31 13:09:12.199778 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:09:11.679413 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a2e95ce_2103_46b2_867b_de098db55e96.slice/crio-aa5a4107323925b1ab5f227c6aff1de12cd98c7cf4745fc40634f24f21abbe84 WatchSource:0}: Error finding container aa5a4107323925b1ab5f227c6aff1de12cd98c7cf4745fc40634f24f21abbe84: Status 404 returned error can't find the container with id aa5a4107323925b1ab5f227c6aff1de12cd98c7cf4745fc40634f24f21abbe84 Oct 31 13:09:12.199778 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:11.923767 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:12.267655 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:12.267626 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:12.524846 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:12.524771 6841 generic.go:356] "Generic (PLEG): container finished" podID="c85bee9f-4267-44ec-a03f-0b97c6b95d24" containerID="1b1b3d24f4bcadf276d1dcf4ebedd9018d267f346abb926a4c64ac2f352d023f" exitCode=0 Oct 31 13:09:12.525069 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:12.524841 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" event={"ID":"c85bee9f-4267-44ec-a03f-0b97c6b95d24","Type":"ContainerDied","Data":"1b1b3d24f4bcadf276d1dcf4ebedd9018d267f346abb926a4c64ac2f352d023f"} Oct 31 13:09:12.525344 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:12.525318 6841 scope.go:122] "RemoveContainer" containerID="1b1b3d24f4bcadf276d1dcf4ebedd9018d267f346abb926a4c64ac2f352d023f" Oct 31 13:09:12.526724 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:12.526689 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh" event={"ID":"4a2e95ce-2103-46b2-867b-de098db55e96","Type":"ContainerStarted","Data":"f6cd0a5ca938530a3e0f5435a0fbaec5fe8e305650a907f2b6a1f3f3f0c92b0f"} Oct 31 13:09:12.526843 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:12.526728 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh" event={"ID":"4a2e95ce-2103-46b2-867b-de098db55e96","Type":"ContainerStarted","Data":"aa5a4107323925b1ab5f227c6aff1de12cd98c7cf4745fc40634f24f21abbe84"} Oct 31 13:09:12.527145 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:12.527110 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:09:12.527258 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:12.527193 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:09:12.527258 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:12.527231 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:09:12.528691 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:12.528669 6841 generic.go:356] "Generic (PLEG): container finished" podID="0fe9fa73-5f1d-47ea-b303-869b3392ac75" containerID="e5ad7fee3046d5d4b15fd5814289d3b69f57aba3304c813816f26f282ea2c55d" exitCode=0 Oct 31 13:09:12.528804 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:12.528696 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" event={"ID":"0fe9fa73-5f1d-47ea-b303-869b3392ac75","Type":"ContainerDied","Data":"e5ad7fee3046d5d4b15fd5814289d3b69f57aba3304c813816f26f282ea2c55d"} Oct 31 13:09:12.529106 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:12.529088 6841 scope.go:122] "RemoveContainer" containerID="e5ad7fee3046d5d4b15fd5814289d3b69f57aba3304c813816f26f282ea2c55d" Oct 31 13:09:12.531038 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:12.530977 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:09:12.531112 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:12.531076 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:09:12.531250 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:12.531116 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:09:12.660324 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:12.660291 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:13.340589 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:13.340559 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:09:13.380006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:13.379083 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh" Oct 31 13:09:13.608714 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:13.608616 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:13.708820 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:13.708785 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" event={"ID":"c85bee9f-4267-44ec-a03f-0b97c6b95d24","Type":"ContainerStarted","Data":"1b0b0012d99c80ccff6505c5d68aa38f4e624f2ee77b13af29ca79b0d0d08ec3"} Oct 31 13:09:13.709016 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:13.708838 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" event={"ID":"0fe9fa73-5f1d-47ea-b303-869b3392ac75","Type":"ContainerStarted","Data":"8f45944cbcfdd44d257c64794984a7e215891df1dfaba1655c0ecb284ea16ac6"} Oct 31 13:09:13.792006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:13.791637 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh" podStartSLOduration=3.79161201 podStartE2EDuration="3.79161201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:09:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:09:12.59927018 +0000 UTC m=+2199.082393080" watchObservedRunningTime="2026-10-31 13:09:13.79161201 +0000 UTC m=+2200.274734911" Oct 31 13:09:14.091377 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:14.090190 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:09:14.092836 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:14.092802 6841 patch_prober.go:29] interesting pod/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 31 13:09:14.093044 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:14.092868 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" podUID="0fe9fa73-5f1d-47ea-b303-869b3392ac75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 31 13:09:14.193104 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:14.193061 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw"] Oct 31 13:09:14.202134 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:14.201266 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw" Oct 31 13:09:14.206679 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:14.206636 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw"] Oct 31 13:09:14.209220 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:14.208540 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-569075\"/\"default-dockercfg-hb9tc\"" Oct 31 13:09:14.209220 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:14.208663 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-569075\"/\"auth-test-model-569075-kserve-self-signed-certs\"" Oct 31 13:09:14.209220 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:14.208899 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-569075\"/\"kube-root-ca.crt\"" Oct 31 13:09:14.209220 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:14.209081 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-569075\"/\"openshift-service-ca.crt\"" Oct 31 13:09:14.293298 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:14.291952 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a034a05-49d4-4c08-a9b5-5299af987eb1-tmp-dir\") pod \"auth-test-model-569075-kserve-7d5f64445c-tnphw\" (UID: \"4a034a05-49d4-4c08-a9b5-5299af987eb1\") " pod="ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw" Oct 31 13:09:14.293298 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:14.292100 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-274hp\" (UniqueName: \"kubernetes.io/projected/4a034a05-49d4-4c08-a9b5-5299af987eb1-kube-api-access-274hp\") pod \"auth-test-model-569075-kserve-7d5f64445c-tnphw\" (UID: \"4a034a05-49d4-4c08-a9b5-5299af987eb1\") " pod="ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw" Oct 31 13:09:14.293298 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:14.292500 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a034a05-49d4-4c08-a9b5-5299af987eb1-tls-certs\") pod \"auth-test-model-569075-kserve-7d5f64445c-tnphw\" (UID: \"4a034a05-49d4-4c08-a9b5-5299af987eb1\") " pod="ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw" Oct 31 13:09:14.293298 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:14.292562 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a034a05-49d4-4c08-a9b5-5299af987eb1-dshm\") pod \"auth-test-model-569075-kserve-7d5f64445c-tnphw\" (UID: \"4a034a05-49d4-4c08-a9b5-5299af987eb1\") " pod="ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw" Oct 31 13:09:14.293298 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:14.292673 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a034a05-49d4-4c08-a9b5-5299af987eb1-model-cache\") pod \"auth-test-model-569075-kserve-7d5f64445c-tnphw\" (UID: \"4a034a05-49d4-4c08-a9b5-5299af987eb1\") " pod="ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw" Oct 31 13:09:14.293298 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:14.292780 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a034a05-49d4-4c08-a9b5-5299af987eb1-home\") pod \"auth-test-model-569075-kserve-7d5f64445c-tnphw\" (UID: \"4a034a05-49d4-4c08-a9b5-5299af987eb1\") " pod="ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw" Oct 31 13:09:14.310117 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:14.310071 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:14.397063 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:14.395114 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a034a05-49d4-4c08-a9b5-5299af987eb1-tls-certs\") pod \"auth-test-model-569075-kserve-7d5f64445c-tnphw\" (UID: \"4a034a05-49d4-4c08-a9b5-5299af987eb1\") " pod="ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw" Oct 31 13:09:14.397063 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:14.395172 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a034a05-49d4-4c08-a9b5-5299af987eb1-dshm\") pod \"auth-test-model-569075-kserve-7d5f64445c-tnphw\" (UID: \"4a034a05-49d4-4c08-a9b5-5299af987eb1\") " pod="ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw" Oct 31 13:09:14.397063 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:14.395209 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a034a05-49d4-4c08-a9b5-5299af987eb1-model-cache\") pod \"auth-test-model-569075-kserve-7d5f64445c-tnphw\" (UID: \"4a034a05-49d4-4c08-a9b5-5299af987eb1\") " pod="ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw" Oct 31 13:09:14.397063 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:14.395270 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a034a05-49d4-4c08-a9b5-5299af987eb1-home\") pod \"auth-test-model-569075-kserve-7d5f64445c-tnphw\" (UID: \"4a034a05-49d4-4c08-a9b5-5299af987eb1\") " pod="ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw" Oct 31 13:09:14.397063 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:14.395356 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a034a05-49d4-4c08-a9b5-5299af987eb1-tmp-dir\") pod \"auth-test-model-569075-kserve-7d5f64445c-tnphw\" (UID: \"4a034a05-49d4-4c08-a9b5-5299af987eb1\") " pod="ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw" Oct 31 13:09:14.397063 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:14.395380 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-274hp\" (UniqueName: \"kubernetes.io/projected/4a034a05-49d4-4c08-a9b5-5299af987eb1-kube-api-access-274hp\") pod \"auth-test-model-569075-kserve-7d5f64445c-tnphw\" (UID: \"4a034a05-49d4-4c08-a9b5-5299af987eb1\") " pod="ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw" Oct 31 13:09:14.403288 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:14.403255 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a034a05-49d4-4c08-a9b5-5299af987eb1-model-cache\") pod \"auth-test-model-569075-kserve-7d5f64445c-tnphw\" (UID: \"4a034a05-49d4-4c08-a9b5-5299af987eb1\") " pod="ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw" Oct 31 13:09:14.405249 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:14.404068 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a034a05-49d4-4c08-a9b5-5299af987eb1-tmp-dir\") pod \"auth-test-model-569075-kserve-7d5f64445c-tnphw\" (UID: \"4a034a05-49d4-4c08-a9b5-5299af987eb1\") " pod="ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw" Oct 31 13:09:14.405646 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:14.405625 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a034a05-49d4-4c08-a9b5-5299af987eb1-home\") pod \"auth-test-model-569075-kserve-7d5f64445c-tnphw\" (UID: \"4a034a05-49d4-4c08-a9b5-5299af987eb1\") " pod="ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw" Oct 31 13:09:14.456957 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:14.456715 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a034a05-49d4-4c08-a9b5-5299af987eb1-dshm\") pod \"auth-test-model-569075-kserve-7d5f64445c-tnphw\" (UID: \"4a034a05-49d4-4c08-a9b5-5299af987eb1\") " pod="ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw" Oct 31 13:09:14.457822 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:14.457794 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-274hp\" (UniqueName: \"kubernetes.io/projected/4a034a05-49d4-4c08-a9b5-5299af987eb1-kube-api-access-274hp\") pod \"auth-test-model-569075-kserve-7d5f64445c-tnphw\" (UID: \"4a034a05-49d4-4c08-a9b5-5299af987eb1\") " pod="ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw" Oct 31 13:09:14.459373 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:14.459350 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a034a05-49d4-4c08-a9b5-5299af987eb1-tls-certs\") pod \"auth-test-model-569075-kserve-7d5f64445c-tnphw\" (UID: \"4a034a05-49d4-4c08-a9b5-5299af987eb1\") " pod="ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw" Oct 31 13:09:14.532518 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:14.531288 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw" Oct 31 13:09:14.533569 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:14.533538 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:14.698670 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:14.698639 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:09:14.704214 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:14.703778 6841 patch_prober.go:29] interesting pod/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 31 13:09:14.704214 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:14.703831 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" podUID="c85bee9f-4267-44ec-a03f-0b97c6b95d24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 31 13:09:15.024596 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:15.024564 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw"] Oct 31 13:09:15.024857 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:09:15.024835 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a034a05_49d4_4c08_a9b5_5299af987eb1.slice/crio-162cf2954450b9f9bc606e983160875117ae4bc374d8e562cb2b7c663a86bd15 WatchSource:0}: Error finding container 162cf2954450b9f9bc606e983160875117ae4bc374d8e562cb2b7c663a86bd15: Status 404 returned error can't find the container with id 162cf2954450b9f9bc606e983160875117ae4bc374d8e562cb2b7c663a86bd15 Oct 31 13:09:15.097006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:15.095157 6841 patch_prober.go:29] interesting pod/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 31 13:09:15.097006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:15.095234 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" podUID="0fe9fa73-5f1d-47ea-b303-869b3392ac75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 31 13:09:15.259016 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:15.249790 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:15.694419 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:15.694383 6841 patch_prober.go:29] interesting pod/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 31 13:09:15.695042 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:15.695007 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" podUID="c85bee9f-4267-44ec-a03f-0b97c6b95d24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 31 13:09:15.723548 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:15.723516 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw" event={"ID":"4a034a05-49d4-4c08-a9b5-5299af987eb1","Type":"ContainerStarted","Data":"6c4711618e06d4b00edba9e966c3368f00bbe14d5f5db75bb9c06435eb09e6f2"} Oct 31 13:09:15.723681 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:15.723562 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw" event={"ID":"4a034a05-49d4-4c08-a9b5-5299af987eb1","Type":"ContainerStarted","Data":"162cf2954450b9f9bc606e983160875117ae4bc374d8e562cb2b7c663a86bd15"} Oct 31 13:09:15.790641 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:15.790576 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw" podStartSLOduration=1.790553226 podStartE2EDuration="1.790553226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:09:15.765261786 +0000 UTC m=+2202.248384690" watchObservedRunningTime="2026-10-31 13:09:15.790553226 +0000 UTC m=+2202.273676128" Oct 31 13:09:16.091088 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:16.090452 6841 patch_prober.go:29] interesting pod/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 31 13:09:16.091088 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:16.090514 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" podUID="0fe9fa73-5f1d-47ea-b303-869b3392ac75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 31 13:09:16.295030 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:16.289473 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:16.399243 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:16.399150 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:09:16.534222 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:16.534186 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw" Oct 31 13:09:16.694926 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:16.692344 6841 patch_prober.go:29] interesting pod/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 31 13:09:16.694926 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:16.692409 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" podUID="c85bee9f-4267-44ec-a03f-0b97c6b95d24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 31 13:09:17.092263 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:17.091440 6841 patch_prober.go:29] interesting pod/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 31 13:09:17.092263 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:17.091519 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" podUID="0fe9fa73-5f1d-47ea-b303-869b3392ac75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 31 13:09:17.191879 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:17.191843 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:17.697009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:17.693435 6841 patch_prober.go:29] interesting pod/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 31 13:09:17.697009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:17.693516 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" podUID="c85bee9f-4267-44ec-a03f-0b97c6b95d24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 31 13:09:18.044074 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:18.030240 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:18.092007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:18.091371 6841 patch_prober.go:29] interesting pod/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 31 13:09:18.092007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:18.091446 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" podUID="0fe9fa73-5f1d-47ea-b303-869b3392ac75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 31 13:09:18.692068 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:18.691638 6841 patch_prober.go:29] interesting pod/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 31 13:09:18.692068 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:18.691712 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" podUID="c85bee9f-4267-44ec-a03f-0b97c6b95d24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 31 13:09:18.798928 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:18.798717 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:19.089538 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:19.089505 6841 patch_prober.go:29] interesting pod/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 31 13:09:19.089740 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:19.089555 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" podUID="0fe9fa73-5f1d-47ea-b303-869b3392ac75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 31 13:09:19.735373 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:19.735315 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:09:19.738271 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:19.738241 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:09:19.741828 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:19.741787 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:20.035854 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:09:20.035772 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:09:20.140519 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:20.138637 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:09:20.140519 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:20.140343 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:09:20.173471 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:20.173437 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:09:20.175852 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:20.175821 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:20.645200 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:20.645163 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:21.350666 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:21.350632 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh" Oct 31 13:09:21.363661 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:21.363633 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh" Oct 31 13:09:21.536657 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:21.536614 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:21.830293 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:21.830266 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh" Oct 31 13:09:22.488013 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:22.487936 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:23.202173 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:23.202133 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:23.970819 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:23.967952 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:24.534225 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:24.534179 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw" Oct 31 13:09:24.547075 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:24.547045 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw" Oct 31 13:09:24.790677 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:24.790591 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:24.853293 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:24.853263 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw" Oct 31 13:09:25.573005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:25.572636 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:25.799005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:25.795180 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:26.097822 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:26.097782 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:27.044010 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:27.041941 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:27.822797 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:27.822756 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:28.518011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:28.517381 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:28.825306 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:28.825218 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:29.386949 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:29.386907 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:30.220665 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:30.220629 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:31.062961 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:31.062927 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:31.840284 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:31.838503 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:32.774191 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:32.774154 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:33.526252 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:33.526155 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:34.208963 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:34.208927 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:34.909768 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:34.905958 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:35.654176 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:35.654089 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:35.734164 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:35.734120 6841 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:09:35.734399 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:35.734173 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" podUID="4f2bc576-3d6a-44d0-b30a-3eae4f627961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:09:36.000045 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:36.000013 6841 generic.go:356] "Generic (PLEG): container finished" podID="4f2bc576-3d6a-44d0-b30a-3eae4f627961" containerID="87e55479dcc685d7818ca1d579dbd3fa589e7cccc2fc22ec0f440007157d5a66" exitCode=0 Oct 31 13:09:36.000534 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:36.000092 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" event={"ID":"4f2bc576-3d6a-44d0-b30a-3eae4f627961","Type":"ContainerDied","Data":"87e55479dcc685d7818ca1d579dbd3fa589e7cccc2fc22ec0f440007157d5a66"} Oct 31 13:09:36.000654 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:36.000633 6841 scope.go:122] "RemoveContainer" containerID="87e55479dcc685d7818ca1d579dbd3fa589e7cccc2fc22ec0f440007157d5a66" Oct 31 13:09:36.002361 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:36.002334 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:09:36.002450 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:36.002401 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:09:36.002450 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:36.002435 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:09:36.527523 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:36.527485 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:36.765004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:36.760826 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:37.016515 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:37.016350 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" event={"ID":"4f2bc576-3d6a-44d0-b30a-3eae4f627961","Type":"ContainerStarted","Data":"dc80ed599d793dd32f95edbd08a7dfff0423e1452a6b917be6d6fa28bca0edc4"} Oct 31 13:09:37.280782 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:37.280674 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:37.484584 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:37.484547 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:37.618217 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:37.618131 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:09:37.618368 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:37.618320 6841 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 31 13:09:37.618412 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:37.618376 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" podUID="4f2bc576-3d6a-44d0-b30a-3eae4f627961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 31 13:09:37.683847 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:37.683804 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:38.225006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:38.224772 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:38.618233 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:38.618151 6841 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 31 13:09:38.618233 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:38.618199 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" podUID="4f2bc576-3d6a-44d0-b30a-3eae4f627961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 31 13:09:38.926322 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:38.917324 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:39.618379 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:39.618344 6841 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 31 13:09:39.618808 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:39.618399 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" podUID="4f2bc576-3d6a-44d0-b30a-3eae4f627961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 31 13:09:39.806110 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:39.806021 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:40.422400 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:40.422360 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:40.617712 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:40.617677 6841 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 31 13:09:40.617890 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:40.617731 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" podUID="4f2bc576-3d6a-44d0-b30a-3eae4f627961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 31 13:09:40.993568 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:40.993534 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:41.264059 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:41.258577 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:41.618055 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:41.617960 6841 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 31 13:09:41.618055 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:41.618031 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" podUID="4f2bc576-3d6a-44d0-b30a-3eae4f627961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 31 13:09:42.357014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:42.354552 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:42.627296 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:42.627202 6841 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 31 13:09:42.627557 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:42.627529 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" podUID="4f2bc576-3d6a-44d0-b30a-3eae4f627961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 31 13:09:43.597251 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:43.596559 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:43.623487 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:43.623452 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:09:43.624281 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:43.624247 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:09:43.625024 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:43.624966 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:09:44.569533 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:44.569497 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:44.787955 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:44.787921 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:45.006053 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:45.006019 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:45.337249 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:45.337159 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:45.571668 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:45.571626 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk"] Oct 31 13:09:45.571936 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:45.571907 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" podUID="913b94e9-bc62-42b9-ad68-2fab683a9593" containerName="istio-proxy" containerID="cri-o://0ff971f5a7c643814dbb2f222e94de703925055e2a202eb210ace7ecb09502c2" gracePeriod=30 Oct 31 13:09:45.716086 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:45.699496 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:45.899047 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:45.899016 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:46.333305 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:46.333271 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:46.555558 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:46.554481 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:46.618669 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:46.618582 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw"] Oct 31 13:09:46.618843 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:46.618808 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" podUID="322f29ac-1b41-4e39-b296-bdedc1fd2690" containerName="istio-proxy" containerID="cri-o://80bdaa0e243ce1c5db3afa80c9db27bcef9909a61b8bfcccf98eca45cb0b0af6" gracePeriod=30 Oct 31 13:09:46.921061 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:46.919204 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:47.152069 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:47.152025 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:47.633168 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:47.633133 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:48.560253 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:48.560216 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:48.805088 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:48.805057 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:49.037116 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:49.037081 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:49.290405 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:49.290306 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x"] Oct 31 13:09:49.290571 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:49.290546 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" podUID="2b7734fb-74f8-4eed-bd5f-415d1c6e51ec" containerName="istio-proxy" containerID="cri-o://5fde765dd6300f6c8058f87b1315c8ad4bea25894bfd978e266363ada660fd71" gracePeriod=30 Oct 31 13:09:49.620518 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:49.620304 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:49.655219 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:49.655156 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c"] Oct 31 13:09:49.662534 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:49.662224 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c" Oct 31 13:09:49.673556 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:49.673532 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-95dfcc\"/\"openshift-service-ca.crt\"" Oct 31 13:09:49.673841 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:49.673824 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-95dfcc\"/\"default-dockercfg-mvcj4\"" Oct 31 13:09:49.674680 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:49.674659 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-95dfcc\"/\"kube-root-ca.crt\"" Oct 31 13:09:49.675079 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:49.675060 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-95dfcc\"/\"noten-model-95dfcc-kserve-self-signed-certs\"" Oct 31 13:09:49.679743 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:49.679720 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c"] Oct 31 13:09:49.720931 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:49.720896 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52319b98-c78a-4293-aa0d-17ab4d452d40-tmp-dir\") pod \"noten-model-95dfcc-kserve-7d686f85d5-6ck4c\" (UID: \"52319b98-c78a-4293-aa0d-17ab4d452d40\") " pod="ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c" Oct 31 13:09:49.721106 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:49.721021 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52319b98-c78a-4293-aa0d-17ab4d452d40-tls-certs\") pod \"noten-model-95dfcc-kserve-7d686f85d5-6ck4c\" (UID: \"52319b98-c78a-4293-aa0d-17ab4d452d40\") " pod="ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c" Oct 31 13:09:49.721245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:49.721225 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbwr9\" (UniqueName: \"kubernetes.io/projected/52319b98-c78a-4293-aa0d-17ab4d452d40-kube-api-access-mbwr9\") pod \"noten-model-95dfcc-kserve-7d686f85d5-6ck4c\" (UID: \"52319b98-c78a-4293-aa0d-17ab4d452d40\") " pod="ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c" Oct 31 13:09:49.721365 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:49.721346 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52319b98-c78a-4293-aa0d-17ab4d452d40-model-cache\") pod \"noten-model-95dfcc-kserve-7d686f85d5-6ck4c\" (UID: \"52319b98-c78a-4293-aa0d-17ab4d452d40\") " pod="ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c" Oct 31 13:09:49.721441 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:49.721422 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52319b98-c78a-4293-aa0d-17ab4d452d40-home\") pod \"noten-model-95dfcc-kserve-7d686f85d5-6ck4c\" (UID: \"52319b98-c78a-4293-aa0d-17ab4d452d40\") " pod="ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c" Oct 31 13:09:49.721615 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:49.721588 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52319b98-c78a-4293-aa0d-17ab4d452d40-dshm\") pod \"noten-model-95dfcc-kserve-7d686f85d5-6ck4c\" (UID: \"52319b98-c78a-4293-aa0d-17ab4d452d40\") " pod="ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c" Oct 31 13:09:49.823070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:49.823039 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52319b98-c78a-4293-aa0d-17ab4d452d40-tmp-dir\") pod \"noten-model-95dfcc-kserve-7d686f85d5-6ck4c\" (UID: \"52319b98-c78a-4293-aa0d-17ab4d452d40\") " pod="ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c" Oct 31 13:09:49.823263 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:49.823094 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52319b98-c78a-4293-aa0d-17ab4d452d40-tls-certs\") pod \"noten-model-95dfcc-kserve-7d686f85d5-6ck4c\" (UID: \"52319b98-c78a-4293-aa0d-17ab4d452d40\") " pod="ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c" Oct 31 13:09:49.823263 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:49.823174 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbwr9\" (UniqueName: \"kubernetes.io/projected/52319b98-c78a-4293-aa0d-17ab4d452d40-kube-api-access-mbwr9\") pod \"noten-model-95dfcc-kserve-7d686f85d5-6ck4c\" (UID: \"52319b98-c78a-4293-aa0d-17ab4d452d40\") " pod="ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c" Oct 31 13:09:49.823263 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:49.823195 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52319b98-c78a-4293-aa0d-17ab4d452d40-model-cache\") pod \"noten-model-95dfcc-kserve-7d686f85d5-6ck4c\" (UID: \"52319b98-c78a-4293-aa0d-17ab4d452d40\") " pod="ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c" Oct 31 13:09:49.823263 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:49.823240 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52319b98-c78a-4293-aa0d-17ab4d452d40-home\") pod \"noten-model-95dfcc-kserve-7d686f85d5-6ck4c\" (UID: \"52319b98-c78a-4293-aa0d-17ab4d452d40\") " pod="ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c" Oct 31 13:09:49.823523 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:49.823391 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52319b98-c78a-4293-aa0d-17ab4d452d40-dshm\") pod \"noten-model-95dfcc-kserve-7d686f85d5-6ck4c\" (UID: \"52319b98-c78a-4293-aa0d-17ab4d452d40\") " pod="ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c" Oct 31 13:09:49.823523 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:49.823507 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52319b98-c78a-4293-aa0d-17ab4d452d40-tmp-dir\") pod \"noten-model-95dfcc-kserve-7d686f85d5-6ck4c\" (UID: \"52319b98-c78a-4293-aa0d-17ab4d452d40\") " pod="ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c" Oct 31 13:09:49.823630 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:49.823524 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52319b98-c78a-4293-aa0d-17ab4d452d40-model-cache\") pod \"noten-model-95dfcc-kserve-7d686f85d5-6ck4c\" (UID: \"52319b98-c78a-4293-aa0d-17ab4d452d40\") " pod="ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c" Oct 31 13:09:49.823630 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:49.823609 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52319b98-c78a-4293-aa0d-17ab4d452d40-home\") pod \"noten-model-95dfcc-kserve-7d686f85d5-6ck4c\" (UID: \"52319b98-c78a-4293-aa0d-17ab4d452d40\") " pod="ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c" Oct 31 13:09:49.825604 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:49.825568 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52319b98-c78a-4293-aa0d-17ab4d452d40-dshm\") pod \"noten-model-95dfcc-kserve-7d686f85d5-6ck4c\" (UID: \"52319b98-c78a-4293-aa0d-17ab4d452d40\") " pod="ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c" Oct 31 13:09:49.825746 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:49.825724 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52319b98-c78a-4293-aa0d-17ab4d452d40-tls-certs\") pod \"noten-model-95dfcc-kserve-7d686f85d5-6ck4c\" (UID: \"52319b98-c78a-4293-aa0d-17ab4d452d40\") " pod="ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c" Oct 31 13:09:49.838507 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:49.838485 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbwr9\" (UniqueName: \"kubernetes.io/projected/52319b98-c78a-4293-aa0d-17ab4d452d40-kube-api-access-mbwr9\") pod \"noten-model-95dfcc-kserve-7d686f85d5-6ck4c\" (UID: \"52319b98-c78a-4293-aa0d-17ab4d452d40\") " pod="ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c" Oct 31 13:09:49.881063 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:49.872028 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:49.992204 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:49.987735 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c" Oct 31 13:09:50.102107 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:09:50.102066 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:09:50.354902 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:09:50.354868 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52319b98_c78a_4293_aa0d_17ab4d452d40.slice/crio-6913d4760092b922162cbc9898ce2fcd5ab611bc80d0f55e7716ad897a53ef0a WatchSource:0}: Error finding container 6913d4760092b922162cbc9898ce2fcd5ab611bc80d0f55e7716ad897a53ef0a: Status 404 returned error can't find the container with id 6913d4760092b922162cbc9898ce2fcd5ab611bc80d0f55e7716ad897a53ef0a Oct 31 13:09:50.355358 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:50.355334 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c"] Oct 31 13:09:50.517828 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:50.517778 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:50.871224 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:50.871199 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:09:50.994636 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:50.994597 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/913b94e9-bc62-42b9-ad68-2fab683a9593-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/913b94e9-bc62-42b9-ad68-2fab683a9593-credential-socket\") pod \"913b94e9-bc62-42b9-ad68-2fab683a9593\" (UID: \"913b94e9-bc62-42b9-ad68-2fab683a9593\") " Oct 31 13:09:50.994823 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:50.994677 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/913b94e9-bc62-42b9-ad68-2fab683a9593-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/913b94e9-bc62-42b9-ad68-2fab683a9593-workload-socket\") pod \"913b94e9-bc62-42b9-ad68-2fab683a9593\" (UID: \"913b94e9-bc62-42b9-ad68-2fab683a9593\") " Oct 31 13:09:50.994823 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:50.994716 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/913b94e9-bc62-42b9-ad68-2fab683a9593-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/913b94e9-bc62-42b9-ad68-2fab683a9593-workload-certs\") pod \"913b94e9-bc62-42b9-ad68-2fab683a9593\" (UID: \"913b94e9-bc62-42b9-ad68-2fab683a9593\") " Oct 31 13:09:50.994823 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:50.994765 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/913b94e9-bc62-42b9-ad68-2fab683a9593-istio-token\" (UniqueName: \"kubernetes.io/projected/913b94e9-bc62-42b9-ad68-2fab683a9593-istio-token\") pod \"913b94e9-bc62-42b9-ad68-2fab683a9593\" (UID: \"913b94e9-bc62-42b9-ad68-2fab683a9593\") " Oct 31 13:09:50.994823 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:50.994804 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/913b94e9-bc62-42b9-ad68-2fab683a9593-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/913b94e9-bc62-42b9-ad68-2fab683a9593-istio-envoy\") pod \"913b94e9-bc62-42b9-ad68-2fab683a9593\" (UID: \"913b94e9-bc62-42b9-ad68-2fab683a9593\") " Oct 31 13:09:50.995071 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:50.994891 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/913b94e9-bc62-42b9-ad68-2fab683a9593-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/913b94e9-bc62-42b9-ad68-2fab683a9593-istio-data\") pod \"913b94e9-bc62-42b9-ad68-2fab683a9593\" (UID: \"913b94e9-bc62-42b9-ad68-2fab683a9593\") " Oct 31 13:09:50.995071 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:50.994901 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/913b94e9-bc62-42b9-ad68-2fab683a9593-credential-socket" pod "913b94e9-bc62-42b9-ad68-2fab683a9593" (UID: "913b94e9-bc62-42b9-ad68-2fab683a9593"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:09:50.995177 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:50.995078 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/913b94e9-bc62-42b9-ad68-2fab683a9593-workload-socket" pod "913b94e9-bc62-42b9-ad68-2fab683a9593" (UID: "913b94e9-bc62-42b9-ad68-2fab683a9593"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:09:50.995177 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:50.995100 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/913b94e9-bc62-42b9-ad68-2fab683a9593-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/913b94e9-bc62-42b9-ad68-2fab683a9593-istiod-ca-cert\") pod \"913b94e9-bc62-42b9-ad68-2fab683a9593\" (UID: \"913b94e9-bc62-42b9-ad68-2fab683a9593\") " Oct 31 13:09:50.995301 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:50.995103 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/913b94e9-bc62-42b9-ad68-2fab683a9593-workload-certs" pod "913b94e9-bc62-42b9-ad68-2fab683a9593" (UID: "913b94e9-bc62-42b9-ad68-2fab683a9593"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:09:50.995301 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:50.995193 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/913b94e9-bc62-42b9-ad68-2fab683a9593-istio-data" pod "913b94e9-bc62-42b9-ad68-2fab683a9593" (UID: "913b94e9-bc62-42b9-ad68-2fab683a9593"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:09:50.995301 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:50.995266 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/913b94e9-bc62-42b9-ad68-2fab683a9593-kube-api-access-qlktc\" (UniqueName: \"kubernetes.io/projected/913b94e9-bc62-42b9-ad68-2fab683a9593-kube-api-access-qlktc\") pod \"913b94e9-bc62-42b9-ad68-2fab683a9593\" (UID: \"913b94e9-bc62-42b9-ad68-2fab683a9593\") " Oct 31 13:09:50.995443 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:50.995407 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/913b94e9-bc62-42b9-ad68-2fab683a9593-istiod-ca-cert" pod "913b94e9-bc62-42b9-ad68-2fab683a9593" (UID: "913b94e9-bc62-42b9-ad68-2fab683a9593"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:09:50.995503 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:50.995448 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/913b94e9-bc62-42b9-ad68-2fab683a9593-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/913b94e9-bc62-42b9-ad68-2fab683a9593-istio-podinfo\") pod \"913b94e9-bc62-42b9-ad68-2fab683a9593\" (UID: \"913b94e9-bc62-42b9-ad68-2fab683a9593\") " Oct 31 13:09:50.996272 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:50.996244 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/913b94e9-bc62-42b9-ad68-2fab683a9593-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:50.996423 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:50.996276 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/913b94e9-bc62-42b9-ad68-2fab683a9593-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:50.996423 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:50.996293 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/913b94e9-bc62-42b9-ad68-2fab683a9593-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:50.996423 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:50.996306 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/913b94e9-bc62-42b9-ad68-2fab683a9593-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:50.996423 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:50.996321 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/913b94e9-bc62-42b9-ad68-2fab683a9593-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:50.997383 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:50.997361 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/913b94e9-bc62-42b9-ad68-2fab683a9593-istio-envoy" pod "913b94e9-bc62-42b9-ad68-2fab683a9593" (UID: "913b94e9-bc62-42b9-ad68-2fab683a9593"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:09:50.997477 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:50.997389 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/913b94e9-bc62-42b9-ad68-2fab683a9593-istio-token" pod "913b94e9-bc62-42b9-ad68-2fab683a9593" (UID: "913b94e9-bc62-42b9-ad68-2fab683a9593"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:09:50.997719 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:50.997698 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/913b94e9-bc62-42b9-ad68-2fab683a9593-kube-api-access-qlktc" pod "913b94e9-bc62-42b9-ad68-2fab683a9593" (UID: "913b94e9-bc62-42b9-ad68-2fab683a9593"). InnerVolumeSpecName "kube-api-access-qlktc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:09:50.997971 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:50.997952 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/913b94e9-bc62-42b9-ad68-2fab683a9593-istio-podinfo" pod "913b94e9-bc62-42b9-ad68-2fab683a9593" (UID: "913b94e9-bc62-42b9-ad68-2fab683a9593"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:09:51.097705 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:51.097632 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlktc\" (UniqueName: \"kubernetes.io/projected/913b94e9-bc62-42b9-ad68-2fab683a9593-kube-api-access-qlktc\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:51.097705 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:51.097661 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/913b94e9-bc62-42b9-ad68-2fab683a9593-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:51.097705 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:51.097671 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/913b94e9-bc62-42b9-ad68-2fab683a9593-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:51.097705 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:51.097681 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/913b94e9-bc62-42b9-ad68-2fab683a9593-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:51.196622 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:51.196589 6841 generic.go:356] "Generic (PLEG): container finished" podID="913b94e9-bc62-42b9-ad68-2fab683a9593" containerID="0ff971f5a7c643814dbb2f222e94de703925055e2a202eb210ace7ecb09502c2" exitCode=0 Oct 31 13:09:51.196826 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:51.196647 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" Oct 31 13:09:51.196826 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:51.196667 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" event={"ID":"913b94e9-bc62-42b9-ad68-2fab683a9593","Type":"ContainerDied","Data":"0ff971f5a7c643814dbb2f222e94de703925055e2a202eb210ace7ecb09502c2"} Oct 31 13:09:51.196826 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:51.196725 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk" event={"ID":"913b94e9-bc62-42b9-ad68-2fab683a9593","Type":"ContainerDied","Data":"ef22627763281e4ab8cd1a60ef92b0cbc1bc81e3147e4c9f7a634de8fc315bfc"} Oct 31 13:09:51.196826 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:51.196743 6841 scope.go:122] "RemoveContainer" containerID="0ff971f5a7c643814dbb2f222e94de703925055e2a202eb210ace7ecb09502c2" Oct 31 13:09:51.199583 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:51.199559 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c" event={"ID":"52319b98-c78a-4293-aa0d-17ab4d452d40","Type":"ContainerStarted","Data":"da8df97785863b7f831a2934016e56796e3fba84037fca60e6f536da378849c2"} Oct 31 13:09:51.199727 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:51.199593 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c" event={"ID":"52319b98-c78a-4293-aa0d-17ab4d452d40","Type":"ContainerStarted","Data":"6913d4760092b922162cbc9898ce2fcd5ab611bc80d0f55e7716ad897a53ef0a"} Oct 31 13:09:51.210786 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:51.210769 6841 scope.go:122] "RemoveContainer" containerID="a6a0b04c2bd005394a9dd11838d16ed7be93d59a2e75836d9a9ac3a8d690f495" Oct 31 13:09:51.224322 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:51.224303 6841 scope.go:122] "RemoveContainer" containerID="0ff971f5a7c643814dbb2f222e94de703925055e2a202eb210ace7ecb09502c2" Oct 31 13:09:51.225947 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:09:51.224692 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ff971f5a7c643814dbb2f222e94de703925055e2a202eb210ace7ecb09502c2\": container with ID starting with 0ff971f5a7c643814dbb2f222e94de703925055e2a202eb210ace7ecb09502c2 not found: ID does not exist" containerID="0ff971f5a7c643814dbb2f222e94de703925055e2a202eb210ace7ecb09502c2" Oct 31 13:09:51.225947 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:51.224725 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff971f5a7c643814dbb2f222e94de703925055e2a202eb210ace7ecb09502c2"} err="failed to get container status \"0ff971f5a7c643814dbb2f222e94de703925055e2a202eb210ace7ecb09502c2\": rpc error: code = NotFound desc = could not find container \"0ff971f5a7c643814dbb2f222e94de703925055e2a202eb210ace7ecb09502c2\": container with ID starting with 0ff971f5a7c643814dbb2f222e94de703925055e2a202eb210ace7ecb09502c2 not found: ID does not exist" Oct 31 13:09:51.225947 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:51.224751 6841 scope.go:122] "RemoveContainer" containerID="a6a0b04c2bd005394a9dd11838d16ed7be93d59a2e75836d9a9ac3a8d690f495" Oct 31 13:09:51.225947 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:09:51.225138 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6a0b04c2bd005394a9dd11838d16ed7be93d59a2e75836d9a9ac3a8d690f495\": container with ID starting with a6a0b04c2bd005394a9dd11838d16ed7be93d59a2e75836d9a9ac3a8d690f495 not found: ID does not exist" containerID="a6a0b04c2bd005394a9dd11838d16ed7be93d59a2e75836d9a9ac3a8d690f495" Oct 31 13:09:51.225947 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:51.225165 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6a0b04c2bd005394a9dd11838d16ed7be93d59a2e75836d9a9ac3a8d690f495"} err="failed to get container status \"a6a0b04c2bd005394a9dd11838d16ed7be93d59a2e75836d9a9ac3a8d690f495\": rpc error: code = NotFound desc = could not find container \"a6a0b04c2bd005394a9dd11838d16ed7be93d59a2e75836d9a9ac3a8d690f495\": container with ID starting with a6a0b04c2bd005394a9dd11838d16ed7be93d59a2e75836d9a9ac3a8d690f495 not found: ID does not exist" Oct 31 13:09:51.228226 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:51.228187 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c" podStartSLOduration=2.228174105 podStartE2EDuration="2.228174105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:09:51.224029758 +0000 UTC m=+2237.707152651" watchObservedRunningTime="2026-10-31 13:09:51.228174105 +0000 UTC m=+2237.711297005" Oct 31 13:09:51.244284 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:51.244258 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk"] Oct 31 13:09:51.248423 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:51.248397 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-6106a3-openshift-default-77f97df9f7-2rzzk"] Oct 31 13:09:51.329932 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:51.329899 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:51.644961 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:51.644933 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="913b94e9-bc62-42b9-ad68-2fab683a9593" path="/var/lib/kubelet/pods/913b94e9-bc62-42b9-ad68-2fab683a9593/volumes" Oct 31 13:09:51.909466 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:51.909443 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" Oct 31 13:09:51.992134 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:51.992104 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c" Oct 31 13:09:52.015947 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.015914 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/322f29ac-1b41-4e39-b296-bdedc1fd2690-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/322f29ac-1b41-4e39-b296-bdedc1fd2690-istio-envoy\") pod \"322f29ac-1b41-4e39-b296-bdedc1fd2690\" (UID: \"322f29ac-1b41-4e39-b296-bdedc1fd2690\") " Oct 31 13:09:52.016122 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.016086 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/322f29ac-1b41-4e39-b296-bdedc1fd2690-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/322f29ac-1b41-4e39-b296-bdedc1fd2690-istio-data\") pod \"322f29ac-1b41-4e39-b296-bdedc1fd2690\" (UID: \"322f29ac-1b41-4e39-b296-bdedc1fd2690\") " Oct 31 13:09:52.016122 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.016117 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/322f29ac-1b41-4e39-b296-bdedc1fd2690-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/322f29ac-1b41-4e39-b296-bdedc1fd2690-istiod-ca-cert\") pod \"322f29ac-1b41-4e39-b296-bdedc1fd2690\" (UID: \"322f29ac-1b41-4e39-b296-bdedc1fd2690\") " Oct 31 13:09:52.016202 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.016142 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/322f29ac-1b41-4e39-b296-bdedc1fd2690-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/322f29ac-1b41-4e39-b296-bdedc1fd2690-workload-socket\") pod \"322f29ac-1b41-4e39-b296-bdedc1fd2690\" (UID: \"322f29ac-1b41-4e39-b296-bdedc1fd2690\") " Oct 31 13:09:52.016202 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.016178 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/322f29ac-1b41-4e39-b296-bdedc1fd2690-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/322f29ac-1b41-4e39-b296-bdedc1fd2690-credential-socket\") pod \"322f29ac-1b41-4e39-b296-bdedc1fd2690\" (UID: \"322f29ac-1b41-4e39-b296-bdedc1fd2690\") " Oct 31 13:09:52.016278 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.016237 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/322f29ac-1b41-4e39-b296-bdedc1fd2690-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/322f29ac-1b41-4e39-b296-bdedc1fd2690-workload-certs\") pod \"322f29ac-1b41-4e39-b296-bdedc1fd2690\" (UID: \"322f29ac-1b41-4e39-b296-bdedc1fd2690\") " Oct 31 13:09:52.016325 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.016301 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/322f29ac-1b41-4e39-b296-bdedc1fd2690-kube-api-access-v78p9\" (UniqueName: \"kubernetes.io/projected/322f29ac-1b41-4e39-b296-bdedc1fd2690-kube-api-access-v78p9\") pod \"322f29ac-1b41-4e39-b296-bdedc1fd2690\" (UID: \"322f29ac-1b41-4e39-b296-bdedc1fd2690\") " Oct 31 13:09:52.016383 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.016342 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/322f29ac-1b41-4e39-b296-bdedc1fd2690-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/322f29ac-1b41-4e39-b296-bdedc1fd2690-istio-podinfo\") pod \"322f29ac-1b41-4e39-b296-bdedc1fd2690\" (UID: \"322f29ac-1b41-4e39-b296-bdedc1fd2690\") " Oct 31 13:09:52.016446 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.016413 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/322f29ac-1b41-4e39-b296-bdedc1fd2690-istio-token\" (UniqueName: \"kubernetes.io/projected/322f29ac-1b41-4e39-b296-bdedc1fd2690-istio-token\") pod \"322f29ac-1b41-4e39-b296-bdedc1fd2690\" (UID: \"322f29ac-1b41-4e39-b296-bdedc1fd2690\") " Oct 31 13:09:52.016446 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.016389 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/322f29ac-1b41-4e39-b296-bdedc1fd2690-istio-data" pod "322f29ac-1b41-4e39-b296-bdedc1fd2690" (UID: "322f29ac-1b41-4e39-b296-bdedc1fd2690"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:09:52.016553 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.016447 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/322f29ac-1b41-4e39-b296-bdedc1fd2690-workload-socket" pod "322f29ac-1b41-4e39-b296-bdedc1fd2690" (UID: "322f29ac-1b41-4e39-b296-bdedc1fd2690"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:09:52.016655 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.016633 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/322f29ac-1b41-4e39-b296-bdedc1fd2690-workload-certs" pod "322f29ac-1b41-4e39-b296-bdedc1fd2690" (UID: "322f29ac-1b41-4e39-b296-bdedc1fd2690"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:09:52.016655 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.016583 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/322f29ac-1b41-4e39-b296-bdedc1fd2690-istiod-ca-cert" pod "322f29ac-1b41-4e39-b296-bdedc1fd2690" (UID: "322f29ac-1b41-4e39-b296-bdedc1fd2690"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:09:52.016885 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.016673 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/322f29ac-1b41-4e39-b296-bdedc1fd2690-credential-socket" pod "322f29ac-1b41-4e39-b296-bdedc1fd2690" (UID: "322f29ac-1b41-4e39-b296-bdedc1fd2690"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:09:52.017309 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.017284 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/322f29ac-1b41-4e39-b296-bdedc1fd2690-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:52.017417 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.017310 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/322f29ac-1b41-4e39-b296-bdedc1fd2690-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:52.017417 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.017324 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/322f29ac-1b41-4e39-b296-bdedc1fd2690-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:52.017417 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.017337 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/322f29ac-1b41-4e39-b296-bdedc1fd2690-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:52.017417 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.017346 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/322f29ac-1b41-4e39-b296-bdedc1fd2690-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:52.018578 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.018558 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/322f29ac-1b41-4e39-b296-bdedc1fd2690-istio-envoy" pod "322f29ac-1b41-4e39-b296-bdedc1fd2690" (UID: "322f29ac-1b41-4e39-b296-bdedc1fd2690"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:09:52.018726 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.018711 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/322f29ac-1b41-4e39-b296-bdedc1fd2690-istio-token" pod "322f29ac-1b41-4e39-b296-bdedc1fd2690" (UID: "322f29ac-1b41-4e39-b296-bdedc1fd2690"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:09:52.019063 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.019038 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/322f29ac-1b41-4e39-b296-bdedc1fd2690-kube-api-access-v78p9" pod "322f29ac-1b41-4e39-b296-bdedc1fd2690" (UID: "322f29ac-1b41-4e39-b296-bdedc1fd2690"). InnerVolumeSpecName "kube-api-access-v78p9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:09:52.019126 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.019096 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/322f29ac-1b41-4e39-b296-bdedc1fd2690-istio-podinfo" pod "322f29ac-1b41-4e39-b296-bdedc1fd2690" (UID: "322f29ac-1b41-4e39-b296-bdedc1fd2690"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:09:52.124290 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.123070 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v78p9\" (UniqueName: \"kubernetes.io/projected/322f29ac-1b41-4e39-b296-bdedc1fd2690-kube-api-access-v78p9\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:52.124290 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.123116 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/322f29ac-1b41-4e39-b296-bdedc1fd2690-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:52.124290 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.123139 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/322f29ac-1b41-4e39-b296-bdedc1fd2690-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:52.124290 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.123160 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/322f29ac-1b41-4e39-b296-bdedc1fd2690-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:52.131838 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.131749 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:52.208369 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.208335 6841 generic.go:356] "Generic (PLEG): container finished" podID="322f29ac-1b41-4e39-b296-bdedc1fd2690" containerID="80bdaa0e243ce1c5db3afa80c9db27bcef9909a61b8bfcccf98eca45cb0b0af6" exitCode=0 Oct 31 13:09:52.208559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.208389 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" Oct 31 13:09:52.208559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.208420 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" event={"ID":"322f29ac-1b41-4e39-b296-bdedc1fd2690","Type":"ContainerDied","Data":"80bdaa0e243ce1c5db3afa80c9db27bcef9909a61b8bfcccf98eca45cb0b0af6"} Oct 31 13:09:52.208559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.208516 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw" event={"ID":"322f29ac-1b41-4e39-b296-bdedc1fd2690","Type":"ContainerDied","Data":"1c0770280ee40c90ba484d67982010540aa213ca4f2e3bbcae9ac5e427d6aba0"} Oct 31 13:09:52.208559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.208544 6841 scope.go:122] "RemoveContainer" containerID="80bdaa0e243ce1c5db3afa80c9db27bcef9909a61b8bfcccf98eca45cb0b0af6" Oct 31 13:09:52.220539 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.220516 6841 scope.go:122] "RemoveContainer" containerID="80bdaa0e243ce1c5db3afa80c9db27bcef9909a61b8bfcccf98eca45cb0b0af6" Oct 31 13:09:52.220828 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:09:52.220808 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80bdaa0e243ce1c5db3afa80c9db27bcef9909a61b8bfcccf98eca45cb0b0af6\": container with ID starting with 80bdaa0e243ce1c5db3afa80c9db27bcef9909a61b8bfcccf98eca45cb0b0af6 not found: ID does not exist" containerID="80bdaa0e243ce1c5db3afa80c9db27bcef9909a61b8bfcccf98eca45cb0b0af6" Oct 31 13:09:52.220908 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.220844 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80bdaa0e243ce1c5db3afa80c9db27bcef9909a61b8bfcccf98eca45cb0b0af6"} err="failed to get container status \"80bdaa0e243ce1c5db3afa80c9db27bcef9909a61b8bfcccf98eca45cb0b0af6\": rpc error: code = NotFound desc = could not find container \"80bdaa0e243ce1c5db3afa80c9db27bcef9909a61b8bfcccf98eca45cb0b0af6\": container with ID starting with 80bdaa0e243ce1c5db3afa80c9db27bcef9909a61b8bfcccf98eca45cb0b0af6 not found: ID does not exist" Oct 31 13:09:52.235504 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.235474 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw"] Oct 31 13:09:52.244018 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.243962 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-zlhxw"] Oct 31 13:09:52.903767 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:52.903733 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:53.644034 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:53.643999 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="322f29ac-1b41-4e39-b296-bdedc1fd2690" path="/var/lib/kubelet/pods/322f29ac-1b41-4e39-b296-bdedc1fd2690/volumes" Oct 31 13:09:53.751540 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:53.750190 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:54.557777 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:54.555567 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:54.648760 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:54.648731 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" Oct 31 13:09:54.766323 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:54.766289 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-credential-socket\") pod \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\" (UID: \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\") " Oct 31 13:09:54.766498 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:54.766383 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-workload-socket\") pod \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\" (UID: \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\") " Oct 31 13:09:54.766498 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:54.766418 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-workload-certs\") pod \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\" (UID: \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\") " Oct 31 13:09:54.766498 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:54.766479 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-istio-data\") pod \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\" (UID: \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\") " Oct 31 13:09:54.766670 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:54.766519 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-kube-api-access-trqtq\" (UniqueName: \"kubernetes.io/projected/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-kube-api-access-trqtq\") pod \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\" (UID: \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\") " Oct 31 13:09:54.766670 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:54.766546 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-credential-socket" pod "2b7734fb-74f8-4eed-bd5f-415d1c6e51ec" (UID: "2b7734fb-74f8-4eed-bd5f-415d1c6e51ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:09:54.766670 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:54.766560 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-istiod-ca-cert\") pod \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\" (UID: \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\") " Oct 31 13:09:54.766670 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:54.766610 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-istio-podinfo\") pod \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\" (UID: \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\") " Oct 31 13:09:54.766888 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:54.766698 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-istio-token\" (UniqueName: \"kubernetes.io/projected/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-istio-token\") pod \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\" (UID: \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\") " Oct 31 13:09:54.766888 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:54.766712 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-workload-socket" pod "2b7734fb-74f8-4eed-bd5f-415d1c6e51ec" (UID: "2b7734fb-74f8-4eed-bd5f-415d1c6e51ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:09:54.766888 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:54.766789 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-istio-envoy\") pod \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\" (UID: \"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec\") " Oct 31 13:09:54.766888 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:54.766800 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-istio-data" pod "2b7734fb-74f8-4eed-bd5f-415d1c6e51ec" (UID: "2b7734fb-74f8-4eed-bd5f-415d1c6e51ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:09:54.766888 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:54.766816 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-workload-certs" pod "2b7734fb-74f8-4eed-bd5f-415d1c6e51ec" (UID: "2b7734fb-74f8-4eed-bd5f-415d1c6e51ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:09:54.767197 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:54.766894 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-istiod-ca-cert" pod "2b7734fb-74f8-4eed-bd5f-415d1c6e51ec" (UID: "2b7734fb-74f8-4eed-bd5f-415d1c6e51ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:09:54.768110 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:54.767942 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:54.768110 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:54.767970 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:54.768110 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:54.768003 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:54.768110 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:54.768016 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:54.768110 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:54.768028 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:54.768844 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:54.768822 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-istio-podinfo" pod "2b7734fb-74f8-4eed-bd5f-415d1c6e51ec" (UID: "2b7734fb-74f8-4eed-bd5f-415d1c6e51ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:09:54.769098 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:54.769074 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-istio-envoy" pod "2b7734fb-74f8-4eed-bd5f-415d1c6e51ec" (UID: "2b7734fb-74f8-4eed-bd5f-415d1c6e51ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:09:54.769360 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:54.769342 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-kube-api-access-trqtq" pod "2b7734fb-74f8-4eed-bd5f-415d1c6e51ec" (UID: "2b7734fb-74f8-4eed-bd5f-415d1c6e51ec"). InnerVolumeSpecName "kube-api-access-trqtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:09:54.769412 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:54.769369 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-istio-token" pod "2b7734fb-74f8-4eed-bd5f-415d1c6e51ec" (UID: "2b7734fb-74f8-4eed-bd5f-415d1c6e51ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:09:54.869194 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:54.869118 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:54.869194 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:54.869147 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trqtq\" (UniqueName: \"kubernetes.io/projected/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-kube-api-access-trqtq\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:54.869194 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:54.869157 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:54.869194 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:54.869165 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:55.235663 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:55.235634 6841 generic.go:356] "Generic (PLEG): container finished" podID="2b7734fb-74f8-4eed-bd5f-415d1c6e51ec" containerID="5fde765dd6300f6c8058f87b1315c8ad4bea25894bfd978e266363ada660fd71" exitCode=0 Oct 31 13:09:55.235823 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:55.235684 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" Oct 31 13:09:55.235823 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:55.235722 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" event={"ID":"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec","Type":"ContainerDied","Data":"5fde765dd6300f6c8058f87b1315c8ad4bea25894bfd978e266363ada660fd71"} Oct 31 13:09:55.235823 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:55.235774 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x" event={"ID":"2b7734fb-74f8-4eed-bd5f-415d1c6e51ec","Type":"ContainerDied","Data":"9764160cf47e4c545e8a35672d118cf5d506052711a4c0d21466b99e1c81d124"} Oct 31 13:09:55.235823 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:55.235796 6841 scope.go:122] "RemoveContainer" containerID="5fde765dd6300f6c8058f87b1315c8ad4bea25894bfd978e266363ada660fd71" Oct 31 13:09:55.292396 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:55.292180 6841 scope.go:122] "RemoveContainer" containerID="5fde765dd6300f6c8058f87b1315c8ad4bea25894bfd978e266363ada660fd71" Oct 31 13:09:55.300002 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:09:55.296436 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fde765dd6300f6c8058f87b1315c8ad4bea25894bfd978e266363ada660fd71\": container with ID starting with 5fde765dd6300f6c8058f87b1315c8ad4bea25894bfd978e266363ada660fd71 not found: ID does not exist" containerID="5fde765dd6300f6c8058f87b1315c8ad4bea25894bfd978e266363ada660fd71" Oct 31 13:09:55.300002 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:55.296485 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fde765dd6300f6c8058f87b1315c8ad4bea25894bfd978e266363ada660fd71"} err="failed to get container status \"5fde765dd6300f6c8058f87b1315c8ad4bea25894bfd978e266363ada660fd71\": rpc error: code = NotFound desc = could not find container \"5fde765dd6300f6c8058f87b1315c8ad4bea25894bfd978e266363ada660fd71\": container with ID starting with 5fde765dd6300f6c8058f87b1315c8ad4bea25894bfd978e266363ada660fd71 not found: ID does not exist" Oct 31 13:09:55.322005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:55.318933 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x"] Oct 31 13:09:55.326049 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:55.324456 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:55.330334 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:55.330065 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5467x"] Oct 31 13:09:55.643937 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:55.643860 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b7734fb-74f8-4eed-bd5f-415d1c6e51ec" path="/var/lib/kubelet/pods/2b7734fb-74f8-4eed-bd5f-415d1c6e51ec/volumes" Oct 31 13:09:56.104213 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:56.101914 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:56.832501 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:56.831579 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:57.611614 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:57.611578 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:58.326701 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:58.326635 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:59.095031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:59.091890 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:59.885105 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:59.884677 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:09:59.989166 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:09:59.989129 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c" Oct 31 13:10:00.002280 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:00.002252 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c" Oct 31 13:10:00.354655 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:00.354620 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c" Oct 31 13:10:00.621456 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:00.621357 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:01.026580 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:01.026550 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c"] Oct 31 13:10:01.192898 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:01.192853 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:01.441581 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:01.441457 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:01.664232 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:10:01.649807 6841 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-self-signed-certs: secret "noten-model-95dfcc-kserve-self-signed-certs" not found Oct 31 13:10:01.664232 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:10:01.649881 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52319b98-c78a-4293-aa0d-17ab4d452d40-tls-certs podName:52319b98-c78a-4293-aa0d-17ab4d452d40 nodeName:}" failed. No retries permitted until 2026-10-31 13:10:02.149861351 +0000 UTC m=+2248.632984244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/52319b98-c78a-4293-aa0d-17ab4d452d40-tls-certs") pod "noten-model-95dfcc-kserve-7d686f85d5-6ck4c" (UID: "52319b98-c78a-4293-aa0d-17ab4d452d40") : secret "noten-model-95dfcc-kserve-self-signed-certs" not found Oct 31 13:10:01.745572 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:01.745534 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:01.949241 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:01.949204 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:02.157514 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:10:02.157429 6841 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-self-signed-certs: secret "noten-model-95dfcc-kserve-self-signed-certs" not found Oct 31 13:10:02.157706 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:10:02.157515 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52319b98-c78a-4293-aa0d-17ab4d452d40-tls-certs podName:52319b98-c78a-4293-aa0d-17ab4d452d40 nodeName:}" failed. No retries permitted until 2026-10-31 13:10:03.157498924 +0000 UTC m=+2249.640621828 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/52319b98-c78a-4293-aa0d-17ab4d452d40-tls-certs") pod "noten-model-95dfcc-kserve-7d686f85d5-6ck4c" (UID: "52319b98-c78a-4293-aa0d-17ab4d452d40") : secret "noten-model-95dfcc-kserve-self-signed-certs" not found Oct 31 13:10:02.394776 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:02.394736 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:03.145662 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:03.141031 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:03.176032 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:10:03.175910 6841 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-self-signed-certs: secret "noten-model-95dfcc-kserve-self-signed-certs" not found Oct 31 13:10:03.176032 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:10:03.175990 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52319b98-c78a-4293-aa0d-17ab4d452d40-tls-certs podName:52319b98-c78a-4293-aa0d-17ab4d452d40 nodeName:}" failed. No retries permitted until 2026-10-31 13:10:05.175963351 +0000 UTC m=+2251.659086247 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/52319b98-c78a-4293-aa0d-17ab4d452d40-tls-certs") pod "noten-model-95dfcc-kserve-7d686f85d5-6ck4c" (UID: "52319b98-c78a-4293-aa0d-17ab4d452d40") : secret "noten-model-95dfcc-kserve-self-signed-certs" not found Oct 31 13:10:03.832148 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:03.832113 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:04.550342 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:04.550308 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:05.216785 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:10:05.216756 6841 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-self-signed-certs: secret "noten-model-95dfcc-kserve-self-signed-certs" not found Oct 31 13:10:05.216973 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:10:05.216834 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52319b98-c78a-4293-aa0d-17ab4d452d40-tls-certs podName:52319b98-c78a-4293-aa0d-17ab4d452d40 nodeName:}" failed. No retries permitted until 2026-10-31 13:10:09.216820354 +0000 UTC m=+2255.699943246 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/52319b98-c78a-4293-aa0d-17ab4d452d40-tls-certs") pod "noten-model-95dfcc-kserve-7d686f85d5-6ck4c" (UID: "52319b98-c78a-4293-aa0d-17ab4d452d40") : secret "noten-model-95dfcc-kserve-self-signed-certs" not found Oct 31 13:10:05.414546 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:05.411508 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:06.419744 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:06.419704 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:06.857413 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:06.857373 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:07.236836 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:07.234687 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:08.030537 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:08.030357 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:08.745595 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:08.745280 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:09.314128 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:10:09.314093 6841 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-self-signed-certs: secret "noten-model-95dfcc-kserve-self-signed-certs" not found Oct 31 13:10:09.314623 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:10:09.314188 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52319b98-c78a-4293-aa0d-17ab4d452d40-tls-certs podName:52319b98-c78a-4293-aa0d-17ab4d452d40 nodeName:}" failed. No retries permitted until 2026-10-31 13:10:17.314169044 +0000 UTC m=+2263.797291923 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/52319b98-c78a-4293-aa0d-17ab4d452d40-tls-certs") pod "noten-model-95dfcc-kserve-7d686f85d5-6ck4c" (UID: "52319b98-c78a-4293-aa0d-17ab4d452d40") : secret "noten-model-95dfcc-kserve-self-signed-certs" not found Oct 31 13:10:09.514219 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:09.513601 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:10.340862 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:10.340822 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:11.035012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:11.026322 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:11.755974 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:11.755845 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:12.502707 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:12.502670 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:13.216616 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:13.216583 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:13.920420 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:13.920296 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:14.620093 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:14.620046 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:15.410473 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:15.410434 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:16.137854 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:16.137484 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:17.030016 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:17.029920 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:17.349363 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:10:17.349279 6841 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-self-signed-certs: secret "noten-model-95dfcc-kserve-self-signed-certs" not found Oct 31 13:10:17.349363 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:10:17.349364 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52319b98-c78a-4293-aa0d-17ab4d452d40-tls-certs podName:52319b98-c78a-4293-aa0d-17ab4d452d40 nodeName:}" failed. No retries permitted until 2026-10-31 13:10:33.349345308 +0000 UTC m=+2279.832468190 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/52319b98-c78a-4293-aa0d-17ab4d452d40-tls-certs") pod "noten-model-95dfcc-kserve-7d686f85d5-6ck4c" (UID: "52319b98-c78a-4293-aa0d-17ab4d452d40") : secret "noten-model-95dfcc-kserve-self-signed-certs" not found Oct 31 13:10:17.539731 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:17.539687 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c" podUID="52319b98-c78a-4293-aa0d-17ab4d452d40" containerName="main" containerID="cri-o://da8df97785863b7f831a2934016e56796e3fba84037fca60e6f536da378849c2" gracePeriod=45 Oct 31 13:10:17.662863 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:17.662830 6841 generic.go:356] "Generic (PLEG): container finished" podID="52319b98-c78a-4293-aa0d-17ab4d452d40" containerID="da8df97785863b7f831a2934016e56796e3fba84037fca60e6f536da378849c2" exitCode=0 Oct 31 13:10:17.662974 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:17.662896 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c" event={"ID":"52319b98-c78a-4293-aa0d-17ab4d452d40","Type":"ContainerDied","Data":"da8df97785863b7f831a2934016e56796e3fba84037fca60e6f536da378849c2"} Oct 31 13:10:17.883236 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:17.883203 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:17.896738 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:17.896714 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c" Oct 31 13:10:18.062241 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:18.062203 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52319b98-c78a-4293-aa0d-17ab4d452d40-kube-api-access-mbwr9\" (UniqueName: \"kubernetes.io/projected/52319b98-c78a-4293-aa0d-17ab4d452d40-kube-api-access-mbwr9\") pod \"52319b98-c78a-4293-aa0d-17ab4d452d40\" (UID: \"52319b98-c78a-4293-aa0d-17ab4d452d40\") " Oct 31 13:10:18.062436 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:18.062370 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52319b98-c78a-4293-aa0d-17ab4d452d40-tls-certs\" (UniqueName: \"kubernetes.io/secret/52319b98-c78a-4293-aa0d-17ab4d452d40-tls-certs\") pod \"52319b98-c78a-4293-aa0d-17ab4d452d40\" (UID: \"52319b98-c78a-4293-aa0d-17ab4d452d40\") " Oct 31 13:10:18.062436 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:18.062410 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52319b98-c78a-4293-aa0d-17ab4d452d40-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52319b98-c78a-4293-aa0d-17ab4d452d40-model-cache\") pod \"52319b98-c78a-4293-aa0d-17ab4d452d40\" (UID: \"52319b98-c78a-4293-aa0d-17ab4d452d40\") " Oct 31 13:10:18.062559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:18.062439 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52319b98-c78a-4293-aa0d-17ab4d452d40-home\" (UniqueName: \"kubernetes.io/empty-dir/52319b98-c78a-4293-aa0d-17ab4d452d40-home\") pod \"52319b98-c78a-4293-aa0d-17ab4d452d40\" (UID: \"52319b98-c78a-4293-aa0d-17ab4d452d40\") " Oct 31 13:10:18.062559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:18.062515 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52319b98-c78a-4293-aa0d-17ab4d452d40-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52319b98-c78a-4293-aa0d-17ab4d452d40-tmp-dir\") pod \"52319b98-c78a-4293-aa0d-17ab4d452d40\" (UID: \"52319b98-c78a-4293-aa0d-17ab4d452d40\") " Oct 31 13:10:18.062689 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:18.062588 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52319b98-c78a-4293-aa0d-17ab4d452d40-dshm\" (UniqueName: \"kubernetes.io/empty-dir/52319b98-c78a-4293-aa0d-17ab4d452d40-dshm\") pod \"52319b98-c78a-4293-aa0d-17ab4d452d40\" (UID: \"52319b98-c78a-4293-aa0d-17ab4d452d40\") " Oct 31 13:10:18.062751 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:18.062693 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52319b98-c78a-4293-aa0d-17ab4d452d40-model-cache" pod "52319b98-c78a-4293-aa0d-17ab4d452d40" (UID: "52319b98-c78a-4293-aa0d-17ab4d452d40"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:10:18.062842 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:18.062802 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52319b98-c78a-4293-aa0d-17ab4d452d40-home" pod "52319b98-c78a-4293-aa0d-17ab4d452d40" (UID: "52319b98-c78a-4293-aa0d-17ab4d452d40"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:10:18.062922 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:18.062901 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52319b98-c78a-4293-aa0d-17ab4d452d40-tmp-dir" pod "52319b98-c78a-4293-aa0d-17ab4d452d40" (UID: "52319b98-c78a-4293-aa0d-17ab4d452d40"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:10:18.063312 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:18.063292 6841 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52319b98-c78a-4293-aa0d-17ab4d452d40-model-cache\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:10:18.063312 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:18.063315 6841 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52319b98-c78a-4293-aa0d-17ab4d452d40-home\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:10:18.063451 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:18.063333 6841 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52319b98-c78a-4293-aa0d-17ab4d452d40-tmp-dir\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:10:18.064912 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:18.064886 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52319b98-c78a-4293-aa0d-17ab4d452d40-dshm" pod "52319b98-c78a-4293-aa0d-17ab4d452d40" (UID: "52319b98-c78a-4293-aa0d-17ab4d452d40"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:10:18.065031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:18.064978 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52319b98-c78a-4293-aa0d-17ab4d452d40-tls-certs" pod "52319b98-c78a-4293-aa0d-17ab4d452d40" (UID: "52319b98-c78a-4293-aa0d-17ab4d452d40"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:10:18.065092 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:18.065061 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52319b98-c78a-4293-aa0d-17ab4d452d40-kube-api-access-mbwr9" pod "52319b98-c78a-4293-aa0d-17ab4d452d40" (UID: "52319b98-c78a-4293-aa0d-17ab4d452d40"). InnerVolumeSpecName "kube-api-access-mbwr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:10:18.164665 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:18.164624 6841 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52319b98-c78a-4293-aa0d-17ab4d452d40-dshm\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:10:18.164665 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:18.164658 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbwr9\" (UniqueName: \"kubernetes.io/projected/52319b98-c78a-4293-aa0d-17ab4d452d40-kube-api-access-mbwr9\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:10:18.164665 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:18.164670 6841 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52319b98-c78a-4293-aa0d-17ab4d452d40-tls-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:10:18.715840 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:18.715721 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:18.720159 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:18.720131 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c" Oct 31 13:10:18.720436 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:18.720132 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c" event={"ID":"52319b98-c78a-4293-aa0d-17ab4d452d40","Type":"ContainerDied","Data":"6913d4760092b922162cbc9898ce2fcd5ab611bc80d0f55e7716ad897a53ef0a"} Oct 31 13:10:18.720620 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:18.720606 6841 scope.go:122] "RemoveContainer" containerID="da8df97785863b7f831a2934016e56796e3fba84037fca60e6f536da378849c2" Oct 31 13:10:18.755249 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:18.755220 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c"] Oct 31 13:10:18.758706 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:18.758681 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-95dfcc/noten-model-95dfcc-kserve-7d686f85d5-6ck4c"] Oct 31 13:10:19.515556 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:19.515518 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:19.644057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:19.644024 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52319b98-c78a-4293-aa0d-17ab4d452d40" path="/var/lib/kubelet/pods/52319b98-c78a-4293-aa0d-17ab4d452d40/volumes" Oct 31 13:10:20.035414 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:10:20.035370 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:10:20.374386 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:20.374302 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:21.110093 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:21.106649 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:21.810840 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:21.810808 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:22.532784 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:22.532349 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:23.232751 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:23.232721 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:23.953045 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:23.952038 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:24.726969 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:24.726566 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:25.399048 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:25.399010 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:26.113286 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:26.113250 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:26.959431 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:26.959390 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:27.632769 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:27.632731 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:28.433691 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:28.433656 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:29.114281 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:29.113919 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:29.838002 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:29.835404 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:30.617509 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:30.616777 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:31.383067 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:31.383029 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:32.125029 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:32.124975 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:32.939224 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:32.939190 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:33.598007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:33.597631 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:34.356901 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:34.356850 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:35.239682 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:35.239648 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:36.149916 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:36.149877 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:37.145193 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:37.145156 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:37.680826 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:37.680792 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:37.898250 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:37.898211 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:38.542004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:38.540327 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:39.312820 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:39.312788 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:40.088004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:40.083931 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:40.837677 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:40.837635 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:41.678341 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:41.678300 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:42.426384 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:42.426110 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:43.127005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:43.126375 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:43.929181 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:43.929146 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:44.673183 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:44.673134 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:44.962291 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:44.962263 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:45.087452 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:45.087417 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11397-276159"] Oct 31 13:10:45.088210 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:45.088193 6841 cpu_manager.go:414] "containerMap: removing container" podUID="913b94e9-bc62-42b9-ad68-2fab683a9593" containerName="istio-proxy" Oct 31 13:10:45.088276 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:45.088211 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="913b94e9-bc62-42b9-ad68-2fab683a9593" containerName="istio-proxy" Oct 31 13:10:45.088276 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:45.088226 6841 cpu_manager.go:414] "containerMap: removing container" podUID="52319b98-c78a-4293-aa0d-17ab4d452d40" containerName="main" Oct 31 13:10:45.088276 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:45.088232 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52319b98-c78a-4293-aa0d-17ab4d452d40" containerName="main" Oct 31 13:10:45.088276 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:45.088249 6841 cpu_manager.go:414] "containerMap: removing container" podUID="2b7734fb-74f8-4eed-bd5f-415d1c6e51ec" containerName="istio-proxy" Oct 31 13:10:45.088276 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:45.088254 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b7734fb-74f8-4eed-bd5f-415d1c6e51ec" containerName="istio-proxy" Oct 31 13:10:45.088276 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:45.088264 6841 cpu_manager.go:414] "containerMap: removing container" podUID="913b94e9-bc62-42b9-ad68-2fab683a9593" containerName="istio-proxy" Oct 31 13:10:45.088276 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:45.088269 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="913b94e9-bc62-42b9-ad68-2fab683a9593" containerName="istio-proxy" Oct 31 13:10:45.088499 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:45.088300 6841 cpu_manager.go:414] "containerMap: removing container" podUID="322f29ac-1b41-4e39-b296-bdedc1fd2690" containerName="istio-proxy" Oct 31 13:10:45.088499 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:45.088306 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="322f29ac-1b41-4e39-b296-bdedc1fd2690" containerName="istio-proxy" Oct 31 13:10:45.088499 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:45.088401 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="913b94e9-bc62-42b9-ad68-2fab683a9593" containerName="istio-proxy" Oct 31 13:10:45.088499 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:45.088414 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="322f29ac-1b41-4e39-b296-bdedc1fd2690" containerName="istio-proxy" Oct 31 13:10:45.088499 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:45.088423 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="913b94e9-bc62-42b9-ad68-2fab683a9593" containerName="istio-proxy" Oct 31 13:10:45.088499 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:45.088430 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="52319b98-c78a-4293-aa0d-17ab4d452d40" containerName="main" Oct 31 13:10:45.088499 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:45.088438 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b7734fb-74f8-4eed-bd5f-415d1c6e51ec" containerName="istio-proxy" Oct 31 13:10:45.094100 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:45.094071 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11397-276159" Oct 31 13:10:45.098331 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:45.098308 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mn5fx\"" Oct 31 13:10:45.100404 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:45.100375 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11397-276159"] Oct 31 13:10:45.177023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:45.176955 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7f5p\" (UniqueName: \"kubernetes.io/projected/1fb89e6a-098d-46a7-8b74-9328f1ed966e-kube-api-access-x7f5p\") pod \"test-curl-11397-276159\" (UID: \"1fb89e6a-098d-46a7-8b74-9328f1ed966e\") " pod="opendatahub/test-curl-11397-276159" Oct 31 13:10:45.278460 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:45.278382 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7f5p\" (UniqueName: \"kubernetes.io/projected/1fb89e6a-098d-46a7-8b74-9328f1ed966e-kube-api-access-x7f5p\") pod \"test-curl-11397-276159\" (UID: \"1fb89e6a-098d-46a7-8b74-9328f1ed966e\") " pod="opendatahub/test-curl-11397-276159" Oct 31 13:10:45.326126 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:45.326098 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7f5p\" (UniqueName: \"kubernetes.io/projected/1fb89e6a-098d-46a7-8b74-9328f1ed966e-kube-api-access-x7f5p\") pod \"test-curl-11397-276159\" (UID: \"1fb89e6a-098d-46a7-8b74-9328f1ed966e\") " pod="opendatahub/test-curl-11397-276159" Oct 31 13:10:45.411712 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:45.411680 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11397-276159" Oct 31 13:10:45.533773 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:45.533688 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:45.780877 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:45.780838 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11397-276159"] Oct 31 13:10:45.780877 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:10:45.780860 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fb89e6a_098d_46a7_8b74_9328f1ed966e.slice/crio-ff13b0421a8bef757e4db192839a3176bacd057fc09dd357287421aff445cddb WatchSource:0}: Error finding container ff13b0421a8bef757e4db192839a3176bacd057fc09dd357287421aff445cddb: Status 404 returned error can't find the container with id ff13b0421a8bef757e4db192839a3176bacd057fc09dd357287421aff445cddb Oct 31 13:10:46.009139 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:46.009101 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11397-276159" event={"ID":"1fb89e6a-098d-46a7-8b74-9328f1ed966e","Type":"ContainerStarted","Data":"7b3c61e477f51b386a9a6aae44848b44e3d9f1967ce8cb58251cde5c6b736efb"} Oct 31 13:10:46.009537 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:46.009150 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11397-276159" event={"ID":"1fb89e6a-098d-46a7-8b74-9328f1ed966e","Type":"ContainerStarted","Data":"ff13b0421a8bef757e4db192839a3176bacd057fc09dd357287421aff445cddb"} Oct 31 13:10:46.033522 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:46.033475 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11397-276159" podStartSLOduration=1.033462363 podStartE2EDuration="1.033462363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:46.030630107 +0000 UTC m=+2292.513753009" watchObservedRunningTime="2026-10-31 13:10:46.033462363 +0000 UTC m=+2292.516585443" Oct 31 13:10:46.439792 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:46.439748 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:46.723807 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:46.723771 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11397-276159"] Oct 31 13:10:46.728047 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:46.728019 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11397-276159"] Oct 31 13:10:46.888764 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:46.888733 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11397-e094bf"] Oct 31 13:10:46.889478 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:46.889461 6841 cpu_manager.go:414] "containerMap: removing container" podUID="1fb89e6a-098d-46a7-8b74-9328f1ed966e" containerName="test-curl-11397-276159" Oct 31 13:10:46.889532 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:46.889479 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fb89e6a-098d-46a7-8b74-9328f1ed966e" containerName="test-curl-11397-276159" Oct 31 13:10:46.889592 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:46.889583 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fb89e6a-098d-46a7-8b74-9328f1ed966e" containerName="test-curl-11397-276159" Oct 31 13:10:46.893346 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:46.893330 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11397-e094bf" Oct 31 13:10:46.902111 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:46.902085 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11397-e094bf"] Oct 31 13:10:47.010906 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:47.010811 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26pzg\" (UniqueName: \"kubernetes.io/projected/8b246df5-d334-4335-9de9-be02ccd65fb4-kube-api-access-26pzg\") pod \"test-curl-11397-e094bf\" (UID: \"8b246df5-d334-4335-9de9-be02ccd65fb4\") " pod="opendatahub/test-curl-11397-e094bf" Oct 31 13:10:47.112238 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:47.112197 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26pzg\" (UniqueName: \"kubernetes.io/projected/8b246df5-d334-4335-9de9-be02ccd65fb4-kube-api-access-26pzg\") pod \"test-curl-11397-e094bf\" (UID: \"8b246df5-d334-4335-9de9-be02ccd65fb4\") " pod="opendatahub/test-curl-11397-e094bf" Oct 31 13:10:47.124671 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:47.124640 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26pzg\" (UniqueName: \"kubernetes.io/projected/8b246df5-d334-4335-9de9-be02ccd65fb4-kube-api-access-26pzg\") pod \"test-curl-11397-e094bf\" (UID: \"8b246df5-d334-4335-9de9-be02ccd65fb4\") " pod="opendatahub/test-curl-11397-e094bf" Oct 31 13:10:47.210794 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:47.210755 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11397-e094bf" Oct 31 13:10:47.324636 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:47.324553 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:47.564916 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:47.564850 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11397-e094bf"] Oct 31 13:10:47.565235 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:10:47.565205 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b246df5_d334_4335_9de9_be02ccd65fb4.slice/crio-b9d06394389ea2ecea9108197c112c28f780c825d36226b2fdcf0a0a84a5acbc WatchSource:0}: Error finding container b9d06394389ea2ecea9108197c112c28f780c825d36226b2fdcf0a0a84a5acbc: Status 404 returned error can't find the container with id b9d06394389ea2ecea9108197c112c28f780c825d36226b2fdcf0a0a84a5acbc Oct 31 13:10:48.025187 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:48.025151 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11397-e094bf" event={"ID":"8b246df5-d334-4335-9de9-be02ccd65fb4","Type":"ContainerStarted","Data":"626df8fc69a9f9671ba41f9334c9f3fdebb1ac65731067854b4367e2f03728e0"} Oct 31 13:10:48.025633 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:48.025193 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11397-e094bf" event={"ID":"8b246df5-d334-4335-9de9-be02ccd65fb4","Type":"ContainerStarted","Data":"b9d06394389ea2ecea9108197c112c28f780c825d36226b2fdcf0a0a84a5acbc"} Oct 31 13:10:48.025633 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:48.025372 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11397-276159" podUID="1fb89e6a-098d-46a7-8b74-9328f1ed966e" containerName="test-curl-11397-276159" containerID="cri-o://7b3c61e477f51b386a9a6aae44848b44e3d9f1967ce8cb58251cde5c6b736efb" gracePeriod=2 Oct 31 13:10:48.045494 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:48.045443 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11397-e094bf" podStartSLOduration=2.045430072 podStartE2EDuration="2.045430072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:48.043927315 +0000 UTC m=+2294.527050207" watchObservedRunningTime="2026-10-31 13:10:48.045430072 +0000 UTC m=+2294.528552973" Oct 31 13:10:48.209817 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:48.209557 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:48.773001 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:48.772918 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11397-e094bf"] Oct 31 13:10:48.776808 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:48.776778 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11397-e094bf"] Oct 31 13:10:48.928011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:48.927365 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:49.497810 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:49.497773 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:49.709098 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:49.709065 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:49.891636 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:49.891553 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44"] Oct 31 13:10:49.892294 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:49.892277 6841 cpu_manager.go:414] "containerMap: removing container" podUID="8b246df5-d334-4335-9de9-be02ccd65fb4" containerName="test-curl-11397-e094bf" Oct 31 13:10:49.892350 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:49.892295 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b246df5-d334-4335-9de9-be02ccd65fb4" containerName="test-curl-11397-e094bf" Oct 31 13:10:49.892441 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:49.892432 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b246df5-d334-4335-9de9-be02ccd65fb4" containerName="test-curl-11397-e094bf" Oct 31 13:10:49.895790 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:49.895757 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:10:49.908602 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:49.908566 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-6c0612-openshift-default-dockercfg-thqpk\"" Oct 31 13:10:49.938024 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:49.914764 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44"] Oct 31 13:10:50.026006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.022779 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:50.167079 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.163428 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-istio-envoy\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44\" (UID: \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:10:50.167079 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.163477 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-istio-podinfo\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44\" (UID: \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:10:50.167079 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.163510 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-workload-socket\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44\" (UID: \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:10:50.167079 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.163547 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-istio-token\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44\" (UID: \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:10:50.167079 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.163592 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-credential-socket\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44\" (UID: \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:10:50.167079 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.163749 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-workload-certs\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44\" (UID: \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:10:50.167079 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.163817 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcgzd\" (UniqueName: \"kubernetes.io/projected/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-kube-api-access-hcgzd\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44\" (UID: \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:10:50.167079 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.163855 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-istio-data\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44\" (UID: \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:10:50.167079 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.163953 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-istiod-ca-cert\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44\" (UID: \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:10:50.287219 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.287172 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:50.288354 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.288324 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-istiod-ca-cert\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44\" (UID: \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:10:50.288543 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.288520 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-istio-envoy\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44\" (UID: \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:10:50.288626 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.288556 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-istio-podinfo\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44\" (UID: \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:10:50.288626 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.288585 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-workload-socket\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44\" (UID: \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:10:50.288745 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.288624 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-istio-token\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44\" (UID: \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:10:50.288745 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.288669 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-credential-socket\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44\" (UID: \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:10:50.288865 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.288810 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-workload-certs\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44\" (UID: \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:10:50.288865 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.288848 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcgzd\" (UniqueName: \"kubernetes.io/projected/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-kube-api-access-hcgzd\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44\" (UID: \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:10:50.288998 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.288896 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-istio-data\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44\" (UID: \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:10:50.289426 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.289404 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-workload-socket\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44\" (UID: \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:10:50.297917 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.297878 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-istio-data\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44\" (UID: \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:10:50.298577 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.298556 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-istiod-ca-cert\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44\" (UID: \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:10:50.299341 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.299307 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11397-e094bf" podUID="8b246df5-d334-4335-9de9-be02ccd65fb4" containerName="test-curl-11397-e094bf" containerID="cri-o://626df8fc69a9f9671ba41f9334c9f3fdebb1ac65731067854b4367e2f03728e0" gracePeriod=2 Oct 31 13:10:50.300329 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.300165 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-credential-socket\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44\" (UID: \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:10:50.300488 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.300411 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-workload-certs\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44\" (UID: \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:10:50.302193 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.302167 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-istio-envoy\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44\" (UID: \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:10:50.304553 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.304520 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-istio-podinfo\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44\" (UID: \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:10:50.318529 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.318029 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcgzd\" (UniqueName: \"kubernetes.io/projected/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-kube-api-access-hcgzd\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44\" (UID: \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:10:50.318529 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.318172 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-istio-token\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44\" (UID: \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:10:50.333293 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:10:50.333261 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:10:50.479727 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.479704 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11397-276159" Oct 31 13:10:50.501359 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.501333 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fb89e6a-098d-46a7-8b74-9328f1ed966e-kube-api-access-x7f5p\" (UniqueName: \"kubernetes.io/projected/1fb89e6a-098d-46a7-8b74-9328f1ed966e-kube-api-access-x7f5p\") pod \"1fb89e6a-098d-46a7-8b74-9328f1ed966e\" (UID: \"1fb89e6a-098d-46a7-8b74-9328f1ed966e\") " Oct 31 13:10:50.503421 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.503395 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb89e6a-098d-46a7-8b74-9328f1ed966e-kube-api-access-x7f5p" pod "1fb89e6a-098d-46a7-8b74-9328f1ed966e" (UID: "1fb89e6a-098d-46a7-8b74-9328f1ed966e"). InnerVolumeSpecName "kube-api-access-x7f5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:10:50.536129 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.517503 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:10:50.584304 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.584132 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:50.603861 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.603832 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7f5p\" (UniqueName: \"kubernetes.io/projected/1fb89e6a-098d-46a7-8b74-9328f1ed966e-kube-api-access-x7f5p\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:10:50.869529 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.869501 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44"] Oct 31 13:10:50.870665 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:10:50.870643 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbae99f9_22a1_49d4_a08d_3bc7a0f69bde.slice/crio-9a5d55f7a4a026e1819333294bc267627934a02af99e913c505255e55e99c62a WatchSource:0}: Error finding container 9a5d55f7a4a026e1819333294bc267627934a02af99e913c505255e55e99c62a: Status 404 returned error can't find the container with id 9a5d55f7a4a026e1819333294bc267627934a02af99e913c505255e55e99c62a Oct 31 13:10:50.873474 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.873441 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:10:50.873554 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.873510 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:10:50.873554 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:50.873546 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:10:51.308031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:51.307965 6841 generic.go:356] "Generic (PLEG): container finished" podID="1fb89e6a-098d-46a7-8b74-9328f1ed966e" containerID="7b3c61e477f51b386a9a6aae44848b44e3d9f1967ce8cb58251cde5c6b736efb" exitCode=137 Oct 31 13:10:51.308031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:51.308030 6841 scope.go:122] "RemoveContainer" containerID="7b3c61e477f51b386a9a6aae44848b44e3d9f1967ce8cb58251cde5c6b736efb" Oct 31 13:10:51.308275 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:51.308044 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11397-276159" Oct 31 13:10:51.310382 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:51.310356 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" event={"ID":"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde","Type":"ContainerStarted","Data":"48ec3c10d81300a560938fa16eecf19be41d253c74ae0a5c468d9456574975d4"} Oct 31 13:10:51.310532 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:51.310403 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" event={"ID":"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde","Type":"ContainerStarted","Data":"9a5d55f7a4a026e1819333294bc267627934a02af99e913c505255e55e99c62a"} Oct 31 13:10:51.324791 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:51.324771 6841 scope.go:122] "RemoveContainer" containerID="7b3c61e477f51b386a9a6aae44848b44e3d9f1967ce8cb58251cde5c6b736efb" Oct 31 13:10:51.325074 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:10:51.325057 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b3c61e477f51b386a9a6aae44848b44e3d9f1967ce8cb58251cde5c6b736efb\": container with ID starting with 7b3c61e477f51b386a9a6aae44848b44e3d9f1967ce8cb58251cde5c6b736efb not found: ID does not exist" containerID="7b3c61e477f51b386a9a6aae44848b44e3d9f1967ce8cb58251cde5c6b736efb" Oct 31 13:10:51.325115 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:51.325082 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b3c61e477f51b386a9a6aae44848b44e3d9f1967ce8cb58251cde5c6b736efb"} err="failed to get container status \"7b3c61e477f51b386a9a6aae44848b44e3d9f1967ce8cb58251cde5c6b736efb\": rpc error: code = NotFound desc = could not find container \"7b3c61e477f51b386a9a6aae44848b44e3d9f1967ce8cb58251cde5c6b736efb\": container with ID starting with 7b3c61e477f51b386a9a6aae44848b44e3d9f1967ce8cb58251cde5c6b736efb not found: ID does not exist" Oct 31 13:10:51.337178 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:10:51.337140 6841 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11397-276159\" is forbidden: User \"system:node:ip-10-0-138-70.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-70.us-east-2.compute.internal' and this object" podUID="1fb89e6a-098d-46a7-8b74-9328f1ed966e" pod="opendatahub/test-curl-11397-276159" Oct 31 13:10:51.338190 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:51.338146 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" podStartSLOduration=2.338131398 podStartE2EDuration="2.338131398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:51.334613343 +0000 UTC m=+2297.817736283" watchObservedRunningTime="2026-10-31 13:10:51.338131398 +0000 UTC m=+2297.821254298" Oct 31 13:10:51.438123 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:51.438087 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:51.521682 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:51.521647 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:10:51.555243 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:51.555212 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:10:51.582656 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:10:51.582554 6841 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11397-276159\" is forbidden: User \"system:node:ip-10-0-138-70.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-70.us-east-2.compute.internal' and this object" podUID="1fb89e6a-098d-46a7-8b74-9328f1ed966e" pod="opendatahub/test-curl-11397-276159" Oct 31 13:10:51.644762 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:51.644727 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fb89e6a-098d-46a7-8b74-9328f1ed966e" path="/var/lib/kubelet/pods/1fb89e6a-098d-46a7-8b74-9328f1ed966e/volumes" Oct 31 13:10:52.138871 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:52.138831 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:52.320523 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:52.320500 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:10:52.321807 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:52.321784 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:10:52.520378 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:52.520353 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11397-e094bf" Oct 31 13:10:52.638422 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:52.638392 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b246df5-d334-4335-9de9-be02ccd65fb4-kube-api-access-26pzg\" (UniqueName: \"kubernetes.io/projected/8b246df5-d334-4335-9de9-be02ccd65fb4-kube-api-access-26pzg\") pod \"8b246df5-d334-4335-9de9-be02ccd65fb4\" (UID: \"8b246df5-d334-4335-9de9-be02ccd65fb4\") " Oct 31 13:10:52.640548 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:52.640521 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b246df5-d334-4335-9de9-be02ccd65fb4-kube-api-access-26pzg" pod "8b246df5-d334-4335-9de9-be02ccd65fb4" (UID: "8b246df5-d334-4335-9de9-be02ccd65fb4"). InnerVolumeSpecName "kube-api-access-26pzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:10:52.740875 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:52.740843 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26pzg\" (UniqueName: \"kubernetes.io/projected/8b246df5-d334-4335-9de9-be02ccd65fb4-kube-api-access-26pzg\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:10:53.035759 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:53.031922 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:53.407105 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:53.395383 6841 generic.go:356] "Generic (PLEG): container finished" podID="8b246df5-d334-4335-9de9-be02ccd65fb4" containerID="626df8fc69a9f9671ba41f9334c9f3fdebb1ac65731067854b4367e2f03728e0" exitCode=137 Oct 31 13:10:53.407105 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:53.400087 6841 scope.go:122] "RemoveContainer" containerID="626df8fc69a9f9671ba41f9334c9f3fdebb1ac65731067854b4367e2f03728e0" Oct 31 13:10:53.407105 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:53.400276 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11397-e094bf" Oct 31 13:10:53.462400 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:10:53.462335 6841 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11397-e094bf\" is forbidden: User \"system:node:ip-10-0-138-70.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-70.us-east-2.compute.internal' and this object" podUID="8b246df5-d334-4335-9de9-be02ccd65fb4" pod="opendatahub/test-curl-11397-e094bf" Oct 31 13:10:53.518006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:53.506179 6841 scope.go:122] "RemoveContainer" containerID="626df8fc69a9f9671ba41f9334c9f3fdebb1ac65731067854b4367e2f03728e0" Oct 31 13:10:53.518006 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:10:53.507405 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"626df8fc69a9f9671ba41f9334c9f3fdebb1ac65731067854b4367e2f03728e0\": container with ID starting with 626df8fc69a9f9671ba41f9334c9f3fdebb1ac65731067854b4367e2f03728e0 not found: ID does not exist" containerID="626df8fc69a9f9671ba41f9334c9f3fdebb1ac65731067854b4367e2f03728e0" Oct 31 13:10:53.518006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:53.507442 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"626df8fc69a9f9671ba41f9334c9f3fdebb1ac65731067854b4367e2f03728e0"} err="failed to get container status \"626df8fc69a9f9671ba41f9334c9f3fdebb1ac65731067854b4367e2f03728e0\": rpc error: code = NotFound desc = could not find container \"626df8fc69a9f9671ba41f9334c9f3fdebb1ac65731067854b4367e2f03728e0\": container with ID starting with 626df8fc69a9f9671ba41f9334c9f3fdebb1ac65731067854b4367e2f03728e0 not found: ID does not exist" Oct 31 13:10:53.643229 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:10:53.643180 6841 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11397-e094bf\" is forbidden: User \"system:node:ip-10-0-138-70.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-70.us-east-2.compute.internal' and this object" podUID="8b246df5-d334-4335-9de9-be02ccd65fb4" pod="opendatahub/test-curl-11397-e094bf" Oct 31 13:10:53.646024 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:53.645996 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b246df5-d334-4335-9de9-be02ccd65fb4" path="/var/lib/kubelet/pods/8b246df5-d334-4335-9de9-be02ccd65fb4/volumes" Oct 31 13:10:53.699176 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:53.699141 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:54.036060 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:54.017947 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:54.231097 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:54.231046 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:54.971198 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:54.971159 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:55.229020 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:55.223314 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:55.442586 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:55.442548 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:56.114618 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:56.114581 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:56.845181 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:56.845138 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:57.546176 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:57.546143 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:58.355605 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:58.355554 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:10:59.249110 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:10:59.249056 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:00.033197 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:00.033167 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:00.901004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:00.897645 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:01.612935 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:01.612650 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:02.339741 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:02.339651 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:03.184644 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:03.184602 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:03.916553 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:03.916465 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:04.674088 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:04.674055 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt"] Oct 31 13:11:04.679404 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:04.679374 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt" Oct 31 13:11:04.689251 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:04.689210 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-6c0612-serving-cert\"" Oct 31 13:11:04.693461 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:04.693434 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt"] Oct 31 13:11:04.766285 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:04.766252 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/909de922-16af-4ee1-b6cf-375ab5fb6053-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt\" (UID: \"909de922-16af-4ee1-b6cf-375ab5fb6053\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt" Oct 31 13:11:04.766442 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:04.766307 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drx7d\" (UniqueName: \"kubernetes.io/projected/909de922-16af-4ee1-b6cf-375ab5fb6053-kube-api-access-drx7d\") pod \"maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt\" (UID: \"909de922-16af-4ee1-b6cf-375ab5fb6053\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt" Oct 31 13:11:04.766682 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:04.766657 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/909de922-16af-4ee1-b6cf-375ab5fb6053-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt\" (UID: \"909de922-16af-4ee1-b6cf-375ab5fb6053\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt" Oct 31 13:11:04.766802 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:04.766793 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/909de922-16af-4ee1-b6cf-375ab5fb6053-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt\" (UID: \"909de922-16af-4ee1-b6cf-375ab5fb6053\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt" Oct 31 13:11:04.835042 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:04.834929 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:04.869062 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:04.869020 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/909de922-16af-4ee1-b6cf-375ab5fb6053-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt\" (UID: \"909de922-16af-4ee1-b6cf-375ab5fb6053\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt" Oct 31 13:11:04.869385 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:11:04.869356 6841 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0612-serving-cert: secret "maas-api-e2e-shared-a-w4-6c0612-serving-cert" not found Oct 31 13:11:04.869479 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:11:04.869442 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/909de922-16af-4ee1-b6cf-375ab5fb6053-maas-api-tls podName:909de922-16af-4ee1-b6cf-375ab5fb6053 nodeName:}" failed. No retries permitted until 2026-10-31 13:11:05.369420491 +0000 UTC m=+2311.852543387 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/909de922-16af-4ee1-b6cf-375ab5fb6053-maas-api-tls") pod "maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt" (UID: "909de922-16af-4ee1-b6cf-375ab5fb6053") : secret "maas-api-e2e-shared-a-w4-6c0612-serving-cert" not found Oct 31 13:11:04.869669 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:04.869593 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/909de922-16af-4ee1-b6cf-375ab5fb6053-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt\" (UID: \"909de922-16af-4ee1-b6cf-375ab5fb6053\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt" Oct 31 13:11:04.875366 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:04.870006 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/909de922-16af-4ee1-b6cf-375ab5fb6053-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt\" (UID: \"909de922-16af-4ee1-b6cf-375ab5fb6053\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt" Oct 31 13:11:04.875584 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:04.875558 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drx7d\" (UniqueName: \"kubernetes.io/projected/909de922-16af-4ee1-b6cf-375ab5fb6053-kube-api-access-drx7d\") pod \"maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt\" (UID: \"909de922-16af-4ee1-b6cf-375ab5fb6053\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt" Oct 31 13:11:04.876249 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:04.870653 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/909de922-16af-4ee1-b6cf-375ab5fb6053-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt\" (UID: \"909de922-16af-4ee1-b6cf-375ab5fb6053\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt" Oct 31 13:11:04.882370 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:04.882344 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/909de922-16af-4ee1-b6cf-375ab5fb6053-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt\" (UID: \"909de922-16af-4ee1-b6cf-375ab5fb6053\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt" Oct 31 13:11:04.891547 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:04.891523 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drx7d\" (UniqueName: \"kubernetes.io/projected/909de922-16af-4ee1-b6cf-375ab5fb6053-kube-api-access-drx7d\") pod \"maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt\" (UID: \"909de922-16af-4ee1-b6cf-375ab5fb6053\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt" Oct 31 13:11:05.043108 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:05.043067 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:05.249872 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:05.249836 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:05.386337 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:05.386237 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/909de922-16af-4ee1-b6cf-375ab5fb6053-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt\" (UID: \"909de922-16af-4ee1-b6cf-375ab5fb6053\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt" Oct 31 13:11:05.415931 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:05.415895 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/909de922-16af-4ee1-b6cf-375ab5fb6053-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt\" (UID: \"909de922-16af-4ee1-b6cf-375ab5fb6053\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt" Oct 31 13:11:05.604968 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:05.604914 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt" Oct 31 13:11:05.848508 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:05.848445 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:05.991852 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:11:05.991814 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod909de922_16af_4ee1_b6cf_375ab5fb6053.slice/crio-ebfb8aade2604a876881254f7df85154cf8394348aee56919e64a9c1f6f7ab1d WatchSource:0}: Error finding container ebfb8aade2604a876881254f7df85154cf8394348aee56919e64a9c1f6f7ab1d: Status 404 returned error can't find the container with id ebfb8aade2604a876881254f7df85154cf8394348aee56919e64a9c1f6f7ab1d Oct 31 13:11:05.992117 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:05.992090 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt"] Oct 31 13:11:06.534612 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:06.534565 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt" event={"ID":"909de922-16af-4ee1-b6cf-375ab5fb6053","Type":"ContainerStarted","Data":"ebfb8aade2604a876881254f7df85154cf8394348aee56919e64a9c1f6f7ab1d"} Oct 31 13:11:06.541344 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:06.541315 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:07.282030 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:07.276377 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:07.544406 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:07.544319 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt" event={"ID":"909de922-16af-4ee1-b6cf-375ab5fb6053","Type":"ContainerStarted","Data":"0135250c8edf94c11553b4e7bc62b863749593952676bac1268674ee970f0730"} Oct 31 13:11:07.544406 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:07.544397 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt" Oct 31 13:11:07.575194 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:07.575141 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt" podStartSLOduration=2.891953708 podStartE2EDuration="3.575126757s" totalImagesPullingTime="683.173049ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:11:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:11:07.571603416 +0000 UTC m=+2314.054726317" watchObservedRunningTime="2026-10-31 13:11:07.575126757 +0000 UTC m=+2314.058249657" Oct 31 13:11:08.113095 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:08.113003 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:08.830136 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:08.830102 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:09.504840 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:09.504803 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:10.237262 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:10.237221 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:10.952520 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:10.952478 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:11.730851 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:11.730809 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:12.511427 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:12.511374 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:13.218716 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:13.218616 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:13.557204 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:13.557128 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt" Oct 31 13:11:13.947060 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:13.946480 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:14.614120 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:14.614034 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:15.437893 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:15.437861 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:16.155457 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:16.155424 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:17.060470 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:17.060427 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:17.888205 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:17.888170 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:18.725347 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:18.725317 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:18.989618 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:18.989517 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:19.462032 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:19.461954 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:19.674312 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:19.674277 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:19.786249 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:19.786178 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp"] Oct 31 13:11:19.791066 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:19.791043 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:11:19.802075 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:19.802048 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-1e72f6-openshift-default-dockercfg-b6b7d\"" Oct 31 13:11:19.808387 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:19.808362 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp"] Oct 31 13:11:19.947087 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:19.935329 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:19.994659 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:19.963327 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01218eaf-82da-49c5-9195-1f8af3775573-credential-socket\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp\" (UID: \"01218eaf-82da-49c5-9195-1f8af3775573\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:11:19.994659 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:19.963384 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01218eaf-82da-49c5-9195-1f8af3775573-workload-certs\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp\" (UID: \"01218eaf-82da-49c5-9195-1f8af3775573\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:11:19.994659 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:19.963484 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01218eaf-82da-49c5-9195-1f8af3775573-istio-envoy\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp\" (UID: \"01218eaf-82da-49c5-9195-1f8af3775573\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:11:19.994659 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:19.963510 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01218eaf-82da-49c5-9195-1f8af3775573-istio-podinfo\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp\" (UID: \"01218eaf-82da-49c5-9195-1f8af3775573\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:11:19.994659 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:19.963665 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01218eaf-82da-49c5-9195-1f8af3775573-istiod-ca-cert\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp\" (UID: \"01218eaf-82da-49c5-9195-1f8af3775573\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:11:19.994659 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:19.963739 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6wxw\" (UniqueName: \"kubernetes.io/projected/01218eaf-82da-49c5-9195-1f8af3775573-kube-api-access-q6wxw\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp\" (UID: \"01218eaf-82da-49c5-9195-1f8af3775573\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:11:19.994659 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:19.963771 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01218eaf-82da-49c5-9195-1f8af3775573-istio-data\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp\" (UID: \"01218eaf-82da-49c5-9195-1f8af3775573\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:11:19.994659 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:19.963835 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01218eaf-82da-49c5-9195-1f8af3775573-workload-socket\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp\" (UID: \"01218eaf-82da-49c5-9195-1f8af3775573\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:11:19.994659 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:19.963915 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01218eaf-82da-49c5-9195-1f8af3775573-istio-token\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp\" (UID: \"01218eaf-82da-49c5-9195-1f8af3775573\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:11:20.070532 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:20.070444 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01218eaf-82da-49c5-9195-1f8af3775573-istio-envoy\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp\" (UID: \"01218eaf-82da-49c5-9195-1f8af3775573\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:11:20.077330 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:20.077300 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01218eaf-82da-49c5-9195-1f8af3775573-istio-podinfo\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp\" (UID: \"01218eaf-82da-49c5-9195-1f8af3775573\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:11:20.096581 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:20.096550 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01218eaf-82da-49c5-9195-1f8af3775573-istiod-ca-cert\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp\" (UID: \"01218eaf-82da-49c5-9195-1f8af3775573\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:11:20.097648 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:20.097621 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6wxw\" (UniqueName: \"kubernetes.io/projected/01218eaf-82da-49c5-9195-1f8af3775573-kube-api-access-q6wxw\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp\" (UID: \"01218eaf-82da-49c5-9195-1f8af3775573\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:11:20.098175 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:20.098151 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01218eaf-82da-49c5-9195-1f8af3775573-istio-data\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp\" (UID: \"01218eaf-82da-49c5-9195-1f8af3775573\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:11:20.098688 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:20.098671 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01218eaf-82da-49c5-9195-1f8af3775573-workload-socket\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp\" (UID: \"01218eaf-82da-49c5-9195-1f8af3775573\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:11:20.099176 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:20.099157 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01218eaf-82da-49c5-9195-1f8af3775573-istio-token\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp\" (UID: \"01218eaf-82da-49c5-9195-1f8af3775573\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:11:20.099665 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:20.099646 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01218eaf-82da-49c5-9195-1f8af3775573-credential-socket\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp\" (UID: \"01218eaf-82da-49c5-9195-1f8af3775573\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:11:20.100118 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:20.100100 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01218eaf-82da-49c5-9195-1f8af3775573-workload-certs\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp\" (UID: \"01218eaf-82da-49c5-9195-1f8af3775573\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:11:20.100647 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:20.099039 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01218eaf-82da-49c5-9195-1f8af3775573-workload-socket\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp\" (UID: \"01218eaf-82da-49c5-9195-1f8af3775573\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:11:20.100779 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:20.097494 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01218eaf-82da-49c5-9195-1f8af3775573-istiod-ca-cert\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp\" (UID: \"01218eaf-82da-49c5-9195-1f8af3775573\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:11:20.100879 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:20.100035 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01218eaf-82da-49c5-9195-1f8af3775573-credential-socket\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp\" (UID: \"01218eaf-82da-49c5-9195-1f8af3775573\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:11:20.101001 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:20.098558 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01218eaf-82da-49c5-9195-1f8af3775573-istio-data\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp\" (UID: \"01218eaf-82da-49c5-9195-1f8af3775573\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:11:20.101102 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:20.090922 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01218eaf-82da-49c5-9195-1f8af3775573-istio-envoy\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp\" (UID: \"01218eaf-82da-49c5-9195-1f8af3775573\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:11:20.101191 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:20.100451 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01218eaf-82da-49c5-9195-1f8af3775573-workload-certs\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp\" (UID: \"01218eaf-82da-49c5-9195-1f8af3775573\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:11:20.117751 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:20.117706 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01218eaf-82da-49c5-9195-1f8af3775573-istio-podinfo\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp\" (UID: \"01218eaf-82da-49c5-9195-1f8af3775573\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:11:20.141011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:20.119458 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01218eaf-82da-49c5-9195-1f8af3775573-istio-token\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp\" (UID: \"01218eaf-82da-49c5-9195-1f8af3775573\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:11:20.151016 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:20.147037 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6wxw\" (UniqueName: \"kubernetes.io/projected/01218eaf-82da-49c5-9195-1f8af3775573-kube-api-access-q6wxw\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp\" (UID: \"01218eaf-82da-49c5-9195-1f8af3775573\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:11:20.151016 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:11:20.149253 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:11:20.155785 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:20.155748 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:20.411631 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:20.411549 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:11:20.525768 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:20.525665 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:20.781235 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:20.781205 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp"] Oct 31 13:11:20.781578 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:11:20.781556 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01218eaf_82da_49c5_9195_1f8af3775573.slice/crio-32e87c434306caefa99837e8ec6fd7e92c0f610a1bc2552b97de56bee7c5eca0 WatchSource:0}: Error finding container 32e87c434306caefa99837e8ec6fd7e92c0f610a1bc2552b97de56bee7c5eca0: Status 404 returned error can't find the container with id 32e87c434306caefa99837e8ec6fd7e92c0f610a1bc2552b97de56bee7c5eca0 Oct 31 13:11:20.784332 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:20.784296 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:11:20.784437 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:20.784365 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:11:20.784437 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:20.784395 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:11:21.366798 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:21.366766 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:21.723610 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:21.722600 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" event={"ID":"01218eaf-82da-49c5-9195-1f8af3775573","Type":"ContainerStarted","Data":"1f3a721c728f679db20ccf78ded720a693a5e9a06bbb4ccfb1e0865506375925"} Oct 31 13:11:21.724034 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:21.723946 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" event={"ID":"01218eaf-82da-49c5-9195-1f8af3775573","Type":"ContainerStarted","Data":"32e87c434306caefa99837e8ec6fd7e92c0f610a1bc2552b97de56bee7c5eca0"} Oct 31 13:11:21.762409 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:21.762346 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" podStartSLOduration=2.762325732 podStartE2EDuration="2.762325732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:11:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:11:21.76221421 +0000 UTC m=+2328.245337110" watchObservedRunningTime="2026-10-31 13:11:21.762325732 +0000 UTC m=+2328.245448636" Oct 31 13:11:22.414105 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:22.414065 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:11:22.578933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:22.578883 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:22.619342 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:22.619311 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:11:22.731513 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:22.731483 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:11:22.732390 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:22.732370 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:11:24.132583 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:24.132528 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:24.368219 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:24.368181 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:24.611724 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:24.611679 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:25.348004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:25.346903 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:25.593040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:25.592973 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:25.827669 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:25.827637 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:26.406682 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:26.406644 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:27.153694 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:27.153648 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:27.908638 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:27.908602 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:28.600944 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:28.600907 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:29.383956 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:29.383922 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:30.134281 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:30.134239 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:30.812076 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:30.811399 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:31.606375 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:31.606325 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:32.350825 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:32.350655 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:33.117621 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:33.117589 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:33.808273 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:33.808034 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:33.962009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:33.961949 6841 generic.go:356] "Generic (PLEG): container finished" podID="fbae99f9-22a1-49d4-a08d-3bc7a0f69bde" containerID="48ec3c10d81300a560938fa16eecf19be41d253c74ae0a5c468d9456574975d4" exitCode=0 Oct 31 13:11:33.962172 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:33.962020 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" event={"ID":"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde","Type":"ContainerDied","Data":"48ec3c10d81300a560938fa16eecf19be41d253c74ae0a5c468d9456574975d4"} Oct 31 13:11:33.962494 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:33.962479 6841 scope.go:122] "RemoveContainer" containerID="48ec3c10d81300a560938fa16eecf19be41d253c74ae0a5c468d9456574975d4" Oct 31 13:11:33.964236 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:33.964207 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:11:33.964321 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:33.964275 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:11:33.964321 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:33.964308 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:11:34.606409 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:34.603669 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:34.977249 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:34.977217 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" event={"ID":"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde","Type":"ContainerStarted","Data":"829b0011aedad82c53d858697c31defb37c628f815a56da8935a7c227206d6f4"} Oct 31 13:11:35.340695 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:35.340604 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:35.520844 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:35.520810 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:11:35.522149 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:35.522121 6841 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 31 13:11:35.522303 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:35.522191 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" podUID="fbae99f9-22a1-49d4-a08d-3bc7a0f69bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 31 13:11:36.024277 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:36.021891 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc"] Oct 31 13:11:36.028021 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:36.027649 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc"] Oct 31 13:11:36.028021 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:36.027731 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc" Oct 31 13:11:36.035017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:36.032125 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-1e72f6-serving-cert\"" Oct 31 13:11:36.109950 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:36.109912 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:36.138356 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:36.138313 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/444d54fd-04f0-4bca-9cb0-ec5cce618593-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc\" (UID: \"444d54fd-04f0-4bca-9cb0-ec5cce618593\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc" Oct 31 13:11:36.150006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:36.146932 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/444d54fd-04f0-4bca-9cb0-ec5cce618593-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc\" (UID: \"444d54fd-04f0-4bca-9cb0-ec5cce618593\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc" Oct 31 13:11:36.150006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:36.147016 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5whbb\" (UniqueName: \"kubernetes.io/projected/444d54fd-04f0-4bca-9cb0-ec5cce618593-kube-api-access-5whbb\") pod \"maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc\" (UID: \"444d54fd-04f0-4bca-9cb0-ec5cce618593\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc" Oct 31 13:11:36.150006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:36.147139 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/444d54fd-04f0-4bca-9cb0-ec5cce618593-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc\" (UID: \"444d54fd-04f0-4bca-9cb0-ec5cce618593\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc" Oct 31 13:11:36.253009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:36.250923 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/444d54fd-04f0-4bca-9cb0-ec5cce618593-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc\" (UID: \"444d54fd-04f0-4bca-9cb0-ec5cce618593\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc" Oct 31 13:11:36.253009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:36.251074 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/444d54fd-04f0-4bca-9cb0-ec5cce618593-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc\" (UID: \"444d54fd-04f0-4bca-9cb0-ec5cce618593\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc" Oct 31 13:11:36.253009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:36.251108 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5whbb\" (UniqueName: \"kubernetes.io/projected/444d54fd-04f0-4bca-9cb0-ec5cce618593-kube-api-access-5whbb\") pod \"maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc\" (UID: \"444d54fd-04f0-4bca-9cb0-ec5cce618593\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc" Oct 31 13:11:36.253009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:36.251202 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/444d54fd-04f0-4bca-9cb0-ec5cce618593-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc\" (UID: \"444d54fd-04f0-4bca-9cb0-ec5cce618593\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc" Oct 31 13:11:36.254445 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:36.254417 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/444d54fd-04f0-4bca-9cb0-ec5cce618593-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc\" (UID: \"444d54fd-04f0-4bca-9cb0-ec5cce618593\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc" Oct 31 13:11:36.271047 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:36.270973 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5whbb\" (UniqueName: \"kubernetes.io/projected/444d54fd-04f0-4bca-9cb0-ec5cce618593-kube-api-access-5whbb\") pod \"maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc\" (UID: \"444d54fd-04f0-4bca-9cb0-ec5cce618593\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc" Oct 31 13:11:36.271740 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:36.271669 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/444d54fd-04f0-4bca-9cb0-ec5cce618593-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc\" (UID: \"444d54fd-04f0-4bca-9cb0-ec5cce618593\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc" Oct 31 13:11:36.273118 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:36.273096 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/444d54fd-04f0-4bca-9cb0-ec5cce618593-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc\" (UID: \"444d54fd-04f0-4bca-9cb0-ec5cce618593\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc" Oct 31 13:11:36.325379 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:36.325314 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:36.364004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:36.363258 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc" Oct 31 13:11:36.588068 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:36.582724 6841 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 31 13:11:36.588068 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:36.582787 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" podUID="fbae99f9-22a1-49d4-a08d-3bc7a0f69bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 31 13:11:36.782317 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:36.782278 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc"] Oct 31 13:11:36.802434 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:11:36.802404 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod444d54fd_04f0_4bca_9cb0_ec5cce618593.slice/crio-5113e9ff2b3dc481ca6719eeb5da9c56b9be2407aeb50ca49e3b7c3b4f318525 WatchSource:0}: Error finding container 5113e9ff2b3dc481ca6719eeb5da9c56b9be2407aeb50ca49e3b7c3b4f318525: Status 404 returned error can't find the container with id 5113e9ff2b3dc481ca6719eeb5da9c56b9be2407aeb50ca49e3b7c3b4f318525 Oct 31 13:11:37.038914 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:37.038873 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:37.083923 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:37.083885 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc" event={"ID":"444d54fd-04f0-4bca-9cb0-ec5cce618593","Type":"ContainerStarted","Data":"5113e9ff2b3dc481ca6719eeb5da9c56b9be2407aeb50ca49e3b7c3b4f318525"} Oct 31 13:11:37.321090 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:37.321061 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:11:37.520898 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:37.520859 6841 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 31 13:11:37.521110 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:37.520918 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" podUID="fbae99f9-22a1-49d4-a08d-3bc7a0f69bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 31 13:11:37.733416 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:37.733385 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:37.930555 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:37.930515 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:38.093308 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:38.093200 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc" event={"ID":"444d54fd-04f0-4bca-9cb0-ec5cce618593","Type":"ContainerStarted","Data":"4fc78dcd62305ffc52201e24ffba76dcb821175fcc28fa65ef79dac33573aabe"} Oct 31 13:11:38.093308 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:38.093273 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc" Oct 31 13:11:38.117396 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:38.117342 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc" podStartSLOduration=2.693479405 podStartE2EDuration="3.117327375s" totalImagesPullingTime="423.84797ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:11:38.113165338 +0000 UTC m=+2344.596288240" watchObservedRunningTime="2026-10-31 13:11:38.117327375 +0000 UTC m=+2344.600450276" Oct 31 13:11:38.521329 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:38.521293 6841 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 31 13:11:38.521512 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:38.521350 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" podUID="fbae99f9-22a1-49d4-a08d-3bc7a0f69bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 31 13:11:38.658600 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:38.658565 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:39.435946 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:39.435897 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:39.521458 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:39.521427 6841 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 31 13:11:39.521636 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:39.521481 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" podUID="fbae99f9-22a1-49d4-a08d-3bc7a0f69bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 31 13:11:40.127893 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:40.127852 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:40.525437 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:40.525406 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:11:40.526583 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:40.526563 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:11:40.822517 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:40.822433 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:41.500781 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:41.500743 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:42.339332 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:42.339279 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:43.135624 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:43.128654 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:43.926137 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:43.926102 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:44.106275 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:44.106241 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc" Oct 31 13:11:44.982308 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:44.982252 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:45.703891 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:45.703855 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:46.426783 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:46.426746 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:47.182443 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:47.182401 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:48.001891 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:48.001852 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:48.822027 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:48.821977 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:50.034679 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:11:50.034637 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:11:50.080396 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:50.080365 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:50.374213 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:50.374181 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:51.159344 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:51.159307 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:52.036107 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:52.036071 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:52.835463 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:52.835425 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:53.858032 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:53.857963 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:54.534924 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:54.534031 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:54.746271 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:54.746231 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:55.374075 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:55.373241 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:56.456438 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:56.456402 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:57.433263 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:57.433205 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:58.230239 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:58.230206 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:11:59.167679 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:11:59.167630 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:00.030608 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:00.030524 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:00.834097 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:00.833180 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:01.536822 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:01.536787 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:02.360753 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:02.360669 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:03.185206 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:03.185130 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:03.931252 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:03.931159 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:04.716397 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:04.716172 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:05.445005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:05.440032 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:06.396505 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:06.396473 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:06.445285 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:06.445255 6841 generic.go:356] "Generic (PLEG): container finished" podID="01218eaf-82da-49c5-9195-1f8af3775573" containerID="1f3a721c728f679db20ccf78ded720a693a5e9a06bbb4ccfb1e0865506375925" exitCode=0 Oct 31 13:12:06.445458 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:06.445334 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" event={"ID":"01218eaf-82da-49c5-9195-1f8af3775573","Type":"ContainerDied","Data":"1f3a721c728f679db20ccf78ded720a693a5e9a06bbb4ccfb1e0865506375925"} Oct 31 13:12:06.445743 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:06.445728 6841 scope.go:122] "RemoveContainer" containerID="1f3a721c728f679db20ccf78ded720a693a5e9a06bbb4ccfb1e0865506375925" Oct 31 13:12:06.447644 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:06.447613 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:12:06.447739 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:06.447703 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:12:06.447782 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:06.447747 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:12:07.341363 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:07.341309 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:07.476303 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:07.476271 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" event={"ID":"01218eaf-82da-49c5-9195-1f8af3775573","Type":"ContainerStarted","Data":"ced144ffcd84f45b4052bc3e30650879dcc8dec4cfe6822d8b93a8175d529a48"} Oct 31 13:12:07.732199 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:07.732160 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:12:08.128469 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:08.128370 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:08.382060 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:08.373040 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:08.412751 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:08.412714 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:12:08.414226 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:08.414200 6841 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 31 13:12:08.414353 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:08.414259 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" podUID="01218eaf-82da-49c5-9195-1f8af3775573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 31 13:12:08.658356 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:08.658264 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:08.936999 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:08.936864 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:09.413332 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:09.413292 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:09.413891 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:09.413864 6841 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 31 13:12:09.414157 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:09.414126 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" podUID="01218eaf-82da-49c5-9195-1f8af3775573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 31 13:12:10.418503 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:10.414371 6841 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 31 13:12:10.418503 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:10.414449 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" podUID="01218eaf-82da-49c5-9195-1f8af3775573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 31 13:12:10.459713 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:10.459659 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:11.195889 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:11.195848 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:11.411851 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:11.411816 6841 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 31 13:12:11.412077 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:11.411884 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" podUID="01218eaf-82da-49c5-9195-1f8af3775573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 31 13:12:11.414342 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:11.414312 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:11.952916 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:11.952473 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:12.296321 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:12.296244 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:12.412554 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:12.412520 6841 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 31 13:12:12.412763 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:12.412569 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" podUID="01218eaf-82da-49c5-9195-1f8af3775573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 31 13:12:12.527673 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:12.527624 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:13.213822 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:13.213769 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:13.474180 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:13.471897 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:13.696891 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:13.696838 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:12:13.698464 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:13.698441 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:12:14.235783 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:14.235751 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:15.119125 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:15.119078 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:15.828644 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:15.828535 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:16.686363 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:16.686318 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:17.552026 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:17.551976 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:18.579095 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:18.579060 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:19.209343 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:19.209309 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:19.462403 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:19.462308 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:20.035780 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:12:20.035746 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:12:20.227019 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:20.226900 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:20.940826 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:20.940789 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:21.807011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:21.802835 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:22.061911 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:22.061823 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:22.745577 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:22.741616 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:23.483758 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:23.483618 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:24.217125 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:24.217020 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:24.843521 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:24.843485 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f"] Oct 31 13:12:24.850772 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:24.850739 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f" Oct 31 13:12:24.862566 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:24.862257 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-842ac9-serving-cert\"" Oct 31 13:12:24.866788 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:24.866760 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f"] Oct 31 13:12:24.939834 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:24.939795 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19ae4518-f78f-431b-a5a0-031805427fc8-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f\" (UID: \"19ae4518-f78f-431b-a5a0-031805427fc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f" Oct 31 13:12:24.940031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:24.939927 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19ae4518-f78f-431b-a5a0-031805427fc8-maas-api-tls\") pod \"maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f\" (UID: \"19ae4518-f78f-431b-a5a0-031805427fc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f" Oct 31 13:12:24.940250 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:24.940231 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19ae4518-f78f-431b-a5a0-031805427fc8-metrics-certs\") pod \"maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f\" (UID: \"19ae4518-f78f-431b-a5a0-031805427fc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f" Oct 31 13:12:24.940321 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:24.940266 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s99x5\" (UniqueName: \"kubernetes.io/projected/19ae4518-f78f-431b-a5a0-031805427fc8-kube-api-access-s99x5\") pod \"maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f\" (UID: \"19ae4518-f78f-431b-a5a0-031805427fc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f" Oct 31 13:12:25.026788 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:25.026751 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:25.042252 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:25.042209 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19ae4518-f78f-431b-a5a0-031805427fc8-metrics-certs\") pod \"maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f\" (UID: \"19ae4518-f78f-431b-a5a0-031805427fc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f" Oct 31 13:12:25.042524 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:25.042504 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s99x5\" (UniqueName: \"kubernetes.io/projected/19ae4518-f78f-431b-a5a0-031805427fc8-kube-api-access-s99x5\") pod \"maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f\" (UID: \"19ae4518-f78f-431b-a5a0-031805427fc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f" Oct 31 13:12:25.042689 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:25.042674 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19ae4518-f78f-431b-a5a0-031805427fc8-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f\" (UID: \"19ae4518-f78f-431b-a5a0-031805427fc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f" Oct 31 13:12:25.042852 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:25.042838 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19ae4518-f78f-431b-a5a0-031805427fc8-maas-api-tls\") pod \"maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f\" (UID: \"19ae4518-f78f-431b-a5a0-031805427fc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f" Oct 31 13:12:25.078006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:25.067545 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19ae4518-f78f-431b-a5a0-031805427fc8-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f\" (UID: \"19ae4518-f78f-431b-a5a0-031805427fc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f" Oct 31 13:12:25.078006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:25.075840 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19ae4518-f78f-431b-a5a0-031805427fc8-metrics-certs\") pod \"maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f\" (UID: \"19ae4518-f78f-431b-a5a0-031805427fc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f" Oct 31 13:12:25.110185 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:25.107059 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19ae4518-f78f-431b-a5a0-031805427fc8-maas-api-tls\") pod \"maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f\" (UID: \"19ae4518-f78f-431b-a5a0-031805427fc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f" Oct 31 13:12:25.110185 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:25.107788 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s99x5\" (UniqueName: \"kubernetes.io/projected/19ae4518-f78f-431b-a5a0-031805427fc8-kube-api-access-s99x5\") pod \"maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f\" (UID: \"19ae4518-f78f-431b-a5a0-031805427fc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f" Oct 31 13:12:25.174082 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:25.174048 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f" Oct 31 13:12:25.239043 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:25.238973 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:25.581164 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:25.579956 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f"] Oct 31 13:12:25.590333 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:12:25.590188 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19ae4518_f78f_431b_a5a0_031805427fc8.slice/crio-46885a66c156eb54f857570411315156c34d04a1e3564d190ffdb50e18e651b5 WatchSource:0}: Error finding container 46885a66c156eb54f857570411315156c34d04a1e3564d190ffdb50e18e651b5: Status 404 returned error can't find the container with id 46885a66c156eb54f857570411315156c34d04a1e3564d190ffdb50e18e651b5 Oct 31 13:12:25.783312 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:25.783281 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f" event={"ID":"19ae4518-f78f-431b-a5a0-031805427fc8","Type":"ContainerStarted","Data":"46885a66c156eb54f857570411315156c34d04a1e3564d190ffdb50e18e651b5"} Oct 31 13:12:26.008043 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:26.007932 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:26.840309 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:26.839032 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f" event={"ID":"19ae4518-f78f-431b-a5a0-031805427fc8","Type":"ContainerStarted","Data":"faabd33c71743e2a63a646af54f7a5b616a716215c55e230c2522ba3151fa562"} Oct 31 13:12:26.840732 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:26.840700 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f" Oct 31 13:12:26.848834 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:26.848683 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:26.875167 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:26.875120 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f" podStartSLOduration=2.415184797 podStartE2EDuration="2.875106714s" totalImagesPullingTime="459.921917ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:26.871848389 +0000 UTC m=+2393.354971293" watchObservedRunningTime="2026-10-31 13:12:26.875106714 +0000 UTC m=+2393.358229615" Oct 31 13:12:27.660963 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:27.660924 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:28.430591 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:28.430553 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:29.230296 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:29.230233 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:30.160688 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:30.160624 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:30.560848 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:30.560815 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm"] Oct 31 13:12:30.715661 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:30.710761 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc"] Oct 31 13:12:30.715661 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:30.711015 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" podUID="a108df2b-6cb1-499c-aef6-d0d200b580c1" containerName="istio-proxy" containerID="cri-o://2f6e72f4fe24dbddcf9698765c330de4cb884140f00aec5d8fddae62b1f63189" gracePeriod=30 Oct 31 13:12:30.746828 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:30.746641 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:30.818103 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:30.817437 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6"] Oct 31 13:12:30.818103 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:30.817704 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" podUID="3157a099-6a36-4df2-a4cc-b006777e92b0" containerName="istio-proxy" containerID="cri-o://263c6d69cf6cf1e8ba26ca1d69c8554371f65e1490f0beff0f81ce31eb5da80d" gracePeriod=30 Oct 31 13:12:31.006179 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:31.006135 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:31.290543 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:31.290471 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:32.146886 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:32.146832 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:32.413379 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:32.413288 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:32.842214 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:32.842174 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:33.262530 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:33.262490 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:33.861966 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:33.861938 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f" Oct 31 13:12:34.111906 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:34.111857 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:35.459722 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:35.459686 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:36.008568 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:36.008537 6841 generic.go:356] "Generic (PLEG): container finished" podID="a108df2b-6cb1-499c-aef6-d0d200b580c1" containerID="2f6e72f4fe24dbddcf9698765c330de4cb884140f00aec5d8fddae62b1f63189" exitCode=0 Oct 31 13:12:36.008812 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:36.008778 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" event={"ID":"a108df2b-6cb1-499c-aef6-d0d200b580c1","Type":"ContainerDied","Data":"2f6e72f4fe24dbddcf9698765c330de4cb884140f00aec5d8fddae62b1f63189"} Oct 31 13:12:36.742355 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:36.742089 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:36.823707 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:36.823678 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" Oct 31 13:12:36.952547 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:36.952512 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a108df2b-6cb1-499c-aef6-d0d200b580c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a108df2b-6cb1-499c-aef6-d0d200b580c1-istio-data\") pod \"a108df2b-6cb1-499c-aef6-d0d200b580c1\" (UID: \"a108df2b-6cb1-499c-aef6-d0d200b580c1\") " Oct 31 13:12:36.952840 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:36.952821 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a108df2b-6cb1-499c-aef6-d0d200b580c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a108df2b-6cb1-499c-aef6-d0d200b580c1-workload-certs\") pod \"a108df2b-6cb1-499c-aef6-d0d200b580c1\" (UID: \"a108df2b-6cb1-499c-aef6-d0d200b580c1\") " Oct 31 13:12:36.953041 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:36.953025 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a108df2b-6cb1-499c-aef6-d0d200b580c1-kube-api-access-7hq49\" (UniqueName: \"kubernetes.io/projected/a108df2b-6cb1-499c-aef6-d0d200b580c1-kube-api-access-7hq49\") pod \"a108df2b-6cb1-499c-aef6-d0d200b580c1\" (UID: \"a108df2b-6cb1-499c-aef6-d0d200b580c1\") " Oct 31 13:12:36.953174 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:36.953160 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a108df2b-6cb1-499c-aef6-d0d200b580c1-istio-token\" (UniqueName: \"kubernetes.io/projected/a108df2b-6cb1-499c-aef6-d0d200b580c1-istio-token\") pod \"a108df2b-6cb1-499c-aef6-d0d200b580c1\" (UID: \"a108df2b-6cb1-499c-aef6-d0d200b580c1\") " Oct 31 13:12:36.953315 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:36.953275 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a108df2b-6cb1-499c-aef6-d0d200b580c1-istio-data" pod "a108df2b-6cb1-499c-aef6-d0d200b580c1" (UID: "a108df2b-6cb1-499c-aef6-d0d200b580c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:12:36.953382 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:36.953288 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a108df2b-6cb1-499c-aef6-d0d200b580c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a108df2b-6cb1-499c-aef6-d0d200b580c1-istio-envoy\") pod \"a108df2b-6cb1-499c-aef6-d0d200b580c1\" (UID: \"a108df2b-6cb1-499c-aef6-d0d200b580c1\") " Oct 31 13:12:36.953495 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:36.953472 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a108df2b-6cb1-499c-aef6-d0d200b580c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a108df2b-6cb1-499c-aef6-d0d200b580c1-istiod-ca-cert\") pod \"a108df2b-6cb1-499c-aef6-d0d200b580c1\" (UID: \"a108df2b-6cb1-499c-aef6-d0d200b580c1\") " Oct 31 13:12:36.953563 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:36.953546 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a108df2b-6cb1-499c-aef6-d0d200b580c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a108df2b-6cb1-499c-aef6-d0d200b580c1-workload-socket\") pod \"a108df2b-6cb1-499c-aef6-d0d200b580c1\" (UID: \"a108df2b-6cb1-499c-aef6-d0d200b580c1\") " Oct 31 13:12:36.953691 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:36.953670 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a108df2b-6cb1-499c-aef6-d0d200b580c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a108df2b-6cb1-499c-aef6-d0d200b580c1-credential-socket\") pod \"a108df2b-6cb1-499c-aef6-d0d200b580c1\" (UID: \"a108df2b-6cb1-499c-aef6-d0d200b580c1\") " Oct 31 13:12:36.953831 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:36.953813 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a108df2b-6cb1-499c-aef6-d0d200b580c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a108df2b-6cb1-499c-aef6-d0d200b580c1-istio-podinfo\") pod \"a108df2b-6cb1-499c-aef6-d0d200b580c1\" (UID: \"a108df2b-6cb1-499c-aef6-d0d200b580c1\") " Oct 31 13:12:36.955090 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:36.954926 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a108df2b-6cb1-499c-aef6-d0d200b580c1-istiod-ca-cert" pod "a108df2b-6cb1-499c-aef6-d0d200b580c1" (UID: "a108df2b-6cb1-499c-aef6-d0d200b580c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:12:36.955698 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:36.955645 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a108df2b-6cb1-499c-aef6-d0d200b580c1-credential-socket" pod "a108df2b-6cb1-499c-aef6-d0d200b580c1" (UID: "a108df2b-6cb1-499c-aef6-d0d200b580c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:12:36.956545 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:36.956520 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a108df2b-6cb1-499c-aef6-d0d200b580c1-workload-certs" pod "a108df2b-6cb1-499c-aef6-d0d200b580c1" (UID: "a108df2b-6cb1-499c-aef6-d0d200b580c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:12:36.956893 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:36.956858 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a108df2b-6cb1-499c-aef6-d0d200b580c1-workload-socket" pod "a108df2b-6cb1-499c-aef6-d0d200b580c1" (UID: "a108df2b-6cb1-499c-aef6-d0d200b580c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:12:36.974058 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:36.960152 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a108df2b-6cb1-499c-aef6-d0d200b580c1-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:36.974058 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:36.960183 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a108df2b-6cb1-499c-aef6-d0d200b580c1-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:36.974058 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:36.960200 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a108df2b-6cb1-499c-aef6-d0d200b580c1-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:36.974058 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:36.960216 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a108df2b-6cb1-499c-aef6-d0d200b580c1-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:36.974058 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:36.960230 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a108df2b-6cb1-499c-aef6-d0d200b580c1-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:36.983794 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:36.982673 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a108df2b-6cb1-499c-aef6-d0d200b580c1-istio-envoy" pod "a108df2b-6cb1-499c-aef6-d0d200b580c1" (UID: "a108df2b-6cb1-499c-aef6-d0d200b580c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:12:36.983794 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:36.982759 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a108df2b-6cb1-499c-aef6-d0d200b580c1-istio-podinfo" pod "a108df2b-6cb1-499c-aef6-d0d200b580c1" (UID: "a108df2b-6cb1-499c-aef6-d0d200b580c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:12:36.983794 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:36.982863 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a108df2b-6cb1-499c-aef6-d0d200b580c1-istio-token" pod "a108df2b-6cb1-499c-aef6-d0d200b580c1" (UID: "a108df2b-6cb1-499c-aef6-d0d200b580c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:12:36.983794 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:36.982926 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a108df2b-6cb1-499c-aef6-d0d200b580c1-kube-api-access-7hq49" pod "a108df2b-6cb1-499c-aef6-d0d200b580c1" (UID: "a108df2b-6cb1-499c-aef6-d0d200b580c1"). InnerVolumeSpecName "kube-api-access-7hq49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:12:37.030066 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.029812 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" Oct 31 13:12:37.030066 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.029860 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc" event={"ID":"a108df2b-6cb1-499c-aef6-d0d200b580c1","Type":"ContainerDied","Data":"aff4e989227d1c2f3dd46f4d6b276eaf0a87c2d77470cda9613081f4dc1a5093"} Oct 31 13:12:37.030066 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.029926 6841 scope.go:122] "RemoveContainer" containerID="2f6e72f4fe24dbddcf9698765c330de4cb884140f00aec5d8fddae62b1f63189" Oct 31 13:12:37.033806 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.033527 6841 generic.go:356] "Generic (PLEG): container finished" podID="3157a099-6a36-4df2-a4cc-b006777e92b0" containerID="263c6d69cf6cf1e8ba26ca1d69c8554371f65e1490f0beff0f81ce31eb5da80d" exitCode=0 Oct 31 13:12:37.033806 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.033600 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" event={"ID":"3157a099-6a36-4df2-a4cc-b006777e92b0","Type":"ContainerDied","Data":"263c6d69cf6cf1e8ba26ca1d69c8554371f65e1490f0beff0f81ce31eb5da80d"} Oct 31 13:12:37.053348 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.053195 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:37.062236 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.062210 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a108df2b-6cb1-499c-aef6-d0d200b580c1-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:37.062236 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.062234 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hq49\" (UniqueName: \"kubernetes.io/projected/a108df2b-6cb1-499c-aef6-d0d200b580c1-kube-api-access-7hq49\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:37.062455 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.062245 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a108df2b-6cb1-499c-aef6-d0d200b580c1-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:37.062455 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.062254 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a108df2b-6cb1-499c-aef6-d0d200b580c1-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:37.076448 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.076426 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" Oct 31 13:12:37.105997 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.105958 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc"] Oct 31 13:12:37.117371 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.117337 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-659wc"] Oct 31 13:12:37.165626 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.163377 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3157a099-6a36-4df2-a4cc-b006777e92b0-istio-token\" (UniqueName: \"kubernetes.io/projected/3157a099-6a36-4df2-a4cc-b006777e92b0-istio-token\") pod \"3157a099-6a36-4df2-a4cc-b006777e92b0\" (UID: \"3157a099-6a36-4df2-a4cc-b006777e92b0\") " Oct 31 13:12:37.165626 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.163512 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3157a099-6a36-4df2-a4cc-b006777e92b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3157a099-6a36-4df2-a4cc-b006777e92b0-istiod-ca-cert\") pod \"3157a099-6a36-4df2-a4cc-b006777e92b0\" (UID: \"3157a099-6a36-4df2-a4cc-b006777e92b0\") " Oct 31 13:12:37.165626 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.163640 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3157a099-6a36-4df2-a4cc-b006777e92b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3157a099-6a36-4df2-a4cc-b006777e92b0-workload-socket\") pod \"3157a099-6a36-4df2-a4cc-b006777e92b0\" (UID: \"3157a099-6a36-4df2-a4cc-b006777e92b0\") " Oct 31 13:12:37.165626 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.163689 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3157a099-6a36-4df2-a4cc-b006777e92b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3157a099-6a36-4df2-a4cc-b006777e92b0-istio-data\") pod \"3157a099-6a36-4df2-a4cc-b006777e92b0\" (UID: \"3157a099-6a36-4df2-a4cc-b006777e92b0\") " Oct 31 13:12:37.165626 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.163727 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3157a099-6a36-4df2-a4cc-b006777e92b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3157a099-6a36-4df2-a4cc-b006777e92b0-istio-envoy\") pod \"3157a099-6a36-4df2-a4cc-b006777e92b0\" (UID: \"3157a099-6a36-4df2-a4cc-b006777e92b0\") " Oct 31 13:12:37.165626 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.163780 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3157a099-6a36-4df2-a4cc-b006777e92b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3157a099-6a36-4df2-a4cc-b006777e92b0-istio-podinfo\") pod \"3157a099-6a36-4df2-a4cc-b006777e92b0\" (UID: \"3157a099-6a36-4df2-a4cc-b006777e92b0\") " Oct 31 13:12:37.165626 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.163838 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3157a099-6a36-4df2-a4cc-b006777e92b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3157a099-6a36-4df2-a4cc-b006777e92b0-workload-certs\") pod \"3157a099-6a36-4df2-a4cc-b006777e92b0\" (UID: \"3157a099-6a36-4df2-a4cc-b006777e92b0\") " Oct 31 13:12:37.165626 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.163894 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3157a099-6a36-4df2-a4cc-b006777e92b0-kube-api-access-k9vv9\" (UniqueName: \"kubernetes.io/projected/3157a099-6a36-4df2-a4cc-b006777e92b0-kube-api-access-k9vv9\") pod \"3157a099-6a36-4df2-a4cc-b006777e92b0\" (UID: \"3157a099-6a36-4df2-a4cc-b006777e92b0\") " Oct 31 13:12:37.165626 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.163974 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3157a099-6a36-4df2-a4cc-b006777e92b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3157a099-6a36-4df2-a4cc-b006777e92b0-credential-socket\") pod \"3157a099-6a36-4df2-a4cc-b006777e92b0\" (UID: \"3157a099-6a36-4df2-a4cc-b006777e92b0\") " Oct 31 13:12:37.165626 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.164489 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3157a099-6a36-4df2-a4cc-b006777e92b0-workload-certs" pod "3157a099-6a36-4df2-a4cc-b006777e92b0" (UID: "3157a099-6a36-4df2-a4cc-b006777e92b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:12:37.165626 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.164703 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3157a099-6a36-4df2-a4cc-b006777e92b0-credential-socket" pod "3157a099-6a36-4df2-a4cc-b006777e92b0" (UID: "3157a099-6a36-4df2-a4cc-b006777e92b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:12:37.165626 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.164817 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3157a099-6a36-4df2-a4cc-b006777e92b0-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:37.165626 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.164843 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3157a099-6a36-4df2-a4cc-b006777e92b0-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:37.165626 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.164845 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3157a099-6a36-4df2-a4cc-b006777e92b0-istio-data" pod "3157a099-6a36-4df2-a4cc-b006777e92b0" (UID: "3157a099-6a36-4df2-a4cc-b006777e92b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:12:37.165626 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.165088 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3157a099-6a36-4df2-a4cc-b006777e92b0-istiod-ca-cert" pod "3157a099-6a36-4df2-a4cc-b006777e92b0" (UID: "3157a099-6a36-4df2-a4cc-b006777e92b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:12:37.165626 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.165438 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3157a099-6a36-4df2-a4cc-b006777e92b0-workload-socket" pod "3157a099-6a36-4df2-a4cc-b006777e92b0" (UID: "3157a099-6a36-4df2-a4cc-b006777e92b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:12:37.173469 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.167547 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3157a099-6a36-4df2-a4cc-b006777e92b0-kube-api-access-k9vv9" pod "3157a099-6a36-4df2-a4cc-b006777e92b0" (UID: "3157a099-6a36-4df2-a4cc-b006777e92b0"). InnerVolumeSpecName "kube-api-access-k9vv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:12:37.173469 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.168342 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3157a099-6a36-4df2-a4cc-b006777e92b0-istio-envoy" pod "3157a099-6a36-4df2-a4cc-b006777e92b0" (UID: "3157a099-6a36-4df2-a4cc-b006777e92b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:12:37.173469 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.168393 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3157a099-6a36-4df2-a4cc-b006777e92b0-istio-token" pod "3157a099-6a36-4df2-a4cc-b006777e92b0" (UID: "3157a099-6a36-4df2-a4cc-b006777e92b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:12:37.173469 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.168525 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3157a099-6a36-4df2-a4cc-b006777e92b0-istio-podinfo" pod "3157a099-6a36-4df2-a4cc-b006777e92b0" (UID: "3157a099-6a36-4df2-a4cc-b006777e92b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:12:37.266774 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.266682 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9vv9\" (UniqueName: \"kubernetes.io/projected/3157a099-6a36-4df2-a4cc-b006777e92b0-kube-api-access-k9vv9\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:37.266774 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.266726 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3157a099-6a36-4df2-a4cc-b006777e92b0-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:37.266774 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.266736 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3157a099-6a36-4df2-a4cc-b006777e92b0-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:37.266774 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.266746 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3157a099-6a36-4df2-a4cc-b006777e92b0-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:37.266774 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.266754 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3157a099-6a36-4df2-a4cc-b006777e92b0-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:37.266774 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.266762 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3157a099-6a36-4df2-a4cc-b006777e92b0-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:37.266774 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.266770 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3157a099-6a36-4df2-a4cc-b006777e92b0-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:37.457643 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.457598 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:37.645729 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.645639 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a108df2b-6cb1-499c-aef6-d0d200b580c1" path="/var/lib/kubelet/pods/a108df2b-6cb1-499c-aef6-d0d200b580c1/volumes" Oct 31 13:12:37.930786 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:37.930693 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:38.044026 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:38.043967 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" Oct 31 13:12:38.044026 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:38.044017 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6" event={"ID":"3157a099-6a36-4df2-a4cc-b006777e92b0","Type":"ContainerDied","Data":"93452ff3b5183b37aada4375ff5cc2c416ecad80534b27f3de11c19e3d124755"} Oct 31 13:12:38.044285 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:38.044081 6841 scope.go:122] "RemoveContainer" containerID="263c6d69cf6cf1e8ba26ca1d69c8554371f65e1490f0beff0f81ce31eb5da80d" Oct 31 13:12:38.099084 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:38.099048 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6"] Oct 31 13:12:38.120323 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:38.120294 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-87kl6"] Oct 31 13:12:38.436002 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:38.432009 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:38.873120 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:38.873082 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:39.001404 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:39.001366 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11394-97ca26"] Oct 31 13:12:39.002193 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:39.002175 6841 cpu_manager.go:414] "containerMap: removing container" podUID="3157a099-6a36-4df2-a4cc-b006777e92b0" containerName="istio-proxy" Oct 31 13:12:39.002193 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:39.002194 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3157a099-6a36-4df2-a4cc-b006777e92b0" containerName="istio-proxy" Oct 31 13:12:39.002300 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:39.002208 6841 cpu_manager.go:414] "containerMap: removing container" podUID="a108df2b-6cb1-499c-aef6-d0d200b580c1" containerName="istio-proxy" Oct 31 13:12:39.002300 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:39.002214 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a108df2b-6cb1-499c-aef6-d0d200b580c1" containerName="istio-proxy" Oct 31 13:12:39.002371 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:39.002320 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="3157a099-6a36-4df2-a4cc-b006777e92b0" containerName="istio-proxy" Oct 31 13:12:39.002371 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:39.002329 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="a108df2b-6cb1-499c-aef6-d0d200b580c1" containerName="istio-proxy" Oct 31 13:12:39.008135 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:39.008107 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11394-97ca26" Oct 31 13:12:39.011860 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:39.011835 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11394-97ca26"] Oct 31 13:12:39.013818 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:39.013798 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mn5fx\"" Oct 31 13:12:39.167695 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:39.167605 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqbwq\" (UniqueName: \"kubernetes.io/projected/146dfb47-a6a9-40c8-85c0-5f8739bb457c-kube-api-access-zqbwq\") pod \"test-curl-11394-97ca26\" (UID: \"146dfb47-a6a9-40c8-85c0-5f8739bb457c\") " pod="opendatahub/test-curl-11394-97ca26" Oct 31 13:12:39.268951 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:39.268914 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqbwq\" (UniqueName: \"kubernetes.io/projected/146dfb47-a6a9-40c8-85c0-5f8739bb457c-kube-api-access-zqbwq\") pod \"test-curl-11394-97ca26\" (UID: \"146dfb47-a6a9-40c8-85c0-5f8739bb457c\") " pod="opendatahub/test-curl-11394-97ca26" Oct 31 13:12:39.281477 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:39.281316 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqbwq\" (UniqueName: \"kubernetes.io/projected/146dfb47-a6a9-40c8-85c0-5f8739bb457c-kube-api-access-zqbwq\") pod \"test-curl-11394-97ca26\" (UID: \"146dfb47-a6a9-40c8-85c0-5f8739bb457c\") " pod="opendatahub/test-curl-11394-97ca26" Oct 31 13:12:39.324406 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:39.324371 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11394-97ca26" Oct 31 13:12:39.447147 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:39.447109 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:39.644652 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:39.644620 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3157a099-6a36-4df2-a4cc-b006777e92b0" path="/var/lib/kubelet/pods/3157a099-6a36-4df2-a4cc-b006777e92b0/volumes" Oct 31 13:12:39.703922 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:39.703891 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11394-97ca26"] Oct 31 13:12:39.704485 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:12:39.704464 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod146dfb47_a6a9_40c8_85c0_5f8739bb457c.slice/crio-4ebd95777d8d7bbf0efdc7e60f405fdad0ee6199f961926f8d00ab7f17731f80 WatchSource:0}: Error finding container 4ebd95777d8d7bbf0efdc7e60f405fdad0ee6199f961926f8d00ab7f17731f80: Status 404 returned error can't find the container with id 4ebd95777d8d7bbf0efdc7e60f405fdad0ee6199f961926f8d00ab7f17731f80 Oct 31 13:12:40.107946 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:40.107851 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11394-97ca26" event={"ID":"146dfb47-a6a9-40c8-85c0-5f8739bb457c","Type":"ContainerStarted","Data":"c3c8b2a3b60628bcfedf313cc15b2f022f2f981eae45fd7d5b21f7db0212fd81"} Oct 31 13:12:40.108614 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:40.108573 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11394-97ca26" event={"ID":"146dfb47-a6a9-40c8-85c0-5f8739bb457c","Type":"ContainerStarted","Data":"4ebd95777d8d7bbf0efdc7e60f405fdad0ee6199f961926f8d00ab7f17731f80"} Oct 31 13:12:40.132130 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:40.132076 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11394-97ca26" podStartSLOduration=2.132057392 podStartE2EDuration="2.132057392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:40.12909683 +0000 UTC m=+2406.612219731" watchObservedRunningTime="2026-10-31 13:12:40.132057392 +0000 UTC m=+2406.615180294" Oct 31 13:12:40.437511 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:40.437236 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:40.966032 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:40.965961 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11394-97ca26"] Oct 31 13:12:40.986545 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:40.986508 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11394-97ca26"] Oct 31 13:12:41.392920 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:41.392821 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:42.122776 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:42.122737 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11394-97ca26" podUID="146dfb47-a6a9-40c8-85c0-5f8739bb457c" containerName="test-curl-11394-97ca26" containerID="cri-o://c3c8b2a3b60628bcfedf313cc15b2f022f2f981eae45fd7d5b21f7db0212fd81" gracePeriod=2 Oct 31 13:12:42.217625 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:42.217223 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:43.012810 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:43.012773 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:43.912965 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:43.912903 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:44.354742 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:44.354718 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11394-97ca26" Oct 31 13:12:44.403631 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:44.403594 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/146dfb47-a6a9-40c8-85c0-5f8739bb457c-kube-api-access-zqbwq\" (UniqueName: \"kubernetes.io/projected/146dfb47-a6a9-40c8-85c0-5f8739bb457c-kube-api-access-zqbwq\") pod \"146dfb47-a6a9-40c8-85c0-5f8739bb457c\" (UID: \"146dfb47-a6a9-40c8-85c0-5f8739bb457c\") " Oct 31 13:12:44.405802 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:44.405769 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146dfb47-a6a9-40c8-85c0-5f8739bb457c-kube-api-access-zqbwq" pod "146dfb47-a6a9-40c8-85c0-5f8739bb457c" (UID: "146dfb47-a6a9-40c8-85c0-5f8739bb457c"). InnerVolumeSpecName "kube-api-access-zqbwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:12:44.505880 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:44.505850 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqbwq\" (UniqueName: \"kubernetes.io/projected/146dfb47-a6a9-40c8-85c0-5f8739bb457c-kube-api-access-zqbwq\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:44.834054 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:44.833915 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:45.146889 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:45.146804 6841 generic.go:356] "Generic (PLEG): container finished" podID="146dfb47-a6a9-40c8-85c0-5f8739bb457c" containerID="c3c8b2a3b60628bcfedf313cc15b2f022f2f981eae45fd7d5b21f7db0212fd81" exitCode=137 Oct 31 13:12:45.146889 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:45.146844 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11394-97ca26" Oct 31 13:12:45.147142 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:45.146893 6841 scope.go:122] "RemoveContainer" containerID="c3c8b2a3b60628bcfedf313cc15b2f022f2f981eae45fd7d5b21f7db0212fd81" Oct 31 13:12:45.157889 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:45.157869 6841 scope.go:122] "RemoveContainer" containerID="c3c8b2a3b60628bcfedf313cc15b2f022f2f981eae45fd7d5b21f7db0212fd81" Oct 31 13:12:45.158174 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:12:45.158153 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3c8b2a3b60628bcfedf313cc15b2f022f2f981eae45fd7d5b21f7db0212fd81\": container with ID starting with c3c8b2a3b60628bcfedf313cc15b2f022f2f981eae45fd7d5b21f7db0212fd81 not found: ID does not exist" containerID="c3c8b2a3b60628bcfedf313cc15b2f022f2f981eae45fd7d5b21f7db0212fd81" Oct 31 13:12:45.158235 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:45.158183 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3c8b2a3b60628bcfedf313cc15b2f022f2f981eae45fd7d5b21f7db0212fd81"} err="failed to get container status \"c3c8b2a3b60628bcfedf313cc15b2f022f2f981eae45fd7d5b21f7db0212fd81\": rpc error: code = NotFound desc = could not find container \"c3c8b2a3b60628bcfedf313cc15b2f022f2f981eae45fd7d5b21f7db0212fd81\": container with ID starting with c3c8b2a3b60628bcfedf313cc15b2f022f2f981eae45fd7d5b21f7db0212fd81 not found: ID does not exist" Oct 31 13:12:45.158790 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:12:45.158768 6841 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11394-97ca26\" is forbidden: User \"system:node:ip-10-0-138-70.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-70.us-east-2.compute.internal' and this object" podUID="146dfb47-a6a9-40c8-85c0-5f8739bb457c" pod="opendatahub/test-curl-11394-97ca26" Oct 31 13:12:45.612620 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:45.612571 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm" podUID="09e7a6fc-9d99-47f3-aae0-310859bcc6bb" containerName="main" containerID="cri-o://5e0f3a4abb4229f77cd7b0e4e9acb534be79e792f0596b42d8d6bc4c1f8bdb22" gracePeriod=45 Oct 31 13:12:45.645246 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:45.645216 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:45.645480 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:45.645458 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="146dfb47-a6a9-40c8-85c0-5f8739bb457c" path="/var/lib/kubelet/pods/146dfb47-a6a9-40c8-85c0-5f8739bb457c/volumes" Oct 31 13:12:45.914594 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:45.914570 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm" Oct 31 13:12:46.029227 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:46.029191 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-kube-api-access-2sf75\" (UniqueName: \"kubernetes.io/projected/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-kube-api-access-2sf75\") pod \"09e7a6fc-9d99-47f3-aae0-310859bcc6bb\" (UID: \"09e7a6fc-9d99-47f3-aae0-310859bcc6bb\") " Oct 31 13:12:46.029429 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:46.029272 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-tmp-dir\") pod \"09e7a6fc-9d99-47f3-aae0-310859bcc6bb\" (UID: \"09e7a6fc-9d99-47f3-aae0-310859bcc6bb\") " Oct 31 13:12:46.029429 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:46.029327 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-home\" (UniqueName: \"kubernetes.io/empty-dir/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-home\") pod \"09e7a6fc-9d99-47f3-aae0-310859bcc6bb\" (UID: \"09e7a6fc-9d99-47f3-aae0-310859bcc6bb\") " Oct 31 13:12:46.029429 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:46.029403 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-model-cache\") pod \"09e7a6fc-9d99-47f3-aae0-310859bcc6bb\" (UID: \"09e7a6fc-9d99-47f3-aae0-310859bcc6bb\") " Oct 31 13:12:46.029614 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:46.029484 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-tls-certs\" (UniqueName: \"kubernetes.io/secret/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-tls-certs\") pod \"09e7a6fc-9d99-47f3-aae0-310859bcc6bb\" (UID: \"09e7a6fc-9d99-47f3-aae0-310859bcc6bb\") " Oct 31 13:12:46.029614 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:46.029537 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-dshm\") pod \"09e7a6fc-9d99-47f3-aae0-310859bcc6bb\" (UID: \"09e7a6fc-9d99-47f3-aae0-310859bcc6bb\") " Oct 31 13:12:46.029614 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:46.029573 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-tmp-dir" pod "09e7a6fc-9d99-47f3-aae0-310859bcc6bb" (UID: "09e7a6fc-9d99-47f3-aae0-310859bcc6bb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:12:46.029730 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:46.029615 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-home" pod "09e7a6fc-9d99-47f3-aae0-310859bcc6bb" (UID: "09e7a6fc-9d99-47f3-aae0-310859bcc6bb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:12:46.029730 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:46.029655 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-model-cache" pod "09e7a6fc-9d99-47f3-aae0-310859bcc6bb" (UID: "09e7a6fc-9d99-47f3-aae0-310859bcc6bb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:12:46.030276 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:46.030254 6841 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-model-cache\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:46.030276 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:46.030275 6841 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-tmp-dir\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:46.030470 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:46.030285 6841 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-home\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:46.031653 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:46.031629 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-dshm" pod "09e7a6fc-9d99-47f3-aae0-310859bcc6bb" (UID: "09e7a6fc-9d99-47f3-aae0-310859bcc6bb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:12:46.031653 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:46.031640 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-tls-certs" pod "09e7a6fc-9d99-47f3-aae0-310859bcc6bb" (UID: "09e7a6fc-9d99-47f3-aae0-310859bcc6bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:12:46.031778 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:46.031665 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-kube-api-access-2sf75" pod "09e7a6fc-9d99-47f3-aae0-310859bcc6bb" (UID: "09e7a6fc-9d99-47f3-aae0-310859bcc6bb"). InnerVolumeSpecName "kube-api-access-2sf75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:12:46.131661 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:46.131618 6841 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-tls-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:46.131661 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:46.131658 6841 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-dshm\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:46.131661 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:46.131669 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sf75\" (UniqueName: \"kubernetes.io/projected/09e7a6fc-9d99-47f3-aae0-310859bcc6bb-kube-api-access-2sf75\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:46.156345 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:46.156275 6841 generic.go:356] "Generic (PLEG): container finished" podID="09e7a6fc-9d99-47f3-aae0-310859bcc6bb" containerID="5e0f3a4abb4229f77cd7b0e4e9acb534be79e792f0596b42d8d6bc4c1f8bdb22" exitCode=0 Oct 31 13:12:46.156345 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:46.156318 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm" Oct 31 13:12:46.156545 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:46.156366 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm" event={"ID":"09e7a6fc-9d99-47f3-aae0-310859bcc6bb","Type":"ContainerDied","Data":"5e0f3a4abb4229f77cd7b0e4e9acb534be79e792f0596b42d8d6bc4c1f8bdb22"} Oct 31 13:12:46.156545 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:46.156415 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm" event={"ID":"09e7a6fc-9d99-47f3-aae0-310859bcc6bb","Type":"ContainerDied","Data":"5fb704739815bb82b1c7a08364f780d2d9835b192c0fbef614704f09fa6f901d"} Oct 31 13:12:46.156545 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:46.156440 6841 scope.go:122] "RemoveContainer" containerID="5e0f3a4abb4229f77cd7b0e4e9acb534be79e792f0596b42d8d6bc4c1f8bdb22" Oct 31 13:12:46.168058 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:46.168042 6841 scope.go:122] "RemoveContainer" containerID="5e0f3a4abb4229f77cd7b0e4e9acb534be79e792f0596b42d8d6bc4c1f8bdb22" Oct 31 13:12:46.168326 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:12:46.168306 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e0f3a4abb4229f77cd7b0e4e9acb534be79e792f0596b42d8d6bc4c1f8bdb22\": container with ID starting with 5e0f3a4abb4229f77cd7b0e4e9acb534be79e792f0596b42d8d6bc4c1f8bdb22 not found: ID does not exist" containerID="5e0f3a4abb4229f77cd7b0e4e9acb534be79e792f0596b42d8d6bc4c1f8bdb22" Oct 31 13:12:46.168401 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:46.168337 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e0f3a4abb4229f77cd7b0e4e9acb534be79e792f0596b42d8d6bc4c1f8bdb22"} err="failed to get container status \"5e0f3a4abb4229f77cd7b0e4e9acb534be79e792f0596b42d8d6bc4c1f8bdb22\": rpc error: code = NotFound desc = could not find container \"5e0f3a4abb4229f77cd7b0e4e9acb534be79e792f0596b42d8d6bc4c1f8bdb22\": container with ID starting with 5e0f3a4abb4229f77cd7b0e4e9acb534be79e792f0596b42d8d6bc4c1f8bdb22 not found: ID does not exist" Oct 31 13:12:46.192229 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:46.192203 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm"] Oct 31 13:12:46.199027 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:46.199003 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-377f60/auth-test-model-377f60-kserve-5db5968cb9-2n2lm"] Oct 31 13:12:46.560425 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:46.560326 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:47.447847 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:47.447801 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:47.643948 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:47.643912 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09e7a6fc-9d99-47f3-aae0-310859bcc6bb" path="/var/lib/kubelet/pods/09e7a6fc-9d99-47f3-aae0-310859bcc6bb/volumes" Oct 31 13:12:47.716482 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:47.716392 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:48.440777 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:48.440746 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:48.933646 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:48.933616 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:49.325495 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.325463 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d"] Oct 31 13:12:49.326274 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.326255 6841 cpu_manager.go:414] "containerMap: removing container" podUID="146dfb47-a6a9-40c8-85c0-5f8739bb457c" containerName="test-curl-11394-97ca26" Oct 31 13:12:49.326343 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.326275 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="146dfb47-a6a9-40c8-85c0-5f8739bb457c" containerName="test-curl-11394-97ca26" Oct 31 13:12:49.326343 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.326309 6841 cpu_manager.go:414] "containerMap: removing container" podUID="09e7a6fc-9d99-47f3-aae0-310859bcc6bb" containerName="main" Oct 31 13:12:49.326343 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.326314 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09e7a6fc-9d99-47f3-aae0-310859bcc6bb" containerName="main" Oct 31 13:12:49.326460 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.326432 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="146dfb47-a6a9-40c8-85c0-5f8739bb457c" containerName="test-curl-11394-97ca26" Oct 31 13:12:49.326460 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.326446 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="09e7a6fc-9d99-47f3-aae0-310859bcc6bb" containerName="main" Oct 31 13:12:49.331556 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.331535 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:12:49.350862 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.350836 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-15a6a2-openshift-default-dockercfg-kg58k\"" Oct 31 13:12:49.355427 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.355405 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d"] Oct 31 13:12:49.365103 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.365072 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:49.389059 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.389027 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdaa9d86-a826-4768-ae0d-471ce6e65e15-istiod-ca-cert\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d\" (UID: \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:12:49.389208 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.389154 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdaa9d86-a826-4768-ae0d-471ce6e65e15-workload-socket\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d\" (UID: \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:12:49.389208 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.389187 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdaa9d86-a826-4768-ae0d-471ce6e65e15-credential-socket\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d\" (UID: \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:12:49.389335 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.389314 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdaa9d86-a826-4768-ae0d-471ce6e65e15-istio-token\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d\" (UID: \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:12:49.389407 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.389352 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdaa9d86-a826-4768-ae0d-471ce6e65e15-istio-envoy\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d\" (UID: \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:12:49.389407 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.389397 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdaa9d86-a826-4768-ae0d-471ce6e65e15-istio-data\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d\" (UID: \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:12:49.389601 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.389509 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdaa9d86-a826-4768-ae0d-471ce6e65e15-istio-podinfo\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d\" (UID: \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:12:49.389601 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.389560 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4zsz\" (UniqueName: \"kubernetes.io/projected/bdaa9d86-a826-4768-ae0d-471ce6e65e15-kube-api-access-m4zsz\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d\" (UID: \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:12:49.389707 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.389612 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdaa9d86-a826-4768-ae0d-471ce6e65e15-workload-certs\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d\" (UID: \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:12:49.549008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.548018 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdaa9d86-a826-4768-ae0d-471ce6e65e15-istio-podinfo\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d\" (UID: \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:12:49.549008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.548107 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4zsz\" (UniqueName: \"kubernetes.io/projected/bdaa9d86-a826-4768-ae0d-471ce6e65e15-kube-api-access-m4zsz\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d\" (UID: \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:12:49.549008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.548148 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdaa9d86-a826-4768-ae0d-471ce6e65e15-workload-certs\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d\" (UID: \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:12:49.549008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.548341 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdaa9d86-a826-4768-ae0d-471ce6e65e15-istiod-ca-cert\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d\" (UID: \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:12:49.549008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.548423 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdaa9d86-a826-4768-ae0d-471ce6e65e15-workload-socket\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d\" (UID: \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:12:49.549008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.548454 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdaa9d86-a826-4768-ae0d-471ce6e65e15-credential-socket\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d\" (UID: \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:12:49.549008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.548519 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdaa9d86-a826-4768-ae0d-471ce6e65e15-istio-token\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d\" (UID: \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:12:49.549008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.548551 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdaa9d86-a826-4768-ae0d-471ce6e65e15-istio-envoy\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d\" (UID: \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:12:49.549008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.548591 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdaa9d86-a826-4768-ae0d-471ce6e65e15-istio-data\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d\" (UID: \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:12:49.553942 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.553908 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdaa9d86-a826-4768-ae0d-471ce6e65e15-workload-socket\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d\" (UID: \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:12:49.576057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.554712 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdaa9d86-a826-4768-ae0d-471ce6e65e15-workload-certs\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d\" (UID: \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:12:49.577148 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.577116 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdaa9d86-a826-4768-ae0d-471ce6e65e15-istiod-ca-cert\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d\" (UID: \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:12:49.603995 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.600253 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdaa9d86-a826-4768-ae0d-471ce6e65e15-credential-socket\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d\" (UID: \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:12:49.603995 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.600994 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:49.603995 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.601328 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdaa9d86-a826-4768-ae0d-471ce6e65e15-istio-data\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d\" (UID: \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:12:49.604385 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.604357 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdaa9d86-a826-4768-ae0d-471ce6e65e15-istio-envoy\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d\" (UID: \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:12:49.618784 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.618752 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4zsz\" (UniqueName: \"kubernetes.io/projected/bdaa9d86-a826-4768-ae0d-471ce6e65e15-kube-api-access-m4zsz\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d\" (UID: \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:12:49.661004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.655718 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdaa9d86-a826-4768-ae0d-471ce6e65e15-istio-token\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d\" (UID: \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:12:49.709052 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.703965 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdaa9d86-a826-4768-ae0d-471ce6e65e15-istio-podinfo\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d\" (UID: \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:12:49.952055 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:49.945968 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:12:50.050140 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:12:50.050090 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:12:50.059463 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:50.059305 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:50.257698 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:12:50.257668 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdaa9d86_a826_4768_ae0d_471ce6e65e15.slice/crio-f2072ebde24061bb88ed6424649003e54210db00862a65744a90c90ae026ad6d WatchSource:0}: Error finding container f2072ebde24061bb88ed6424649003e54210db00862a65744a90c90ae026ad6d: Status 404 returned error can't find the container with id f2072ebde24061bb88ed6424649003e54210db00862a65744a90c90ae026ad6d Oct 31 13:12:50.258060 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:50.258037 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d"] Oct 31 13:12:50.260418 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:50.260387 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:12:50.260511 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:50.260448 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:12:50.260511 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:50.260477 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:12:50.529058 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:50.528860 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:51.208461 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:51.208426 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" event={"ID":"bdaa9d86-a826-4768-ae0d-471ce6e65e15","Type":"ContainerStarted","Data":"0b855bab5c92549530dbfaddf0071e0362ee969a7fc687a03a902ac4b17cabcd"} Oct 31 13:12:51.208945 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:51.208481 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" event={"ID":"bdaa9d86-a826-4768-ae0d-471ce6e65e15","Type":"ContainerStarted","Data":"f2072ebde24061bb88ed6424649003e54210db00862a65744a90c90ae026ad6d"} Oct 31 13:12:51.246694 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:51.246642 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" podStartSLOduration=2.246624991 podStartE2EDuration="2.246624991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:51.243597487 +0000 UTC m=+2417.726720389" watchObservedRunningTime="2026-10-31 13:12:51.246624991 +0000 UTC m=+2417.729747893" Oct 31 13:12:51.461633 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:51.461546 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:51.946668 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:51.946642 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:12:51.954187 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:51.954159 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:12:52.109631 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:52.109592 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:52.216313 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:52.216225 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:12:52.217127 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:52.217107 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:12:52.850601 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:52.850570 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42"] Oct 31 13:12:52.855306 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:52.855282 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42" Oct 31 13:12:52.863519 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:52.863484 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-9ef744-serving-cert\"" Oct 31 13:12:52.866139 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:52.866110 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42"] Oct 31 13:12:52.923910 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:52.923860 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8437acf6-79cd-4e8e-b0d5-9e81f3602748-metrics-certs\") pod \"maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42\" (UID: \"8437acf6-79cd-4e8e-b0d5-9e81f3602748\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42" Oct 31 13:12:52.924085 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:52.923928 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp98s\" (UniqueName: \"kubernetes.io/projected/8437acf6-79cd-4e8e-b0d5-9e81f3602748-kube-api-access-qp98s\") pod \"maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42\" (UID: \"8437acf6-79cd-4e8e-b0d5-9e81f3602748\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42" Oct 31 13:12:52.924241 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:52.924211 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8437acf6-79cd-4e8e-b0d5-9e81f3602748-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42\" (UID: \"8437acf6-79cd-4e8e-b0d5-9e81f3602748\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42" Oct 31 13:12:52.924344 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:52.924324 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8437acf6-79cd-4e8e-b0d5-9e81f3602748-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42\" (UID: \"8437acf6-79cd-4e8e-b0d5-9e81f3602748\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42" Oct 31 13:12:53.047003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:53.043968 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8437acf6-79cd-4e8e-b0d5-9e81f3602748-metrics-certs\") pod \"maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42\" (UID: \"8437acf6-79cd-4e8e-b0d5-9e81f3602748\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42" Oct 31 13:12:53.047003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:53.044101 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp98s\" (UniqueName: \"kubernetes.io/projected/8437acf6-79cd-4e8e-b0d5-9e81f3602748-kube-api-access-qp98s\") pod \"maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42\" (UID: \"8437acf6-79cd-4e8e-b0d5-9e81f3602748\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42" Oct 31 13:12:53.047003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:53.044366 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8437acf6-79cd-4e8e-b0d5-9e81f3602748-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42\" (UID: \"8437acf6-79cd-4e8e-b0d5-9e81f3602748\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42" Oct 31 13:12:53.047003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:53.044463 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8437acf6-79cd-4e8e-b0d5-9e81f3602748-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42\" (UID: \"8437acf6-79cd-4e8e-b0d5-9e81f3602748\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42" Oct 31 13:12:53.047003 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:12:53.044851 6841 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ef744-serving-cert: secret "maas-api-e2e-extmod-a-9ef744-serving-cert" not found Oct 31 13:12:53.047003 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:12:53.044928 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8437acf6-79cd-4e8e-b0d5-9e81f3602748-maas-api-tls podName:8437acf6-79cd-4e8e-b0d5-9e81f3602748 nodeName:}" failed. No retries permitted until 2026-10-31 13:12:53.544906985 +0000 UTC m=+2420.028029874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8437acf6-79cd-4e8e-b0d5-9e81f3602748-maas-api-tls") pod "maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42" (UID: "8437acf6-79cd-4e8e-b0d5-9e81f3602748") : secret "maas-api-e2e-extmod-a-9ef744-serving-cert" not found Oct 31 13:12:53.047003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:53.045293 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8437acf6-79cd-4e8e-b0d5-9e81f3602748-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42\" (UID: \"8437acf6-79cd-4e8e-b0d5-9e81f3602748\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42" Oct 31 13:12:53.086106 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:53.083822 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8437acf6-79cd-4e8e-b0d5-9e81f3602748-metrics-certs\") pod \"maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42\" (UID: \"8437acf6-79cd-4e8e-b0d5-9e81f3602748\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42" Oct 31 13:12:53.086106 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:53.085836 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp98s\" (UniqueName: \"kubernetes.io/projected/8437acf6-79cd-4e8e-b0d5-9e81f3602748-kube-api-access-qp98s\") pod \"maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42\" (UID: \"8437acf6-79cd-4e8e-b0d5-9e81f3602748\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42" Oct 31 13:12:53.108056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:53.098220 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:53.320360 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:53.320329 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:53.568016 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:53.567178 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8437acf6-79cd-4e8e-b0d5-9e81f3602748-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42\" (UID: \"8437acf6-79cd-4e8e-b0d5-9e81f3602748\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42" Oct 31 13:12:53.617003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:53.615876 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8437acf6-79cd-4e8e-b0d5-9e81f3602748-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42\" (UID: \"8437acf6-79cd-4e8e-b0d5-9e81f3602748\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42" Oct 31 13:12:53.789008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:53.778563 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42" Oct 31 13:12:53.837353 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:53.837270 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:54.082401 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:54.082368 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:54.326559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:54.326532 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42"] Oct 31 13:12:54.328944 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:12:54.328914 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8437acf6_79cd_4e8e_b0d5_9e81f3602748.slice/crio-b5ad6d33468e2b6aa96f0fc9044f2a2603816a748ab9fe6afad615ae6c12f021 WatchSource:0}: Error finding container b5ad6d33468e2b6aa96f0fc9044f2a2603816a748ab9fe6afad615ae6c12f021: Status 404 returned error can't find the container with id b5ad6d33468e2b6aa96f0fc9044f2a2603816a748ab9fe6afad615ae6c12f021 Oct 31 13:12:54.363350 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:54.363314 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:54.976726 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:54.976685 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:55.254610 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:55.254575 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:55.324467 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:55.324411 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42" event={"ID":"8437acf6-79cd-4e8e-b0d5-9e81f3602748","Type":"ContainerStarted","Data":"4e45b5dcb76377df37506916d7d8295e58df389826c2030d8f3cf9b7cfe9e1d3"} Oct 31 13:12:55.324686 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:55.324583 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42" event={"ID":"8437acf6-79cd-4e8e-b0d5-9e81f3602748","Type":"ContainerStarted","Data":"b5ad6d33468e2b6aa96f0fc9044f2a2603816a748ab9fe6afad615ae6c12f021"} Oct 31 13:12:55.325841 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:55.325818 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42" Oct 31 13:12:55.378052 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:55.377997 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42" podStartSLOduration=2.9570350960000003 podStartE2EDuration="3.377965635s" totalImagesPullingTime="420.930539ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:55.373587217 +0000 UTC m=+2421.856710115" watchObservedRunningTime="2026-10-31 13:12:55.377965635 +0000 UTC m=+2421.861088537" Oct 31 13:12:55.631465 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:55.631316 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:56.584977 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:56.584932 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:57.334635 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:57.325883 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:58.145484 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:58.145449 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:12:59.044693 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:12:59.044659 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:00.016864 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:00.016830 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:01.132331 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:01.132287 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:01.566093 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:01.566055 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:01.954278 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:01.947724 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:02.351469 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:02.351382 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42" Oct 31 13:13:02.856764 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:02.856723 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:02.910318 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:02.910287 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr"] Oct 31 13:13:02.914808 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:02.914788 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" Oct 31 13:13:02.927405 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:02.927379 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-5452fc-openshift-default-dockercfg-vgclq\"" Oct 31 13:13:02.931408 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:02.931136 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr"] Oct 31 13:13:02.966730 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:02.966703 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24ea07e9-65f1-4208-b5db-1d283caf05e1-istio-envoy\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr\" (UID: \"24ea07e9-65f1-4208-b5db-1d283caf05e1\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" Oct 31 13:13:02.966909 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:02.966737 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95vh4\" (UniqueName: \"kubernetes.io/projected/24ea07e9-65f1-4208-b5db-1d283caf05e1-kube-api-access-95vh4\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr\" (UID: \"24ea07e9-65f1-4208-b5db-1d283caf05e1\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" Oct 31 13:13:02.966909 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:02.966893 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24ea07e9-65f1-4208-b5db-1d283caf05e1-istio-podinfo\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr\" (UID: \"24ea07e9-65f1-4208-b5db-1d283caf05e1\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" Oct 31 13:13:02.967067 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:02.966995 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24ea07e9-65f1-4208-b5db-1d283caf05e1-workload-socket\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr\" (UID: \"24ea07e9-65f1-4208-b5db-1d283caf05e1\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" Oct 31 13:13:02.967067 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:02.967038 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24ea07e9-65f1-4208-b5db-1d283caf05e1-istio-token\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr\" (UID: \"24ea07e9-65f1-4208-b5db-1d283caf05e1\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" Oct 31 13:13:02.967181 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:02.967143 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24ea07e9-65f1-4208-b5db-1d283caf05e1-credential-socket\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr\" (UID: \"24ea07e9-65f1-4208-b5db-1d283caf05e1\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" Oct 31 13:13:02.967333 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:02.967317 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24ea07e9-65f1-4208-b5db-1d283caf05e1-workload-certs\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr\" (UID: \"24ea07e9-65f1-4208-b5db-1d283caf05e1\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" Oct 31 13:13:02.967467 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:02.967450 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24ea07e9-65f1-4208-b5db-1d283caf05e1-istiod-ca-cert\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr\" (UID: \"24ea07e9-65f1-4208-b5db-1d283caf05e1\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" Oct 31 13:13:02.967543 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:02.967529 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24ea07e9-65f1-4208-b5db-1d283caf05e1-istio-data\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr\" (UID: \"24ea07e9-65f1-4208-b5db-1d283caf05e1\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" Oct 31 13:13:03.069406 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:03.069369 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24ea07e9-65f1-4208-b5db-1d283caf05e1-workload-certs\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr\" (UID: \"24ea07e9-65f1-4208-b5db-1d283caf05e1\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" Oct 31 13:13:03.069586 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:03.069464 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24ea07e9-65f1-4208-b5db-1d283caf05e1-istiod-ca-cert\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr\" (UID: \"24ea07e9-65f1-4208-b5db-1d283caf05e1\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" Oct 31 13:13:03.069586 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:03.069507 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24ea07e9-65f1-4208-b5db-1d283caf05e1-istio-data\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr\" (UID: \"24ea07e9-65f1-4208-b5db-1d283caf05e1\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" Oct 31 13:13:03.069586 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:03.069527 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24ea07e9-65f1-4208-b5db-1d283caf05e1-istio-envoy\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr\" (UID: \"24ea07e9-65f1-4208-b5db-1d283caf05e1\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" Oct 31 13:13:03.069586 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:03.069545 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95vh4\" (UniqueName: \"kubernetes.io/projected/24ea07e9-65f1-4208-b5db-1d283caf05e1-kube-api-access-95vh4\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr\" (UID: \"24ea07e9-65f1-4208-b5db-1d283caf05e1\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" Oct 31 13:13:03.069791 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:03.069610 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24ea07e9-65f1-4208-b5db-1d283caf05e1-istio-podinfo\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr\" (UID: \"24ea07e9-65f1-4208-b5db-1d283caf05e1\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" Oct 31 13:13:03.069791 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:03.069647 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24ea07e9-65f1-4208-b5db-1d283caf05e1-workload-socket\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr\" (UID: \"24ea07e9-65f1-4208-b5db-1d283caf05e1\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" Oct 31 13:13:03.069901 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:03.069784 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24ea07e9-65f1-4208-b5db-1d283caf05e1-istio-token\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr\" (UID: \"24ea07e9-65f1-4208-b5db-1d283caf05e1\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" Oct 31 13:13:03.069901 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:03.069857 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24ea07e9-65f1-4208-b5db-1d283caf05e1-workload-certs\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr\" (UID: \"24ea07e9-65f1-4208-b5db-1d283caf05e1\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" Oct 31 13:13:03.070065 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:03.069911 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24ea07e9-65f1-4208-b5db-1d283caf05e1-credential-socket\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr\" (UID: \"24ea07e9-65f1-4208-b5db-1d283caf05e1\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" Oct 31 13:13:03.070065 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:03.069976 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24ea07e9-65f1-4208-b5db-1d283caf05e1-workload-socket\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr\" (UID: \"24ea07e9-65f1-4208-b5db-1d283caf05e1\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" Oct 31 13:13:03.070331 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:03.070298 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24ea07e9-65f1-4208-b5db-1d283caf05e1-istio-data\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr\" (UID: \"24ea07e9-65f1-4208-b5db-1d283caf05e1\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" Oct 31 13:13:03.070448 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:03.070312 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24ea07e9-65f1-4208-b5db-1d283caf05e1-istiod-ca-cert\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr\" (UID: \"24ea07e9-65f1-4208-b5db-1d283caf05e1\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" Oct 31 13:13:03.070448 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:03.070312 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24ea07e9-65f1-4208-b5db-1d283caf05e1-credential-socket\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr\" (UID: \"24ea07e9-65f1-4208-b5db-1d283caf05e1\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" Oct 31 13:13:03.072543 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:03.072517 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24ea07e9-65f1-4208-b5db-1d283caf05e1-istio-envoy\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr\" (UID: \"24ea07e9-65f1-4208-b5db-1d283caf05e1\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" Oct 31 13:13:03.072722 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:03.072706 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24ea07e9-65f1-4208-b5db-1d283caf05e1-istio-podinfo\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr\" (UID: \"24ea07e9-65f1-4208-b5db-1d283caf05e1\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" Oct 31 13:13:03.082618 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:03.082587 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24ea07e9-65f1-4208-b5db-1d283caf05e1-istio-token\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr\" (UID: \"24ea07e9-65f1-4208-b5db-1d283caf05e1\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" Oct 31 13:13:03.082713 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:03.082637 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95vh4\" (UniqueName: \"kubernetes.io/projected/24ea07e9-65f1-4208-b5db-1d283caf05e1-kube-api-access-95vh4\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr\" (UID: \"24ea07e9-65f1-4208-b5db-1d283caf05e1\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" Oct 31 13:13:03.232950 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:03.232916 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" Oct 31 13:13:03.531858 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:03.531830 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr"] Oct 31 13:13:03.532506 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:13:03.532475 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ea07e9_65f1_4208_b5db_1d283caf05e1.slice/crio-2658dd7c48f816b5e2b636c3afeddc70150fd1406f2b51f6a21397364af2fede WatchSource:0}: Error finding container 2658dd7c48f816b5e2b636c3afeddc70150fd1406f2b51f6a21397364af2fede: Status 404 returned error can't find the container with id 2658dd7c48f816b5e2b636c3afeddc70150fd1406f2b51f6a21397364af2fede Oct 31 13:13:03.535538 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:03.535503 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:13:03.535657 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:03.535618 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:13:03.535727 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:03.535668 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:13:03.594730 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:03.594705 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh"] Oct 31 13:13:03.796064 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:03.786683 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:04.082741 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:04.082640 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:04.421808 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:04.421719 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" event={"ID":"24ea07e9-65f1-4208-b5db-1d283caf05e1","Type":"ContainerStarted","Data":"e2d1183359ee3db9a2e86c3413bffd2ee936d17d29108ce4a7ebe33d93977f37"} Oct 31 13:13:04.421808 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:04.421781 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" event={"ID":"24ea07e9-65f1-4208-b5db-1d283caf05e1","Type":"ContainerStarted","Data":"2658dd7c48f816b5e2b636c3afeddc70150fd1406f2b51f6a21397364af2fede"} Oct 31 13:13:04.452853 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:04.452797 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" podStartSLOduration=2.452783219 podStartE2EDuration="2.452783219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:04.452064564 +0000 UTC m=+2430.935187465" watchObservedRunningTime="2026-10-31 13:13:04.452783219 +0000 UTC m=+2430.935906120" Oct 31 13:13:04.949682 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:04.949633 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:05.233125 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:05.233033 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" Oct 31 13:13:05.235057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:05.234671 6841 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 31 13:13:05.235057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:05.234731 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" podUID="24ea07e9-65f1-4208-b5db-1d283caf05e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 31 13:13:05.316843 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:05.316805 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:05.680547 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:05.680355 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx"] Oct 31 13:13:05.686464 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:05.686433 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:13:05.690511 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:05.690477 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx"] Oct 31 13:13:05.695641 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:05.695610 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-54a16b-openshift-default-dockercfg-rtrgd\"" Oct 31 13:13:05.830039 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:05.828163 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:05.860245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:05.857149 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d7ee65-7958-458c-a202-693bd09893c7-istio-envoy\") pod \"e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx\" (UID: \"25d7ee65-7958-458c-a202-693bd09893c7\") " pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:13:05.860245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:05.857309 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d7ee65-7958-458c-a202-693bd09893c7-istio-token\") pod \"e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx\" (UID: \"25d7ee65-7958-458c-a202-693bd09893c7\") " pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:13:05.860245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:05.857358 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d7ee65-7958-458c-a202-693bd09893c7-istiod-ca-cert\") pod \"e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx\" (UID: \"25d7ee65-7958-458c-a202-693bd09893c7\") " pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:13:05.860245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:05.857401 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d7ee65-7958-458c-a202-693bd09893c7-workload-socket\") pod \"e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx\" (UID: \"25d7ee65-7958-458c-a202-693bd09893c7\") " pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:13:05.860245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:05.857473 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d7ee65-7958-458c-a202-693bd09893c7-workload-certs\") pod \"e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx\" (UID: \"25d7ee65-7958-458c-a202-693bd09893c7\") " pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:13:05.860245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:05.857553 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq7b8\" (UniqueName: \"kubernetes.io/projected/25d7ee65-7958-458c-a202-693bd09893c7-kube-api-access-gq7b8\") pod \"e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx\" (UID: \"25d7ee65-7958-458c-a202-693bd09893c7\") " pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:13:05.860245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:05.857594 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d7ee65-7958-458c-a202-693bd09893c7-istio-data\") pod \"e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx\" (UID: \"25d7ee65-7958-458c-a202-693bd09893c7\") " pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:13:05.860245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:05.857657 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d7ee65-7958-458c-a202-693bd09893c7-credential-socket\") pod \"e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx\" (UID: \"25d7ee65-7958-458c-a202-693bd09893c7\") " pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:13:05.860245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:05.857744 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d7ee65-7958-458c-a202-693bd09893c7-istio-podinfo\") pod \"e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx\" (UID: \"25d7ee65-7958-458c-a202-693bd09893c7\") " pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:13:05.988244 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:05.988210 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d7ee65-7958-458c-a202-693bd09893c7-istio-token\") pod \"e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx\" (UID: \"25d7ee65-7958-458c-a202-693bd09893c7\") " pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:13:06.020573 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:05.992095 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d7ee65-7958-458c-a202-693bd09893c7-istiod-ca-cert\") pod \"e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx\" (UID: \"25d7ee65-7958-458c-a202-693bd09893c7\") " pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:13:06.021050 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:06.021016 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d7ee65-7958-458c-a202-693bd09893c7-workload-socket\") pod \"e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx\" (UID: \"25d7ee65-7958-458c-a202-693bd09893c7\") " pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:13:06.021410 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:06.021274 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d7ee65-7958-458c-a202-693bd09893c7-workload-certs\") pod \"e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx\" (UID: \"25d7ee65-7958-458c-a202-693bd09893c7\") " pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:13:06.021634 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:06.021613 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq7b8\" (UniqueName: \"kubernetes.io/projected/25d7ee65-7958-458c-a202-693bd09893c7-kube-api-access-gq7b8\") pod \"e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx\" (UID: \"25d7ee65-7958-458c-a202-693bd09893c7\") " pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:13:06.021767 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:06.021753 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d7ee65-7958-458c-a202-693bd09893c7-istio-data\") pod \"e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx\" (UID: \"25d7ee65-7958-458c-a202-693bd09893c7\") " pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:13:06.021907 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:06.021893 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d7ee65-7958-458c-a202-693bd09893c7-credential-socket\") pod \"e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx\" (UID: \"25d7ee65-7958-458c-a202-693bd09893c7\") " pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:13:06.032007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:06.022090 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d7ee65-7958-458c-a202-693bd09893c7-istio-podinfo\") pod \"e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx\" (UID: \"25d7ee65-7958-458c-a202-693bd09893c7\") " pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:13:06.032306 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:06.032279 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d7ee65-7958-458c-a202-693bd09893c7-istio-envoy\") pod \"e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx\" (UID: \"25d7ee65-7958-458c-a202-693bd09893c7\") " pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:13:06.051452 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:06.047366 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d7ee65-7958-458c-a202-693bd09893c7-workload-certs\") pod \"e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx\" (UID: \"25d7ee65-7958-458c-a202-693bd09893c7\") " pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:13:06.100008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:06.048384 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d7ee65-7958-458c-a202-693bd09893c7-istiod-ca-cert\") pod \"e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx\" (UID: \"25d7ee65-7958-458c-a202-693bd09893c7\") " pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:13:06.100314 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:06.048629 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d7ee65-7958-458c-a202-693bd09893c7-workload-socket\") pod \"e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx\" (UID: \"25d7ee65-7958-458c-a202-693bd09893c7\") " pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:13:06.100460 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:06.048900 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d7ee65-7958-458c-a202-693bd09893c7-istio-data\") pod \"e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx\" (UID: \"25d7ee65-7958-458c-a202-693bd09893c7\") " pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:13:06.108075 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:06.049543 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d7ee65-7958-458c-a202-693bd09893c7-credential-socket\") pod \"e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx\" (UID: \"25d7ee65-7958-458c-a202-693bd09893c7\") " pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:13:06.125841 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:06.120740 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d7ee65-7958-458c-a202-693bd09893c7-istio-podinfo\") pod \"e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx\" (UID: \"25d7ee65-7958-458c-a202-693bd09893c7\") " pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:13:06.147951 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:06.147802 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:06.186015 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:06.185842 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq7b8\" (UniqueName: \"kubernetes.io/projected/25d7ee65-7958-458c-a202-693bd09893c7-kube-api-access-gq7b8\") pod \"e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx\" (UID: \"25d7ee65-7958-458c-a202-693bd09893c7\") " pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:13:06.233004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:06.227461 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d7ee65-7958-458c-a202-693bd09893c7-istio-envoy\") pod \"e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx\" (UID: \"25d7ee65-7958-458c-a202-693bd09893c7\") " pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:13:06.259057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:06.249154 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d7ee65-7958-458c-a202-693bd09893c7-istio-token\") pod \"e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx\" (UID: \"25d7ee65-7958-458c-a202-693bd09893c7\") " pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:13:06.259057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:06.249452 6841 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 31 13:13:06.259057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:06.249515 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" podUID="24ea07e9-65f1-4208-b5db-1d283caf05e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 31 13:13:06.311106 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:06.311061 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:13:06.635014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:06.634907 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:06.672736 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:06.672691 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:13:06.672849 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:06.672785 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:13:06.672849 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:06.672831 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:13:06.674080 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:06.674057 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx"] Oct 31 13:13:06.965015 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:06.960810 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:07.232799 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:07.230484 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:07.238902 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:07.238867 6841 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 31 13:13:07.239095 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:07.238928 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" podUID="24ea07e9-65f1-4208-b5db-1d283caf05e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 31 13:13:07.492071 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:07.490769 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:07.598857 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:07.598792 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" event={"ID":"25d7ee65-7958-458c-a202-693bd09893c7","Type":"ContainerStarted","Data":"e838473111f333938c845e3e045d0bcac8a456dbcaa3563781125ba6980845cb"} Oct 31 13:13:07.599054 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:07.598862 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" event={"ID":"25d7ee65-7958-458c-a202-693bd09893c7","Type":"ContainerStarted","Data":"60d9cbb0fa30f67d65613da242db24962885c601581160a4742dfb044160f073"} Oct 31 13:13:07.628261 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:07.628205 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" podStartSLOduration=2.628186341 podStartE2EDuration="2.628186341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:07.62403112 +0000 UTC m=+2434.107154022" watchObservedRunningTime="2026-10-31 13:13:07.628186341 +0000 UTC m=+2434.111309242" Oct 31 13:13:08.175119 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:08.174874 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:08.233870 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:08.233838 6841 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 31 13:13:08.234283 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:08.233891 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" podUID="24ea07e9-65f1-4208-b5db-1d283caf05e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 31 13:13:08.311354 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:08.311314 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:13:08.317575 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:08.317549 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:13:08.606600 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:08.606570 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:13:08.607583 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:08.607558 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:13:09.108002 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:09.107652 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:09.234073 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:09.234040 6841 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 31 13:13:09.234451 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:09.234094 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" podUID="24ea07e9-65f1-4208-b5db-1d283caf05e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 31 13:13:09.581012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:09.580407 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw"] Oct 31 13:13:09.760009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:09.756679 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:10.111861 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:10.111830 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:10.240007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:10.237481 6841 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 31 13:13:10.240007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:10.237582 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" podUID="24ea07e9-65f1-4208-b5db-1d283caf05e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 31 13:13:10.355041 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:10.355000 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:11.238008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.236099 6841 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 31 13:13:11.238008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.236164 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" podUID="24ea07e9-65f1-4208-b5db-1d283caf05e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 31 13:13:11.346358 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.346327 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:11.614523 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.614445 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt"] Oct 31 13:13:11.622702 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.622670 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:13:11.634303 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.634261 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt"] Oct 31 13:13:11.646268 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.646241 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-5dc570-openshift-default-dockercfg-dvsb2\"" Oct 31 13:13:11.655274 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.655233 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:11.706845 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.706809 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de4271dd-8fc1-4768-8dd7-2437ee497f9b-istio-data\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt\" (UID: \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:13:11.707120 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.706876 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4271dd-8fc1-4768-8dd7-2437ee497f9b-credential-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt\" (UID: \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:13:11.707120 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.706935 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4271dd-8fc1-4768-8dd7-2437ee497f9b-workload-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt\" (UID: \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:13:11.707302 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.707262 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de4271dd-8fc1-4768-8dd7-2437ee497f9b-istiod-ca-cert\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt\" (UID: \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:13:11.707375 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.707323 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de4271dd-8fc1-4768-8dd7-2437ee497f9b-istio-envoy\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt\" (UID: \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:13:11.707433 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.707391 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de4271dd-8fc1-4768-8dd7-2437ee497f9b-workload-certs\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt\" (UID: \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:13:11.707496 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.707479 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de4271dd-8fc1-4768-8dd7-2437ee497f9b-istio-podinfo\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt\" (UID: \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:13:11.707551 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.707517 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de4271dd-8fc1-4768-8dd7-2437ee497f9b-istio-token\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt\" (UID: \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:13:11.707603 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.707544 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zzh8\" (UniqueName: \"kubernetes.io/projected/de4271dd-8fc1-4768-8dd7-2437ee497f9b-kube-api-access-7zzh8\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt\" (UID: \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:13:11.810112 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.810064 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de4271dd-8fc1-4768-8dd7-2437ee497f9b-istio-envoy\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt\" (UID: \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:13:11.810394 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.810372 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de4271dd-8fc1-4768-8dd7-2437ee497f9b-workload-certs\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt\" (UID: \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:13:11.810593 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.810573 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de4271dd-8fc1-4768-8dd7-2437ee497f9b-istio-podinfo\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt\" (UID: \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:13:11.810732 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.810711 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de4271dd-8fc1-4768-8dd7-2437ee497f9b-istio-token\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt\" (UID: \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:13:11.810858 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.810842 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zzh8\" (UniqueName: \"kubernetes.io/projected/de4271dd-8fc1-4768-8dd7-2437ee497f9b-kube-api-access-7zzh8\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt\" (UID: \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:13:11.811031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.811014 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de4271dd-8fc1-4768-8dd7-2437ee497f9b-istio-data\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt\" (UID: \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:13:11.811197 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.811179 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4271dd-8fc1-4768-8dd7-2437ee497f9b-credential-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt\" (UID: \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:13:11.811331 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.811314 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4271dd-8fc1-4768-8dd7-2437ee497f9b-workload-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt\" (UID: \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:13:11.829008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.811671 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de4271dd-8fc1-4768-8dd7-2437ee497f9b-istiod-ca-cert\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt\" (UID: \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:13:11.829366 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.821657 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de4271dd-8fc1-4768-8dd7-2437ee497f9b-workload-certs\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt\" (UID: \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:13:11.829366 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.821967 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de4271dd-8fc1-4768-8dd7-2437ee497f9b-istio-data\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt\" (UID: \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:13:11.829366 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.822293 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de4271dd-8fc1-4768-8dd7-2437ee497f9b-istio-envoy\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt\" (UID: \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:13:11.829366 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.822545 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4271dd-8fc1-4768-8dd7-2437ee497f9b-credential-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt\" (UID: \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:13:11.829366 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.822786 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4271dd-8fc1-4768-8dd7-2437ee497f9b-workload-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt\" (UID: \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:13:11.829366 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.824729 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de4271dd-8fc1-4768-8dd7-2437ee497f9b-istiod-ca-cert\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt\" (UID: \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:13:11.856014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.836631 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de4271dd-8fc1-4768-8dd7-2437ee497f9b-istio-podinfo\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt\" (UID: \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:13:11.858417 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.856777 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de4271dd-8fc1-4768-8dd7-2437ee497f9b-istio-token\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt\" (UID: \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:13:11.858417 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.857586 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zzh8\" (UniqueName: \"kubernetes.io/projected/de4271dd-8fc1-4768-8dd7-2437ee497f9b-kube-api-access-7zzh8\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt\" (UID: \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:13:11.927516 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.927194 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:11.959318 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:11.959284 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:13:12.299012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:12.297822 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:12.603432 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:12.603354 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" Oct 31 13:13:12.604024 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:12.603950 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" Oct 31 13:13:12.605151 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:12.605129 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" Oct 31 13:13:12.624420 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:12.624377 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt"] Oct 31 13:13:12.651964 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:12.651912 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:13:12.652140 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:12.652072 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:13:12.652140 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:12.652124 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:13:12.655318 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:12.655284 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" event={"ID":"de4271dd-8fc1-4768-8dd7-2437ee497f9b","Type":"ContainerStarted","Data":"ea974dc1d8632beb4e73b563293641ead1a43446acfd6b79b366c37047127c2b"} Oct 31 13:13:12.716585 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:12.716550 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:13.075029 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:13.074992 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:13.856660 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:13.856618 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" event={"ID":"de4271dd-8fc1-4768-8dd7-2437ee497f9b","Type":"ContainerStarted","Data":"7502228c566b3d881de24554dfa7716749434c2266896799063260e4b191daf1"} Oct 31 13:13:13.896017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:13.888445 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" podStartSLOduration=2.888423522 podStartE2EDuration="2.888423522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:13.853760838 +0000 UTC m=+2440.336883740" watchObservedRunningTime="2026-10-31 13:13:13.888423522 +0000 UTC m=+2440.371546424" Oct 31 13:13:13.986263 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:13.967596 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:13:14.006699 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:14.006667 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:13:14.151902 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:14.151808 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:14.827007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:14.816165 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:13:14.842006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:14.828267 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:13:15.397002 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:15.390127 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:16.245108 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:16.245021 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:16.542484 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:16.542197 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:16.870808 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:16.870720 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:17.238216 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:17.238187 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:17.533061 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:17.529902 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:17.803063 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:17.794430 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:18.083094 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:18.082643 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:18.352074 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:18.349621 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:18.628023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:18.627921 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh" podUID="4a2e95ce-2103-46b2-867b-de098db55e96" containerName="main" containerID="cri-o://f6cd0a5ca938530a3e0f5435a0fbaec5fe8e305650a907f2b6a1f3f3f0c92b0f" gracePeriod=45 Oct 31 13:13:18.769216 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:18.769173 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:18.866914 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:18.866880 6841 generic.go:356] "Generic (PLEG): container finished" podID="4a2e95ce-2103-46b2-867b-de098db55e96" containerID="f6cd0a5ca938530a3e0f5435a0fbaec5fe8e305650a907f2b6a1f3f3f0c92b0f" exitCode=0 Oct 31 13:13:18.867336 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:18.866936 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh" event={"ID":"4a2e95ce-2103-46b2-867b-de098db55e96","Type":"ContainerDied","Data":"f6cd0a5ca938530a3e0f5435a0fbaec5fe8e305650a907f2b6a1f3f3f0c92b0f"} Oct 31 13:13:18.936293 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:18.936270 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh" Oct 31 13:13:19.093399 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.093364 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a2e95ce-2103-46b2-867b-de098db55e96-kube-api-access-fm8pv\" (UniqueName: \"kubernetes.io/projected/4a2e95ce-2103-46b2-867b-de098db55e96-kube-api-access-fm8pv\") pod \"4a2e95ce-2103-46b2-867b-de098db55e96\" (UID: \"4a2e95ce-2103-46b2-867b-de098db55e96\") " Oct 31 13:13:19.093583 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.093407 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2e95ce-2103-46b2-867b-de098db55e96-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a2e95ce-2103-46b2-867b-de098db55e96-dshm\") pod \"4a2e95ce-2103-46b2-867b-de098db55e96\" (UID: \"4a2e95ce-2103-46b2-867b-de098db55e96\") " Oct 31 13:13:19.093583 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.093540 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a2e95ce-2103-46b2-867b-de098db55e96-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a2e95ce-2103-46b2-867b-de098db55e96-tls-certs\") pod \"4a2e95ce-2103-46b2-867b-de098db55e96\" (UID: \"4a2e95ce-2103-46b2-867b-de098db55e96\") " Oct 31 13:13:19.093864 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.093841 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2e95ce-2103-46b2-867b-de098db55e96-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a2e95ce-2103-46b2-867b-de098db55e96-model-cache\") pod \"4a2e95ce-2103-46b2-867b-de098db55e96\" (UID: \"4a2e95ce-2103-46b2-867b-de098db55e96\") " Oct 31 13:13:19.094044 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.093908 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2e95ce-2103-46b2-867b-de098db55e96-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a2e95ce-2103-46b2-867b-de098db55e96-tmp-dir\") pod \"4a2e95ce-2103-46b2-867b-de098db55e96\" (UID: \"4a2e95ce-2103-46b2-867b-de098db55e96\") " Oct 31 13:13:19.094210 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.094042 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2e95ce-2103-46b2-867b-de098db55e96-home\" (UniqueName: \"kubernetes.io/empty-dir/4a2e95ce-2103-46b2-867b-de098db55e96-home\") pod \"4a2e95ce-2103-46b2-867b-de098db55e96\" (UID: \"4a2e95ce-2103-46b2-867b-de098db55e96\") " Oct 31 13:13:19.094210 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.094165 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2e95ce-2103-46b2-867b-de098db55e96-model-cache" pod "4a2e95ce-2103-46b2-867b-de098db55e96" (UID: "4a2e95ce-2103-46b2-867b-de098db55e96"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:19.094324 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.094295 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2e95ce-2103-46b2-867b-de098db55e96-tmp-dir" pod "4a2e95ce-2103-46b2-867b-de098db55e96" (UID: "4a2e95ce-2103-46b2-867b-de098db55e96"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:19.094452 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.094429 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2e95ce-2103-46b2-867b-de098db55e96-home" pod "4a2e95ce-2103-46b2-867b-de098db55e96" (UID: "4a2e95ce-2103-46b2-867b-de098db55e96"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:19.095157 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.095099 6841 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a2e95ce-2103-46b2-867b-de098db55e96-model-cache\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:19.095157 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.095136 6841 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a2e95ce-2103-46b2-867b-de098db55e96-tmp-dir\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:19.095157 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.095152 6841 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a2e95ce-2103-46b2-867b-de098db55e96-home\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:19.096616 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.096593 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a2e95ce-2103-46b2-867b-de098db55e96-tls-certs" pod "4a2e95ce-2103-46b2-867b-de098db55e96" (UID: "4a2e95ce-2103-46b2-867b-de098db55e96"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:13:19.096784 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.096761 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a2e95ce-2103-46b2-867b-de098db55e96-kube-api-access-fm8pv" pod "4a2e95ce-2103-46b2-867b-de098db55e96" (UID: "4a2e95ce-2103-46b2-867b-de098db55e96"). InnerVolumeSpecName "kube-api-access-fm8pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:13:19.096943 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.096924 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2e95ce-2103-46b2-867b-de098db55e96-dshm" pod "4a2e95ce-2103-46b2-867b-de098db55e96" (UID: "4a2e95ce-2103-46b2-867b-de098db55e96"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:19.110944 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.110913 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:19.113421 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.112236 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj"] Oct 31 13:13:19.113541 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.113521 6841 cpu_manager.go:414] "containerMap: removing container" podUID="4a2e95ce-2103-46b2-867b-de098db55e96" containerName="main" Oct 31 13:13:19.113604 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.113543 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a2e95ce-2103-46b2-867b-de098db55e96" containerName="main" Oct 31 13:13:19.113764 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.113747 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a2e95ce-2103-46b2-867b-de098db55e96" containerName="main" Oct 31 13:13:19.118645 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.118622 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:13:19.131491 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.131467 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-29e3a0-openshift-default-dockercfg-4zmv8\"" Oct 31 13:13:19.147596 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.147504 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj"] Oct 31 13:13:19.197710 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.196853 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm8pv\" (UniqueName: \"kubernetes.io/projected/4a2e95ce-2103-46b2-867b-de098db55e96-kube-api-access-fm8pv\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:19.197710 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.196899 6841 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a2e95ce-2103-46b2-867b-de098db55e96-dshm\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:19.198056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.198035 6841 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a2e95ce-2103-46b2-867b-de098db55e96-tls-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:19.318031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.312436 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8494c61-00b4-41e6-b8ed-f4c2081f8944-istio-envoy\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj\" (UID: \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:13:19.318031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.312539 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8494c61-00b4-41e6-b8ed-f4c2081f8944-istio-token\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj\" (UID: \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:13:19.318031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.312600 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8494c61-00b4-41e6-b8ed-f4c2081f8944-credential-socket\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj\" (UID: \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:13:19.318031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.312628 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8494c61-00b4-41e6-b8ed-f4c2081f8944-istio-data\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj\" (UID: \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:13:19.318031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.312659 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8494c61-00b4-41e6-b8ed-f4c2081f8944-istio-podinfo\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj\" (UID: \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:13:19.318031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.312688 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8494c61-00b4-41e6-b8ed-f4c2081f8944-istiod-ca-cert\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj\" (UID: \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:13:19.318031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.312790 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpt7t\" (UniqueName: \"kubernetes.io/projected/a8494c61-00b4-41e6-b8ed-f4c2081f8944-kube-api-access-zpt7t\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj\" (UID: \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:13:19.318031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.312936 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8494c61-00b4-41e6-b8ed-f4c2081f8944-workload-socket\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj\" (UID: \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:13:19.318031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.313023 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8494c61-00b4-41e6-b8ed-f4c2081f8944-workload-certs\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj\" (UID: \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:13:19.372070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.364611 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:19.461120 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.443775 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpt7t\" (UniqueName: \"kubernetes.io/projected/a8494c61-00b4-41e6-b8ed-f4c2081f8944-kube-api-access-zpt7t\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj\" (UID: \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:13:19.461120 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.443932 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8494c61-00b4-41e6-b8ed-f4c2081f8944-workload-socket\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj\" (UID: \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:13:19.461120 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.444016 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8494c61-00b4-41e6-b8ed-f4c2081f8944-workload-certs\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj\" (UID: \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:13:19.461120 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.444185 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8494c61-00b4-41e6-b8ed-f4c2081f8944-istio-envoy\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj\" (UID: \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:13:19.461120 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.444262 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8494c61-00b4-41e6-b8ed-f4c2081f8944-istio-token\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj\" (UID: \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:13:19.461120 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.444322 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8494c61-00b4-41e6-b8ed-f4c2081f8944-credential-socket\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj\" (UID: \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:13:19.461120 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.444351 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8494c61-00b4-41e6-b8ed-f4c2081f8944-istio-data\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj\" (UID: \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:13:19.461120 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.444384 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8494c61-00b4-41e6-b8ed-f4c2081f8944-istio-podinfo\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj\" (UID: \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:13:19.461120 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.444415 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8494c61-00b4-41e6-b8ed-f4c2081f8944-istiod-ca-cert\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj\" (UID: \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:13:19.461120 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.455548 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8494c61-00b4-41e6-b8ed-f4c2081f8944-workload-socket\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj\" (UID: \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:13:19.461120 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.456190 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8494c61-00b4-41e6-b8ed-f4c2081f8944-workload-certs\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj\" (UID: \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:13:19.461120 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.456484 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8494c61-00b4-41e6-b8ed-f4c2081f8944-credential-socket\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj\" (UID: \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:13:19.461120 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.457056 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8494c61-00b4-41e6-b8ed-f4c2081f8944-istio-data\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj\" (UID: \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:13:19.461120 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.457766 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8494c61-00b4-41e6-b8ed-f4c2081f8944-istiod-ca-cert\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj\" (UID: \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:13:19.512049 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.502102 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8494c61-00b4-41e6-b8ed-f4c2081f8944-istio-token\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj\" (UID: \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:13:19.518702 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.518668 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8494c61-00b4-41e6-b8ed-f4c2081f8944-istio-envoy\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj\" (UID: \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:13:19.519258 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.519226 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8494c61-00b4-41e6-b8ed-f4c2081f8944-istio-podinfo\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj\" (UID: \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:13:19.528488 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.526972 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpt7t\" (UniqueName: \"kubernetes.io/projected/a8494c61-00b4-41e6-b8ed-f4c2081f8944-kube-api-access-zpt7t\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj\" (UID: \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:13:19.757659 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.757567 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:13:19.819886 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.819838 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh" podUID="4a2e95ce-2103-46b2-867b-de098db55e96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.173:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:13:19.884239 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.884201 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh" event={"ID":"4a2e95ce-2103-46b2-867b-de098db55e96","Type":"ContainerDied","Data":"aa5a4107323925b1ab5f227c6aff1de12cd98c7cf4745fc40634f24f21abbe84"} Oct 31 13:13:19.884721 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.884261 6841 scope.go:122] "RemoveContainer" containerID="f6cd0a5ca938530a3e0f5435a0fbaec5fe8e305650a907f2b6a1f3f3f0c92b0f" Oct 31 13:13:19.884721 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.884398 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh" Oct 31 13:13:19.989006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.988750 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh"] Oct 31 13:13:19.989006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:19.988815 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-5c24e1/auth-test-model-5c24e1-kserve-85b876497c-2nwbh"] Oct 31 13:13:20.035682 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:20.035596 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:20.037680 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:13:20.037641 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:13:20.054681 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:13:20.054655 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8494c61_00b4_41e6_b8ed_f4c2081f8944.slice/crio-594736f1574700a57eed04c45159054a679e3d3f8e43f502ab37c31c56853708 WatchSource:0}: Error finding container 594736f1574700a57eed04c45159054a679e3d3f8e43f502ab37c31c56853708: Status 404 returned error can't find the container with id 594736f1574700a57eed04c45159054a679e3d3f8e43f502ab37c31c56853708 Oct 31 13:13:20.055021 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:20.054971 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj"] Oct 31 13:13:20.057804 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:20.057769 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:13:20.057907 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:20.057836 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:13:20.057907 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:20.057864 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:13:20.442061 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:20.437754 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:20.894786 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:20.894752 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" event={"ID":"a8494c61-00b4-41e6-b8ed-f4c2081f8944","Type":"ContainerStarted","Data":"3019597717a794c14594995799ffde8fde8055320e4d2d73d5c079cb751c1deb"} Oct 31 13:13:20.895243 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:20.894829 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" event={"ID":"a8494c61-00b4-41e6-b8ed-f4c2081f8944","Type":"ContainerStarted","Data":"594736f1574700a57eed04c45159054a679e3d3f8e43f502ab37c31c56853708"} Oct 31 13:13:20.928843 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:20.928775 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" podStartSLOduration=1.928756419 podStartE2EDuration="1.928756419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:20.924261511 +0000 UTC m=+2447.407384413" watchObservedRunningTime="2026-10-31 13:13:20.928756419 +0000 UTC m=+2447.411879321" Oct 31 13:13:21.456006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:21.447173 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:21.514717 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:21.514685 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt"] Oct 31 13:13:21.514950 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:21.514924 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" podUID="8c763c4c-2410-4735-9e8d-98d8ec38142c" containerName="istio-proxy" containerID="cri-o://e1b923b781d65b19a56188d0fcd7efbfe3583d5d54eb0096a2caabb79c0e1bf9" gracePeriod=30 Oct 31 13:13:21.519187 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:21.519157 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z"] Oct 31 13:13:21.519363 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:21.519343 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" podUID="21566728-c2cd-42a6-9ffe-7aaf14243ee8" containerName="istio-proxy" containerID="cri-o://dc8c120a3032b01ecd3fcb1833a1ce143eeed58653cd4ff5050176a9f091c59c" gracePeriod=30 Oct 31 13:13:21.643998 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:21.643958 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a2e95ce-2103-46b2-867b-de098db55e96" path="/var/lib/kubelet/pods/4a2e95ce-2103-46b2-867b-de098db55e96/volumes" Oct 31 13:13:21.757830 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:21.757749 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:13:21.764287 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:21.764261 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:13:21.903561 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:21.903532 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:13:21.904330 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:21.904311 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:13:22.646924 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:22.646880 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:22.885055 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:22.884528 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:23.422397 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:23.422358 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:23.654920 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:23.654882 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:23.930012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:23.929871 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:23.995127 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:23.995074 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh"] Oct 31 13:13:23.995407 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:23.995374 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" podUID="d6fdf066-cd83-408d-9c1c-6c97f9189f33" containerName="istio-proxy" containerID="cri-o://a978080ec4f8e1250f3f8f4caa420a97949bd4955f0d1632ea4208f690ccc40c" gracePeriod=30 Oct 31 13:13:24.001452 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:24.001418 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz"] Oct 31 13:13:24.001720 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:24.001677 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" podUID="88130f73-b9a3-4bdb-aea9-8557803120cd" containerName="istio-proxy" containerID="cri-o://6f64a098a6c9c98c5415c07a9dd0056fa668064189f876792c8718448e12c60b" gracePeriod=30 Oct 31 13:13:24.187629 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:24.187539 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:24.597008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:24.594540 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:24.609616 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:24.609444 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw" podUID="4a034a05-49d4-4c08-a9b5-5299af987eb1" containerName="main" containerID="cri-o://6c4711618e06d4b00edba9e966c3368f00bbe14d5f5db75bb9c06435eb09e6f2" gracePeriod=45 Oct 31 13:13:24.902476 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:24.902445 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw" Oct 31 13:13:24.982559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:24.982523 6841 generic.go:356] "Generic (PLEG): container finished" podID="4a034a05-49d4-4c08-a9b5-5299af987eb1" containerID="6c4711618e06d4b00edba9e966c3368f00bbe14d5f5db75bb9c06435eb09e6f2" exitCode=0 Oct 31 13:13:24.982559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:24.982576 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw" Oct 31 13:13:24.982846 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:24.982617 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw" event={"ID":"4a034a05-49d4-4c08-a9b5-5299af987eb1","Type":"ContainerDied","Data":"6c4711618e06d4b00edba9e966c3368f00bbe14d5f5db75bb9c06435eb09e6f2"} Oct 31 13:13:24.982846 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:24.982651 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw" event={"ID":"4a034a05-49d4-4c08-a9b5-5299af987eb1","Type":"ContainerDied","Data":"162cf2954450b9f9bc606e983160875117ae4bc374d8e562cb2b7c663a86bd15"} Oct 31 13:13:24.982846 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:24.982673 6841 scope.go:122] "RemoveContainer" containerID="6c4711618e06d4b00edba9e966c3368f00bbe14d5f5db75bb9c06435eb09e6f2" Oct 31 13:13:24.997005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:24.996969 6841 scope.go:122] "RemoveContainer" containerID="6c4711618e06d4b00edba9e966c3368f00bbe14d5f5db75bb9c06435eb09e6f2" Oct 31 13:13:24.997334 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:13:24.997313 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c4711618e06d4b00edba9e966c3368f00bbe14d5f5db75bb9c06435eb09e6f2\": container with ID starting with 6c4711618e06d4b00edba9e966c3368f00bbe14d5f5db75bb9c06435eb09e6f2 not found: ID does not exist" containerID="6c4711618e06d4b00edba9e966c3368f00bbe14d5f5db75bb9c06435eb09e6f2" Oct 31 13:13:24.997444 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:24.997341 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c4711618e06d4b00edba9e966c3368f00bbe14d5f5db75bb9c06435eb09e6f2"} err="failed to get container status \"6c4711618e06d4b00edba9e966c3368f00bbe14d5f5db75bb9c06435eb09e6f2\": rpc error: code = NotFound desc = could not find container \"6c4711618e06d4b00edba9e966c3368f00bbe14d5f5db75bb9c06435eb09e6f2\": container with ID starting with 6c4711618e06d4b00edba9e966c3368f00bbe14d5f5db75bb9c06435eb09e6f2 not found: ID does not exist" Oct 31 13:13:25.032847 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:25.032821 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a034a05-49d4-4c08-a9b5-5299af987eb1-kube-api-access-274hp\" (UniqueName: \"kubernetes.io/projected/4a034a05-49d4-4c08-a9b5-5299af987eb1-kube-api-access-274hp\") pod \"4a034a05-49d4-4c08-a9b5-5299af987eb1\" (UID: \"4a034a05-49d4-4c08-a9b5-5299af987eb1\") " Oct 31 13:13:25.033027 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:25.032862 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a034a05-49d4-4c08-a9b5-5299af987eb1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a034a05-49d4-4c08-a9b5-5299af987eb1-model-cache\") pod \"4a034a05-49d4-4c08-a9b5-5299af987eb1\" (UID: \"4a034a05-49d4-4c08-a9b5-5299af987eb1\") " Oct 31 13:13:25.033027 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:25.032901 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a034a05-49d4-4c08-a9b5-5299af987eb1-home\" (UniqueName: \"kubernetes.io/empty-dir/4a034a05-49d4-4c08-a9b5-5299af987eb1-home\") pod \"4a034a05-49d4-4c08-a9b5-5299af987eb1\" (UID: \"4a034a05-49d4-4c08-a9b5-5299af987eb1\") " Oct 31 13:13:25.033131 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:25.033059 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a034a05-49d4-4c08-a9b5-5299af987eb1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a034a05-49d4-4c08-a9b5-5299af987eb1-tmp-dir\") pod \"4a034a05-49d4-4c08-a9b5-5299af987eb1\" (UID: \"4a034a05-49d4-4c08-a9b5-5299af987eb1\") " Oct 31 13:13:25.033242 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:25.033208 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a034a05-49d4-4c08-a9b5-5299af987eb1-home" pod "4a034a05-49d4-4c08-a9b5-5299af987eb1" (UID: "4a034a05-49d4-4c08-a9b5-5299af987eb1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:25.033342 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:25.033217 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a034a05-49d4-4c08-a9b5-5299af987eb1-model-cache" pod "4a034a05-49d4-4c08-a9b5-5299af987eb1" (UID: "4a034a05-49d4-4c08-a9b5-5299af987eb1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:25.033342 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:25.033264 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a034a05-49d4-4c08-a9b5-5299af987eb1-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a034a05-49d4-4c08-a9b5-5299af987eb1-tls-certs\") pod \"4a034a05-49d4-4c08-a9b5-5299af987eb1\" (UID: \"4a034a05-49d4-4c08-a9b5-5299af987eb1\") " Oct 31 13:13:25.033342 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:25.033280 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a034a05-49d4-4c08-a9b5-5299af987eb1-tmp-dir" pod "4a034a05-49d4-4c08-a9b5-5299af987eb1" (UID: "4a034a05-49d4-4c08-a9b5-5299af987eb1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:25.033342 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:25.033294 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a034a05-49d4-4c08-a9b5-5299af987eb1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a034a05-49d4-4c08-a9b5-5299af987eb1-dshm\") pod \"4a034a05-49d4-4c08-a9b5-5299af987eb1\" (UID: \"4a034a05-49d4-4c08-a9b5-5299af987eb1\") " Oct 31 13:13:25.033926 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:25.033904 6841 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a034a05-49d4-4c08-a9b5-5299af987eb1-model-cache\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:25.033926 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:25.033926 6841 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a034a05-49d4-4c08-a9b5-5299af987eb1-home\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:25.034150 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:25.033938 6841 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a034a05-49d4-4c08-a9b5-5299af987eb1-tmp-dir\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:25.035203 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:25.035178 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a034a05-49d4-4c08-a9b5-5299af987eb1-kube-api-access-274hp" pod "4a034a05-49d4-4c08-a9b5-5299af987eb1" (UID: "4a034a05-49d4-4c08-a9b5-5299af987eb1"). InnerVolumeSpecName "kube-api-access-274hp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:13:25.035286 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:25.035221 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a034a05-49d4-4c08-a9b5-5299af987eb1-dshm" pod "4a034a05-49d4-4c08-a9b5-5299af987eb1" (UID: "4a034a05-49d4-4c08-a9b5-5299af987eb1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:25.035479 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:25.035457 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a034a05-49d4-4c08-a9b5-5299af987eb1-tls-certs" pod "4a034a05-49d4-4c08-a9b5-5299af987eb1" (UID: "4a034a05-49d4-4c08-a9b5-5299af987eb1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:13:25.135365 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:25.135289 6841 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a034a05-49d4-4c08-a9b5-5299af987eb1-tls-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:25.135365 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:25.135315 6841 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a034a05-49d4-4c08-a9b5-5299af987eb1-dshm\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:25.135365 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:25.135327 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-274hp\" (UniqueName: \"kubernetes.io/projected/4a034a05-49d4-4c08-a9b5-5299af987eb1-kube-api-access-274hp\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:25.315428 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:25.315396 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw"] Oct 31 13:13:25.328742 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:25.328711 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw"] Oct 31 13:13:25.362385 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:25.362337 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:25.678113 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:25.660639 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:25.844277 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:25.839290 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a034a05-49d4-4c08-a9b5-5299af987eb1" path="/var/lib/kubelet/pods/4a034a05-49d4-4c08-a9b5-5299af987eb1/volumes" Oct 31 13:13:25.844669 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:25.844632 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-569075/auth-test-model-569075-kserve-7d5f64445c-tnphw" podUID="4a034a05-49d4-4c08-a9b5-5299af987eb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.174:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:13:25.898968 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:25.898932 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:26.626706 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.626393 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:26.828209 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.828170 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" Oct 31 13:13:26.851421 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.851399 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" Oct 31 13:13:26.877138 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.877111 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c763c4c-2410-4735-9e8d-98d8ec38142c-kube-api-access-xb5mg\" (UniqueName: \"kubernetes.io/projected/8c763c4c-2410-4735-9e8d-98d8ec38142c-kube-api-access-xb5mg\") pod \"8c763c4c-2410-4735-9e8d-98d8ec38142c\" (UID: \"8c763c4c-2410-4735-9e8d-98d8ec38142c\") " Oct 31 13:13:26.877290 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.877156 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c763c4c-2410-4735-9e8d-98d8ec38142c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c763c4c-2410-4735-9e8d-98d8ec38142c-istio-data\") pod \"8c763c4c-2410-4735-9e8d-98d8ec38142c\" (UID: \"8c763c4c-2410-4735-9e8d-98d8ec38142c\") " Oct 31 13:13:26.877290 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.877217 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c763c4c-2410-4735-9e8d-98d8ec38142c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c763c4c-2410-4735-9e8d-98d8ec38142c-istiod-ca-cert\") pod \"8c763c4c-2410-4735-9e8d-98d8ec38142c\" (UID: \"8c763c4c-2410-4735-9e8d-98d8ec38142c\") " Oct 31 13:13:26.877290 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.877237 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c763c4c-2410-4735-9e8d-98d8ec38142c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c763c4c-2410-4735-9e8d-98d8ec38142c-workload-socket\") pod \"8c763c4c-2410-4735-9e8d-98d8ec38142c\" (UID: \"8c763c4c-2410-4735-9e8d-98d8ec38142c\") " Oct 31 13:13:26.877501 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.877322 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c763c4c-2410-4735-9e8d-98d8ec38142c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c763c4c-2410-4735-9e8d-98d8ec38142c-istio-envoy\") pod \"8c763c4c-2410-4735-9e8d-98d8ec38142c\" (UID: \"8c763c4c-2410-4735-9e8d-98d8ec38142c\") " Oct 31 13:13:26.877501 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.877360 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c763c4c-2410-4735-9e8d-98d8ec38142c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c763c4c-2410-4735-9e8d-98d8ec38142c-credential-socket\") pod \"8c763c4c-2410-4735-9e8d-98d8ec38142c\" (UID: \"8c763c4c-2410-4735-9e8d-98d8ec38142c\") " Oct 31 13:13:26.877501 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.877418 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c763c4c-2410-4735-9e8d-98d8ec38142c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c763c4c-2410-4735-9e8d-98d8ec38142c-istio-podinfo\") pod \"8c763c4c-2410-4735-9e8d-98d8ec38142c\" (UID: \"8c763c4c-2410-4735-9e8d-98d8ec38142c\") " Oct 31 13:13:26.877736 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.877517 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c763c4c-2410-4735-9e8d-98d8ec38142c-istio-data" pod "8c763c4c-2410-4735-9e8d-98d8ec38142c" (UID: "8c763c4c-2410-4735-9e8d-98d8ec38142c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:26.877736 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.877546 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c763c4c-2410-4735-9e8d-98d8ec38142c-istio-token\" (UniqueName: \"kubernetes.io/projected/8c763c4c-2410-4735-9e8d-98d8ec38142c-istio-token\") pod \"8c763c4c-2410-4735-9e8d-98d8ec38142c\" (UID: \"8c763c4c-2410-4735-9e8d-98d8ec38142c\") " Oct 31 13:13:26.877736 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.877607 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c763c4c-2410-4735-9e8d-98d8ec38142c-istiod-ca-cert" pod "8c763c4c-2410-4735-9e8d-98d8ec38142c" (UID: "8c763c4c-2410-4735-9e8d-98d8ec38142c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:13:26.877736 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.877620 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c763c4c-2410-4735-9e8d-98d8ec38142c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c763c4c-2410-4735-9e8d-98d8ec38142c-workload-certs\") pod \"8c763c4c-2410-4735-9e8d-98d8ec38142c\" (UID: \"8c763c4c-2410-4735-9e8d-98d8ec38142c\") " Oct 31 13:13:26.877736 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.877714 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c763c4c-2410-4735-9e8d-98d8ec38142c-credential-socket" pod "8c763c4c-2410-4735-9e8d-98d8ec38142c" (UID: "8c763c4c-2410-4735-9e8d-98d8ec38142c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:26.878042 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.877775 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c763c4c-2410-4735-9e8d-98d8ec38142c-workload-socket" pod "8c763c4c-2410-4735-9e8d-98d8ec38142c" (UID: "8c763c4c-2410-4735-9e8d-98d8ec38142c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:26.878113 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.878080 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c763c4c-2410-4735-9e8d-98d8ec38142c-workload-certs" pod "8c763c4c-2410-4735-9e8d-98d8ec38142c" (UID: "8c763c4c-2410-4735-9e8d-98d8ec38142c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:26.878471 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.878447 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c763c4c-2410-4735-9e8d-98d8ec38142c-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:26.878568 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.878476 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c763c4c-2410-4735-9e8d-98d8ec38142c-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:26.878568 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.878493 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c763c4c-2410-4735-9e8d-98d8ec38142c-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:26.878568 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.878509 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c763c4c-2410-4735-9e8d-98d8ec38142c-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:26.878568 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.878525 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c763c4c-2410-4735-9e8d-98d8ec38142c-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:26.879703 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.879672 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c763c4c-2410-4735-9e8d-98d8ec38142c-istio-podinfo" pod "8c763c4c-2410-4735-9e8d-98d8ec38142c" (UID: "8c763c4c-2410-4735-9e8d-98d8ec38142c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:13:26.879795 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.879730 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c763c4c-2410-4735-9e8d-98d8ec38142c-istio-token" pod "8c763c4c-2410-4735-9e8d-98d8ec38142c" (UID: "8c763c4c-2410-4735-9e8d-98d8ec38142c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:13:26.879837 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.879814 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c763c4c-2410-4735-9e8d-98d8ec38142c-istio-envoy" pod "8c763c4c-2410-4735-9e8d-98d8ec38142c" (UID: "8c763c4c-2410-4735-9e8d-98d8ec38142c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:26.879883 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.879839 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c763c4c-2410-4735-9e8d-98d8ec38142c-kube-api-access-xb5mg" pod "8c763c4c-2410-4735-9e8d-98d8ec38142c" (UID: "8c763c4c-2410-4735-9e8d-98d8ec38142c"). InnerVolumeSpecName "kube-api-access-xb5mg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:13:26.979292 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.979261 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21566728-c2cd-42a6-9ffe-7aaf14243ee8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21566728-c2cd-42a6-9ffe-7aaf14243ee8-istio-envoy\") pod \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\" (UID: \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\") " Oct 31 13:13:26.979451 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.979323 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21566728-c2cd-42a6-9ffe-7aaf14243ee8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21566728-c2cd-42a6-9ffe-7aaf14243ee8-credential-socket\") pod \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\" (UID: \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\") " Oct 31 13:13:26.979451 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.979393 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21566728-c2cd-42a6-9ffe-7aaf14243ee8-kube-api-access-t696z\" (UniqueName: \"kubernetes.io/projected/21566728-c2cd-42a6-9ffe-7aaf14243ee8-kube-api-access-t696z\") pod \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\" (UID: \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\") " Oct 31 13:13:26.979451 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.979433 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21566728-c2cd-42a6-9ffe-7aaf14243ee8-istio-token\" (UniqueName: \"kubernetes.io/projected/21566728-c2cd-42a6-9ffe-7aaf14243ee8-istio-token\") pod \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\" (UID: \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\") " Oct 31 13:13:26.979647 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.979470 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21566728-c2cd-42a6-9ffe-7aaf14243ee8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21566728-c2cd-42a6-9ffe-7aaf14243ee8-workload-socket\") pod \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\" (UID: \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\") " Oct 31 13:13:26.979710 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.979655 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21566728-c2cd-42a6-9ffe-7aaf14243ee8-credential-socket" pod "21566728-c2cd-42a6-9ffe-7aaf14243ee8" (UID: "21566728-c2cd-42a6-9ffe-7aaf14243ee8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:26.979710 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.979664 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21566728-c2cd-42a6-9ffe-7aaf14243ee8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21566728-c2cd-42a6-9ffe-7aaf14243ee8-workload-certs\") pod \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\" (UID: \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\") " Oct 31 13:13:26.979858 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.979823 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21566728-c2cd-42a6-9ffe-7aaf14243ee8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21566728-c2cd-42a6-9ffe-7aaf14243ee8-istio-podinfo\") pod \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\" (UID: \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\") " Oct 31 13:13:26.979858 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.979829 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21566728-c2cd-42a6-9ffe-7aaf14243ee8-workload-socket" pod "21566728-c2cd-42a6-9ffe-7aaf14243ee8" (UID: "21566728-c2cd-42a6-9ffe-7aaf14243ee8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:26.980018 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.979863 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21566728-c2cd-42a6-9ffe-7aaf14243ee8-workload-certs" pod "21566728-c2cd-42a6-9ffe-7aaf14243ee8" (UID: "21566728-c2cd-42a6-9ffe-7aaf14243ee8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:26.980018 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.979885 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21566728-c2cd-42a6-9ffe-7aaf14243ee8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21566728-c2cd-42a6-9ffe-7aaf14243ee8-istiod-ca-cert\") pod \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\" (UID: \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\") " Oct 31 13:13:26.980146 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.980080 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21566728-c2cd-42a6-9ffe-7aaf14243ee8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21566728-c2cd-42a6-9ffe-7aaf14243ee8-istio-data\") pod \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\" (UID: \"21566728-c2cd-42a6-9ffe-7aaf14243ee8\") " Oct 31 13:13:26.980359 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.980333 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21566728-c2cd-42a6-9ffe-7aaf14243ee8-istio-data" pod "21566728-c2cd-42a6-9ffe-7aaf14243ee8" (UID: "21566728-c2cd-42a6-9ffe-7aaf14243ee8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:26.980359 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.980347 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21566728-c2cd-42a6-9ffe-7aaf14243ee8-istiod-ca-cert" pod "21566728-c2cd-42a6-9ffe-7aaf14243ee8" (UID: "21566728-c2cd-42a6-9ffe-7aaf14243ee8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:13:26.980975 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.980953 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c763c4c-2410-4735-9e8d-98d8ec38142c-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:26.980975 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.980995 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21566728-c2cd-42a6-9ffe-7aaf14243ee8-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:26.981111 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.981011 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c763c4c-2410-4735-9e8d-98d8ec38142c-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:26.981111 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.981026 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21566728-c2cd-42a6-9ffe-7aaf14243ee8-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:26.981111 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.981040 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c763c4c-2410-4735-9e8d-98d8ec38142c-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:26.981111 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.981056 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21566728-c2cd-42a6-9ffe-7aaf14243ee8-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:26.981111 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.981071 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb5mg\" (UniqueName: \"kubernetes.io/projected/8c763c4c-2410-4735-9e8d-98d8ec38142c-kube-api-access-xb5mg\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:26.981111 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.981087 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21566728-c2cd-42a6-9ffe-7aaf14243ee8-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:26.981111 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.981100 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21566728-c2cd-42a6-9ffe-7aaf14243ee8-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:26.982035 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.982010 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21566728-c2cd-42a6-9ffe-7aaf14243ee8-istio-envoy" pod "21566728-c2cd-42a6-9ffe-7aaf14243ee8" (UID: "21566728-c2cd-42a6-9ffe-7aaf14243ee8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:26.982113 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.982034 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21566728-c2cd-42a6-9ffe-7aaf14243ee8-istio-podinfo" pod "21566728-c2cd-42a6-9ffe-7aaf14243ee8" (UID: "21566728-c2cd-42a6-9ffe-7aaf14243ee8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:13:26.982113 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.982035 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21566728-c2cd-42a6-9ffe-7aaf14243ee8-istio-token" pod "21566728-c2cd-42a6-9ffe-7aaf14243ee8" (UID: "21566728-c2cd-42a6-9ffe-7aaf14243ee8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:13:26.982496 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:26.982480 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21566728-c2cd-42a6-9ffe-7aaf14243ee8-kube-api-access-t696z" pod "21566728-c2cd-42a6-9ffe-7aaf14243ee8" (UID: "21566728-c2cd-42a6-9ffe-7aaf14243ee8"). InnerVolumeSpecName "kube-api-access-t696z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:13:27.002280 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:27.002251 6841 generic.go:356] "Generic (PLEG): container finished" podID="21566728-c2cd-42a6-9ffe-7aaf14243ee8" containerID="dc8c120a3032b01ecd3fcb1833a1ce143eeed58653cd4ff5050176a9f091c59c" exitCode=0 Oct 31 13:13:27.002406 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:27.002317 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" Oct 31 13:13:27.002406 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:27.002343 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" event={"ID":"21566728-c2cd-42a6-9ffe-7aaf14243ee8","Type":"ContainerDied","Data":"dc8c120a3032b01ecd3fcb1833a1ce143eeed58653cd4ff5050176a9f091c59c"} Oct 31 13:13:27.002406 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:27.002398 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z" event={"ID":"21566728-c2cd-42a6-9ffe-7aaf14243ee8","Type":"ContainerDied","Data":"d2b462983ce8adc9d9f19448a1308963d0f672996f0123f828afc42b041d859d"} Oct 31 13:13:27.002584 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:27.002423 6841 scope.go:122] "RemoveContainer" containerID="dc8c120a3032b01ecd3fcb1833a1ce143eeed58653cd4ff5050176a9f091c59c" Oct 31 13:13:27.004072 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:27.004046 6841 generic.go:356] "Generic (PLEG): container finished" podID="8c763c4c-2410-4735-9e8d-98d8ec38142c" containerID="e1b923b781d65b19a56188d0fcd7efbfe3583d5d54eb0096a2caabb79c0e1bf9" exitCode=0 Oct 31 13:13:27.004178 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:27.004142 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" Oct 31 13:13:27.004245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:27.004173 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" event={"ID":"8c763c4c-2410-4735-9e8d-98d8ec38142c","Type":"ContainerDied","Data":"e1b923b781d65b19a56188d0fcd7efbfe3583d5d54eb0096a2caabb79c0e1bf9"} Oct 31 13:13:27.004245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:27.004221 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt" event={"ID":"8c763c4c-2410-4735-9e8d-98d8ec38142c","Type":"ContainerDied","Data":"fcea4e10fc085ddeff7218ad736ca3e2b7846ee811b37ee2a112e103014da32e"} Oct 31 13:13:27.048306 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:27.039583 6841 scope.go:122] "RemoveContainer" containerID="dc8c120a3032b01ecd3fcb1833a1ce143eeed58653cd4ff5050176a9f091c59c" Oct 31 13:13:27.071007 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:13:27.063976 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc8c120a3032b01ecd3fcb1833a1ce143eeed58653cd4ff5050176a9f091c59c\": container with ID starting with dc8c120a3032b01ecd3fcb1833a1ce143eeed58653cd4ff5050176a9f091c59c not found: ID does not exist" containerID="dc8c120a3032b01ecd3fcb1833a1ce143eeed58653cd4ff5050176a9f091c59c" Oct 31 13:13:27.071007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:27.064038 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc8c120a3032b01ecd3fcb1833a1ce143eeed58653cd4ff5050176a9f091c59c"} err="failed to get container status \"dc8c120a3032b01ecd3fcb1833a1ce143eeed58653cd4ff5050176a9f091c59c\": rpc error: code = NotFound desc = could not find container \"dc8c120a3032b01ecd3fcb1833a1ce143eeed58653cd4ff5050176a9f091c59c\": container with ID starting with dc8c120a3032b01ecd3fcb1833a1ce143eeed58653cd4ff5050176a9f091c59c not found: ID does not exist" Oct 31 13:13:27.071007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:27.064065 6841 scope.go:122] "RemoveContainer" containerID="e1b923b781d65b19a56188d0fcd7efbfe3583d5d54eb0096a2caabb79c0e1bf9" Oct 31 13:13:27.071007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:27.069883 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt"] Oct 31 13:13:27.088929 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:27.079056 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4n2wt"] Oct 31 13:13:27.088929 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:27.083091 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21566728-c2cd-42a6-9ffe-7aaf14243ee8-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:27.088929 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:27.083134 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t696z\" (UniqueName: \"kubernetes.io/projected/21566728-c2cd-42a6-9ffe-7aaf14243ee8-kube-api-access-t696z\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:27.088929 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:27.083151 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21566728-c2cd-42a6-9ffe-7aaf14243ee8-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:27.088929 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:27.083165 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21566728-c2cd-42a6-9ffe-7aaf14243ee8-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:27.105411 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:27.104847 6841 scope.go:122] "RemoveContainer" containerID="e1b923b781d65b19a56188d0fcd7efbfe3583d5d54eb0096a2caabb79c0e1bf9" Oct 31 13:13:27.105411 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:13:27.105293 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1b923b781d65b19a56188d0fcd7efbfe3583d5d54eb0096a2caabb79c0e1bf9\": container with ID starting with e1b923b781d65b19a56188d0fcd7efbfe3583d5d54eb0096a2caabb79c0e1bf9 not found: ID does not exist" containerID="e1b923b781d65b19a56188d0fcd7efbfe3583d5d54eb0096a2caabb79c0e1bf9" Oct 31 13:13:27.105411 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:27.105329 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1b923b781d65b19a56188d0fcd7efbfe3583d5d54eb0096a2caabb79c0e1bf9"} err="failed to get container status \"e1b923b781d65b19a56188d0fcd7efbfe3583d5d54eb0096a2caabb79c0e1bf9\": rpc error: code = NotFound desc = could not find container \"e1b923b781d65b19a56188d0fcd7efbfe3583d5d54eb0096a2caabb79c0e1bf9\": container with ID starting with e1b923b781d65b19a56188d0fcd7efbfe3583d5d54eb0096a2caabb79c0e1bf9 not found: ID does not exist" Oct 31 13:13:27.115113 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:27.115032 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z"] Oct 31 13:13:27.118074 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:27.117964 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-4pj4z"] Oct 31 13:13:27.456119 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:27.456084 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:27.644416 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:27.644383 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21566728-c2cd-42a6-9ffe-7aaf14243ee8" path="/var/lib/kubelet/pods/21566728-c2cd-42a6-9ffe-7aaf14243ee8/volumes" Oct 31 13:13:27.644916 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:27.644900 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c763c4c-2410-4735-9e8d-98d8ec38142c" path="/var/lib/kubelet/pods/8c763c4c-2410-4735-9e8d-98d8ec38142c/volumes" Oct 31 13:13:28.304545 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:28.300857 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:29.006015 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.005703 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:29.281131 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.281096 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" Oct 31 13:13:29.320166 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.320136 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" Oct 31 13:13:29.431467 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.431430 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6fdf066-cd83-408d-9c1c-6c97f9189f33-kube-api-access-5q2nv\" (UniqueName: \"kubernetes.io/projected/d6fdf066-cd83-408d-9c1c-6c97f9189f33-kube-api-access-5q2nv\") pod \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\" (UID: \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\") " Oct 31 13:13:29.431664 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.431478 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6fdf066-cd83-408d-9c1c-6c97f9189f33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6fdf066-cd83-408d-9c1c-6c97f9189f33-credential-socket\") pod \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\" (UID: \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\") " Oct 31 13:13:29.431664 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.431528 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88130f73-b9a3-4bdb-aea9-8557803120cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88130f73-b9a3-4bdb-aea9-8557803120cd-istio-envoy\") pod \"88130f73-b9a3-4bdb-aea9-8557803120cd\" (UID: \"88130f73-b9a3-4bdb-aea9-8557803120cd\") " Oct 31 13:13:29.431664 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.431563 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6fdf066-cd83-408d-9c1c-6c97f9189f33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6fdf066-cd83-408d-9c1c-6c97f9189f33-workload-socket\") pod \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\" (UID: \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\") " Oct 31 13:13:29.431664 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.431611 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6fdf066-cd83-408d-9c1c-6c97f9189f33-istio-token\" (UniqueName: \"kubernetes.io/projected/d6fdf066-cd83-408d-9c1c-6c97f9189f33-istio-token\") pod \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\" (UID: \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\") " Oct 31 13:13:29.431891 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.431668 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88130f73-b9a3-4bdb-aea9-8557803120cd-kube-api-access-5j8bf\" (UniqueName: \"kubernetes.io/projected/88130f73-b9a3-4bdb-aea9-8557803120cd-kube-api-access-5j8bf\") pod \"88130f73-b9a3-4bdb-aea9-8557803120cd\" (UID: \"88130f73-b9a3-4bdb-aea9-8557803120cd\") " Oct 31 13:13:29.431891 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.431696 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88130f73-b9a3-4bdb-aea9-8557803120cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88130f73-b9a3-4bdb-aea9-8557803120cd-istiod-ca-cert\") pod \"88130f73-b9a3-4bdb-aea9-8557803120cd\" (UID: \"88130f73-b9a3-4bdb-aea9-8557803120cd\") " Oct 31 13:13:29.431891 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.431739 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88130f73-b9a3-4bdb-aea9-8557803120cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88130f73-b9a3-4bdb-aea9-8557803120cd-workload-certs\") pod \"88130f73-b9a3-4bdb-aea9-8557803120cd\" (UID: \"88130f73-b9a3-4bdb-aea9-8557803120cd\") " Oct 31 13:13:29.431891 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.431826 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88130f73-b9a3-4bdb-aea9-8557803120cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88130f73-b9a3-4bdb-aea9-8557803120cd-istio-data\") pod \"88130f73-b9a3-4bdb-aea9-8557803120cd\" (UID: \"88130f73-b9a3-4bdb-aea9-8557803120cd\") " Oct 31 13:13:29.431891 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.431849 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6fdf066-cd83-408d-9c1c-6c97f9189f33-credential-socket" pod "d6fdf066-cd83-408d-9c1c-6c97f9189f33" (UID: "d6fdf066-cd83-408d-9c1c-6c97f9189f33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:29.432222 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.431896 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6fdf066-cd83-408d-9c1c-6c97f9189f33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6fdf066-cd83-408d-9c1c-6c97f9189f33-istio-data\") pod \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\" (UID: \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\") " Oct 31 13:13:29.432222 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.431924 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6fdf066-cd83-408d-9c1c-6c97f9189f33-workload-socket" pod "d6fdf066-cd83-408d-9c1c-6c97f9189f33" (UID: "d6fdf066-cd83-408d-9c1c-6c97f9189f33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:29.432222 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.431963 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6fdf066-cd83-408d-9c1c-6c97f9189f33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6fdf066-cd83-408d-9c1c-6c97f9189f33-istio-envoy\") pod \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\" (UID: \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\") " Oct 31 13:13:29.432222 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.432049 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6fdf066-cd83-408d-9c1c-6c97f9189f33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6fdf066-cd83-408d-9c1c-6c97f9189f33-istio-podinfo\") pod \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\" (UID: \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\") " Oct 31 13:13:29.432222 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.432121 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88130f73-b9a3-4bdb-aea9-8557803120cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88130f73-b9a3-4bdb-aea9-8557803120cd-credential-socket\") pod \"88130f73-b9a3-4bdb-aea9-8557803120cd\" (UID: \"88130f73-b9a3-4bdb-aea9-8557803120cd\") " Oct 31 13:13:29.432222 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.432169 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88130f73-b9a3-4bdb-aea9-8557803120cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88130f73-b9a3-4bdb-aea9-8557803120cd-istio-podinfo\") pod \"88130f73-b9a3-4bdb-aea9-8557803120cd\" (UID: \"88130f73-b9a3-4bdb-aea9-8557803120cd\") " Oct 31 13:13:29.434245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.432264 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88130f73-b9a3-4bdb-aea9-8557803120cd-workload-certs" pod "88130f73-b9a3-4bdb-aea9-8557803120cd" (UID: "88130f73-b9a3-4bdb-aea9-8557803120cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:29.434245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.432274 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88130f73-b9a3-4bdb-aea9-8557803120cd-istio-token\" (UniqueName: \"kubernetes.io/projected/88130f73-b9a3-4bdb-aea9-8557803120cd-istio-token\") pod \"88130f73-b9a3-4bdb-aea9-8557803120cd\" (UID: \"88130f73-b9a3-4bdb-aea9-8557803120cd\") " Oct 31 13:13:29.434245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.432331 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88130f73-b9a3-4bdb-aea9-8557803120cd-istio-data" pod "88130f73-b9a3-4bdb-aea9-8557803120cd" (UID: "88130f73-b9a3-4bdb-aea9-8557803120cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:29.434245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.432523 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88130f73-b9a3-4bdb-aea9-8557803120cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88130f73-b9a3-4bdb-aea9-8557803120cd-workload-socket\") pod \"88130f73-b9a3-4bdb-aea9-8557803120cd\" (UID: \"88130f73-b9a3-4bdb-aea9-8557803120cd\") " Oct 31 13:13:29.434245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.432724 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6fdf066-cd83-408d-9c1c-6c97f9189f33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6fdf066-cd83-408d-9c1c-6c97f9189f33-istiod-ca-cert\") pod \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\" (UID: \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\") " Oct 31 13:13:29.434245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.432823 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6fdf066-cd83-408d-9c1c-6c97f9189f33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6fdf066-cd83-408d-9c1c-6c97f9189f33-workload-certs\") pod \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\" (UID: \"d6fdf066-cd83-408d-9c1c-6c97f9189f33\") " Oct 31 13:13:29.434245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.432906 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88130f73-b9a3-4bdb-aea9-8557803120cd-istiod-ca-cert" pod "88130f73-b9a3-4bdb-aea9-8557803120cd" (UID: "88130f73-b9a3-4bdb-aea9-8557803120cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:13:29.434245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.433203 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6fdf066-cd83-408d-9c1c-6c97f9189f33-istio-data" pod "d6fdf066-cd83-408d-9c1c-6c97f9189f33" (UID: "d6fdf066-cd83-408d-9c1c-6c97f9189f33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:29.434245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.433412 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6fdf066-cd83-408d-9c1c-6c97f9189f33-workload-certs" pod "d6fdf066-cd83-408d-9c1c-6c97f9189f33" (UID: "d6fdf066-cd83-408d-9c1c-6c97f9189f33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:29.434245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.433709 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6fdf066-cd83-408d-9c1c-6c97f9189f33-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:29.434245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.433733 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6fdf066-cd83-408d-9c1c-6c97f9189f33-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:29.434245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.433748 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6fdf066-cd83-408d-9c1c-6c97f9189f33-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:29.434245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.433762 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88130f73-b9a3-4bdb-aea9-8557803120cd-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:29.434245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.433778 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88130f73-b9a3-4bdb-aea9-8557803120cd-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:29.434245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.433793 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88130f73-b9a3-4bdb-aea9-8557803120cd-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:29.434245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.433809 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6fdf066-cd83-408d-9c1c-6c97f9189f33-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:29.434245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.434081 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88130f73-b9a3-4bdb-aea9-8557803120cd-workload-socket" pod "88130f73-b9a3-4bdb-aea9-8557803120cd" (UID: "88130f73-b9a3-4bdb-aea9-8557803120cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:29.435143 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.434372 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6fdf066-cd83-408d-9c1c-6c97f9189f33-istiod-ca-cert" pod "d6fdf066-cd83-408d-9c1c-6c97f9189f33" (UID: "d6fdf066-cd83-408d-9c1c-6c97f9189f33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:13:29.435143 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.434510 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88130f73-b9a3-4bdb-aea9-8557803120cd-istio-envoy" pod "88130f73-b9a3-4bdb-aea9-8557803120cd" (UID: "88130f73-b9a3-4bdb-aea9-8557803120cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:29.435143 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.434551 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6fdf066-cd83-408d-9c1c-6c97f9189f33-kube-api-access-5q2nv" pod "d6fdf066-cd83-408d-9c1c-6c97f9189f33" (UID: "d6fdf066-cd83-408d-9c1c-6c97f9189f33"). InnerVolumeSpecName "kube-api-access-5q2nv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:13:29.435143 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.434585 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88130f73-b9a3-4bdb-aea9-8557803120cd-kube-api-access-5j8bf" pod "88130f73-b9a3-4bdb-aea9-8557803120cd" (UID: "88130f73-b9a3-4bdb-aea9-8557803120cd"). InnerVolumeSpecName "kube-api-access-5j8bf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:13:29.435143 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.434622 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88130f73-b9a3-4bdb-aea9-8557803120cd-credential-socket" pod "88130f73-b9a3-4bdb-aea9-8557803120cd" (UID: "88130f73-b9a3-4bdb-aea9-8557803120cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:29.435143 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.434903 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88130f73-b9a3-4bdb-aea9-8557803120cd-istio-token" pod "88130f73-b9a3-4bdb-aea9-8557803120cd" (UID: "88130f73-b9a3-4bdb-aea9-8557803120cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:13:29.435143 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.434960 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6fdf066-cd83-408d-9c1c-6c97f9189f33-istio-token" pod "d6fdf066-cd83-408d-9c1c-6c97f9189f33" (UID: "d6fdf066-cd83-408d-9c1c-6c97f9189f33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:13:29.435822 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.435804 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6fdf066-cd83-408d-9c1c-6c97f9189f33-istio-podinfo" pod "d6fdf066-cd83-408d-9c1c-6c97f9189f33" (UID: "d6fdf066-cd83-408d-9c1c-6c97f9189f33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:13:29.436319 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.436299 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88130f73-b9a3-4bdb-aea9-8557803120cd-istio-podinfo" pod "88130f73-b9a3-4bdb-aea9-8557803120cd" (UID: "88130f73-b9a3-4bdb-aea9-8557803120cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:13:29.436478 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.436466 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6fdf066-cd83-408d-9c1c-6c97f9189f33-istio-envoy" pod "d6fdf066-cd83-408d-9c1c-6c97f9189f33" (UID: "d6fdf066-cd83-408d-9c1c-6c97f9189f33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:29.535460 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.535375 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6fdf066-cd83-408d-9c1c-6c97f9189f33-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:29.535460 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.535409 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6fdf066-cd83-408d-9c1c-6c97f9189f33-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:29.535460 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.535421 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88130f73-b9a3-4bdb-aea9-8557803120cd-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:29.535460 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.535439 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88130f73-b9a3-4bdb-aea9-8557803120cd-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:29.535460 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.535448 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88130f73-b9a3-4bdb-aea9-8557803120cd-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:29.535460 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.535459 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88130f73-b9a3-4bdb-aea9-8557803120cd-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:29.535771 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.535473 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6fdf066-cd83-408d-9c1c-6c97f9189f33-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:29.535771 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.535486 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q2nv\" (UniqueName: \"kubernetes.io/projected/d6fdf066-cd83-408d-9c1c-6c97f9189f33-kube-api-access-5q2nv\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:29.535771 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.535502 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88130f73-b9a3-4bdb-aea9-8557803120cd-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:29.535771 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.535517 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6fdf066-cd83-408d-9c1c-6c97f9189f33-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:29.535771 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.535529 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j8bf\" (UniqueName: \"kubernetes.io/projected/88130f73-b9a3-4bdb-aea9-8557803120cd-kube-api-access-5j8bf\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:29.785018 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:29.782401 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:30.038029 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.037977 6841 generic.go:356] "Generic (PLEG): container finished" podID="88130f73-b9a3-4bdb-aea9-8557803120cd" containerID="6f64a098a6c9c98c5415c07a9dd0056fa668064189f876792c8718448e12c60b" exitCode=0 Oct 31 13:13:30.038200 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.038082 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" Oct 31 13:13:30.038200 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.038103 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" event={"ID":"88130f73-b9a3-4bdb-aea9-8557803120cd","Type":"ContainerDied","Data":"6f64a098a6c9c98c5415c07a9dd0056fa668064189f876792c8718448e12c60b"} Oct 31 13:13:30.038200 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.038173 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz" event={"ID":"88130f73-b9a3-4bdb-aea9-8557803120cd","Type":"ContainerDied","Data":"6e0ac44c7785581911206d8e5665881110cc03668915506247d646a97c6efe1b"} Oct 31 13:13:30.038390 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.038208 6841 scope.go:122] "RemoveContainer" containerID="6f64a098a6c9c98c5415c07a9dd0056fa668064189f876792c8718448e12c60b" Oct 31 13:13:30.040054 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.040031 6841 generic.go:356] "Generic (PLEG): container finished" podID="d6fdf066-cd83-408d-9c1c-6c97f9189f33" containerID="a978080ec4f8e1250f3f8f4caa420a97949bd4955f0d1632ea4208f690ccc40c" exitCode=0 Oct 31 13:13:30.040187 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.040086 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" Oct 31 13:13:30.040187 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.040091 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" event={"ID":"d6fdf066-cd83-408d-9c1c-6c97f9189f33","Type":"ContainerDied","Data":"a978080ec4f8e1250f3f8f4caa420a97949bd4955f0d1632ea4208f690ccc40c"} Oct 31 13:13:30.040187 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.040138 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh" event={"ID":"d6fdf066-cd83-408d-9c1c-6c97f9189f33","Type":"ContainerDied","Data":"dec44256f518e7139bd768fd3237e7629058d6db5877c89c727220c3deed5f56"} Oct 31 13:13:30.062162 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.062077 6841 scope.go:122] "RemoveContainer" containerID="6f64a098a6c9c98c5415c07a9dd0056fa668064189f876792c8718448e12c60b" Oct 31 13:13:30.062686 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:13:30.062654 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f64a098a6c9c98c5415c07a9dd0056fa668064189f876792c8718448e12c60b\": container with ID starting with 6f64a098a6c9c98c5415c07a9dd0056fa668064189f876792c8718448e12c60b not found: ID does not exist" containerID="6f64a098a6c9c98c5415c07a9dd0056fa668064189f876792c8718448e12c60b" Oct 31 13:13:30.062825 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.062698 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f64a098a6c9c98c5415c07a9dd0056fa668064189f876792c8718448e12c60b"} err="failed to get container status \"6f64a098a6c9c98c5415c07a9dd0056fa668064189f876792c8718448e12c60b\": rpc error: code = NotFound desc = could not find container \"6f64a098a6c9c98c5415c07a9dd0056fa668064189f876792c8718448e12c60b\": container with ID starting with 6f64a098a6c9c98c5415c07a9dd0056fa668064189f876792c8718448e12c60b not found: ID does not exist" Oct 31 13:13:30.062825 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.062724 6841 scope.go:122] "RemoveContainer" containerID="a978080ec4f8e1250f3f8f4caa420a97949bd4955f0d1632ea4208f690ccc40c" Oct 31 13:13:30.074187 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.074168 6841 scope.go:122] "RemoveContainer" containerID="a978080ec4f8e1250f3f8f4caa420a97949bd4955f0d1632ea4208f690ccc40c" Oct 31 13:13:30.074488 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:13:30.074468 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a978080ec4f8e1250f3f8f4caa420a97949bd4955f0d1632ea4208f690ccc40c\": container with ID starting with a978080ec4f8e1250f3f8f4caa420a97949bd4955f0d1632ea4208f690ccc40c not found: ID does not exist" containerID="a978080ec4f8e1250f3f8f4caa420a97949bd4955f0d1632ea4208f690ccc40c" Oct 31 13:13:30.074562 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.074495 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a978080ec4f8e1250f3f8f4caa420a97949bd4955f0d1632ea4208f690ccc40c"} err="failed to get container status \"a978080ec4f8e1250f3f8f4caa420a97949bd4955f0d1632ea4208f690ccc40c\": rpc error: code = NotFound desc = could not find container \"a978080ec4f8e1250f3f8f4caa420a97949bd4955f0d1632ea4208f690ccc40c\": container with ID starting with a978080ec4f8e1250f3f8f4caa420a97949bd4955f0d1632ea4208f690ccc40c not found: ID does not exist" Oct 31 13:13:30.076462 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.076440 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz"] Oct 31 13:13:30.082829 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.082806 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jsnzz"] Oct 31 13:13:30.095567 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.095543 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh"] Oct 31 13:13:30.103586 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.103561 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jrmsh"] Oct 31 13:13:30.371191 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.371095 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd"] Oct 31 13:13:30.373210 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.373179 6841 cpu_manager.go:414] "containerMap: removing container" podUID="4a034a05-49d4-4c08-a9b5-5299af987eb1" containerName="main" Oct 31 13:13:30.373210 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.373210 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a034a05-49d4-4c08-a9b5-5299af987eb1" containerName="main" Oct 31 13:13:30.373360 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.373226 6841 cpu_manager.go:414] "containerMap: removing container" podUID="21566728-c2cd-42a6-9ffe-7aaf14243ee8" containerName="istio-proxy" Oct 31 13:13:30.373360 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.373238 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21566728-c2cd-42a6-9ffe-7aaf14243ee8" containerName="istio-proxy" Oct 31 13:13:30.373360 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.373263 6841 cpu_manager.go:414] "containerMap: removing container" podUID="88130f73-b9a3-4bdb-aea9-8557803120cd" containerName="istio-proxy" Oct 31 13:13:30.373360 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.373272 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88130f73-b9a3-4bdb-aea9-8557803120cd" containerName="istio-proxy" Oct 31 13:13:30.373360 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.373295 6841 cpu_manager.go:414] "containerMap: removing container" podUID="8c763c4c-2410-4735-9e8d-98d8ec38142c" containerName="istio-proxy" Oct 31 13:13:30.373360 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.373303 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c763c4c-2410-4735-9e8d-98d8ec38142c" containerName="istio-proxy" Oct 31 13:13:30.373360 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.373318 6841 cpu_manager.go:414] "containerMap: removing container" podUID="d6fdf066-cd83-408d-9c1c-6c97f9189f33" containerName="istio-proxy" Oct 31 13:13:30.373360 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.373324 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6fdf066-cd83-408d-9c1c-6c97f9189f33" containerName="istio-proxy" Oct 31 13:13:30.373622 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.373538 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a034a05-49d4-4c08-a9b5-5299af987eb1" containerName="main" Oct 31 13:13:30.373622 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.373557 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="21566728-c2cd-42a6-9ffe-7aaf14243ee8" containerName="istio-proxy" Oct 31 13:13:30.373622 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.373568 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="88130f73-b9a3-4bdb-aea9-8557803120cd" containerName="istio-proxy" Oct 31 13:13:30.373622 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.373579 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c763c4c-2410-4735-9e8d-98d8ec38142c" containerName="istio-proxy" Oct 31 13:13:30.373622 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.373588 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6fdf066-cd83-408d-9c1c-6c97f9189f33" containerName="istio-proxy" Oct 31 13:13:30.379964 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.379943 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd" Oct 31 13:13:30.385778 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.385754 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd"] Oct 31 13:13:30.387697 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.387674 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-54a16b-serving-cert\"" Oct 31 13:13:30.528651 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.528617 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:30.558334 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.558302 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdltx\" (UniqueName: \"kubernetes.io/projected/def1f5e7-9be6-4470-b70c-e387a4469d2f-kube-api-access-sdltx\") pod \"maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd\" (UID: \"def1f5e7-9be6-4470-b70c-e387a4469d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd" Oct 31 13:13:30.558532 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.558383 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/def1f5e7-9be6-4470-b70c-e387a4469d2f-metrics-certs\") pod \"maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd\" (UID: \"def1f5e7-9be6-4470-b70c-e387a4469d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd" Oct 31 13:13:30.558657 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.558634 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/def1f5e7-9be6-4470-b70c-e387a4469d2f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd\" (UID: \"def1f5e7-9be6-4470-b70c-e387a4469d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd" Oct 31 13:13:30.558721 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.558694 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/def1f5e7-9be6-4470-b70c-e387a4469d2f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd\" (UID: \"def1f5e7-9be6-4470-b70c-e387a4469d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd" Oct 31 13:13:30.667074 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.660288 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/def1f5e7-9be6-4470-b70c-e387a4469d2f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd\" (UID: \"def1f5e7-9be6-4470-b70c-e387a4469d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd" Oct 31 13:13:30.667074 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.660358 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/def1f5e7-9be6-4470-b70c-e387a4469d2f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd\" (UID: \"def1f5e7-9be6-4470-b70c-e387a4469d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd" Oct 31 13:13:30.667074 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.660544 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdltx\" (UniqueName: \"kubernetes.io/projected/def1f5e7-9be6-4470-b70c-e387a4469d2f-kube-api-access-sdltx\") pod \"maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd\" (UID: \"def1f5e7-9be6-4470-b70c-e387a4469d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd" Oct 31 13:13:30.667074 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.660634 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/def1f5e7-9be6-4470-b70c-e387a4469d2f-metrics-certs\") pod \"maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd\" (UID: \"def1f5e7-9be6-4470-b70c-e387a4469d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd" Oct 31 13:13:30.667074 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.662292 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/def1f5e7-9be6-4470-b70c-e387a4469d2f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd\" (UID: \"def1f5e7-9be6-4470-b70c-e387a4469d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd" Oct 31 13:13:30.671184 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.670856 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/def1f5e7-9be6-4470-b70c-e387a4469d2f-metrics-certs\") pod \"maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd\" (UID: \"def1f5e7-9be6-4470-b70c-e387a4469d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd" Oct 31 13:13:30.677202 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.676819 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/def1f5e7-9be6-4470-b70c-e387a4469d2f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd\" (UID: \"def1f5e7-9be6-4470-b70c-e387a4469d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd" Oct 31 13:13:30.682350 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.679655 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdltx\" (UniqueName: \"kubernetes.io/projected/def1f5e7-9be6-4470-b70c-e387a4469d2f-kube-api-access-sdltx\") pod \"maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd\" (UID: \"def1f5e7-9be6-4470-b70c-e387a4469d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd" Oct 31 13:13:30.701535 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.701505 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd" Oct 31 13:13:30.738320 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:30.738292 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:31.072822 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:31.072785 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd"] Oct 31 13:13:31.119167 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:31.119138 6841 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 13:13:31.425018 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:31.424819 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:31.644936 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:31.644910 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88130f73-b9a3-4bdb-aea9-8557803120cd" path="/var/lib/kubelet/pods/88130f73-b9a3-4bdb-aea9-8557803120cd/volumes" Oct 31 13:13:31.645589 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:31.645566 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6fdf066-cd83-408d-9c1c-6c97f9189f33" path="/var/lib/kubelet/pods/d6fdf066-cd83-408d-9c1c-6c97f9189f33/volumes" Oct 31 13:13:32.151777 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:32.151740 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd" event={"ID":"def1f5e7-9be6-4470-b70c-e387a4469d2f","Type":"ContainerStarted","Data":"2fa10498bcf16f65525b47ab458442f284135491eed9b7840c25940a5e9f0af3"} Oct 31 13:13:32.151972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:32.151835 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd" event={"ID":"def1f5e7-9be6-4470-b70c-e387a4469d2f","Type":"ContainerStarted","Data":"a4cd1256cb3a51ccd810f551cd1d712374d58cbb1e561951fab1ec5819cb9920"} Oct 31 13:13:32.151972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:32.151868 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd" Oct 31 13:13:32.176100 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:32.176048 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd" podStartSLOduration=1.7361408059999999 podStartE2EDuration="2.176031552s" totalImagesPullingTime="439.890746ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:32.172567549 +0000 UTC m=+2458.655690464" watchObservedRunningTime="2026-10-31 13:13:32.176031552 +0000 UTC m=+2458.659154459" Oct 31 13:13:32.422657 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:32.422568 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:33.453021 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:33.452974 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:34.433074 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:34.433033 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:34.564442 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:34.564403 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx"] Oct 31 13:13:34.564914 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:34.564602 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" podUID="66f79621-82da-431c-8c50-12eb405fc80e" containerName="istio-proxy" containerID="cri-o://11e28327d752bee8b194d902b9a722da8c96e5e59de7a3e66cb238b33c1f1a37" gracePeriod=30 Oct 31 13:13:35.439428 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:35.439395 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:36.228077 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:36.227719 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:36.861457 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:36.861426 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94"] Oct 31 13:13:36.868139 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:36.868109 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94"] Oct 31 13:13:36.868290 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:36.868237 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94" Oct 31 13:13:36.873340 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:36.873311 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-5dc570-serving-cert\"" Oct 31 13:13:36.990449 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:36.990412 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b508f7f6-8c58-4f97-bf9b-60ef66ad84db-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94\" (UID: \"b508f7f6-8c58-4f97-bf9b-60ef66ad84db\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94" Oct 31 13:13:36.990656 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:36.990468 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b508f7f6-8c58-4f97-bf9b-60ef66ad84db-maas-api-tls\") pod \"maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94\" (UID: \"b508f7f6-8c58-4f97-bf9b-60ef66ad84db\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94" Oct 31 13:13:36.990656 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:36.990489 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz85j\" (UniqueName: \"kubernetes.io/projected/b508f7f6-8c58-4f97-bf9b-60ef66ad84db-kube-api-access-cz85j\") pod \"maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94\" (UID: \"b508f7f6-8c58-4f97-bf9b-60ef66ad84db\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94" Oct 31 13:13:36.990791 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:36.990669 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b508f7f6-8c58-4f97-bf9b-60ef66ad84db-metrics-certs\") pod \"maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94\" (UID: \"b508f7f6-8c58-4f97-bf9b-60ef66ad84db\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94" Oct 31 13:13:36.995763 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:36.995736 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:37.098533 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:37.096004 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b508f7f6-8c58-4f97-bf9b-60ef66ad84db-maas-api-tls\") pod \"maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94\" (UID: \"b508f7f6-8c58-4f97-bf9b-60ef66ad84db\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94" Oct 31 13:13:37.098533 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:37.096066 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz85j\" (UniqueName: \"kubernetes.io/projected/b508f7f6-8c58-4f97-bf9b-60ef66ad84db-kube-api-access-cz85j\") pod \"maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94\" (UID: \"b508f7f6-8c58-4f97-bf9b-60ef66ad84db\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94" Oct 31 13:13:37.098533 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:37.096185 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b508f7f6-8c58-4f97-bf9b-60ef66ad84db-metrics-certs\") pod \"maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94\" (UID: \"b508f7f6-8c58-4f97-bf9b-60ef66ad84db\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94" Oct 31 13:13:37.098533 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:37.096467 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b508f7f6-8c58-4f97-bf9b-60ef66ad84db-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94\" (UID: \"b508f7f6-8c58-4f97-bf9b-60ef66ad84db\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94" Oct 31 13:13:37.098533 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:37.097425 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b508f7f6-8c58-4f97-bf9b-60ef66ad84db-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94\" (UID: \"b508f7f6-8c58-4f97-bf9b-60ef66ad84db\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94" Oct 31 13:13:37.105187 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:37.105146 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b508f7f6-8c58-4f97-bf9b-60ef66ad84db-metrics-certs\") pod \"maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94\" (UID: \"b508f7f6-8c58-4f97-bf9b-60ef66ad84db\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94" Oct 31 13:13:37.120069 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:37.112834 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b508f7f6-8c58-4f97-bf9b-60ef66ad84db-maas-api-tls\") pod \"maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94\" (UID: \"b508f7f6-8c58-4f97-bf9b-60ef66ad84db\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94" Oct 31 13:13:37.122762 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:37.121496 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz85j\" (UniqueName: \"kubernetes.io/projected/b508f7f6-8c58-4f97-bf9b-60ef66ad84db-kube-api-access-cz85j\") pod \"maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94\" (UID: \"b508f7f6-8c58-4f97-bf9b-60ef66ad84db\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94" Oct 31 13:13:37.189351 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:37.189318 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94" Oct 31 13:13:37.206268 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:37.206239 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:37.564346 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:37.563904 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94"] Oct 31 13:13:37.571186 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:13:37.571148 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb508f7f6_8c58_4f97_bf9b_60ef66ad84db.slice/crio-406a67ce52fbf30a7e9606a8199113880effae2fdf7a85ac8f0be42eb83c3fc9 WatchSource:0}: Error finding container 406a67ce52fbf30a7e9606a8199113880effae2fdf7a85ac8f0be42eb83c3fc9: Status 404 returned error can't find the container with id 406a67ce52fbf30a7e9606a8199113880effae2fdf7a85ac8f0be42eb83c3fc9 Oct 31 13:13:37.861127 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:37.861037 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:38.059315 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:38.059272 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:38.165057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:38.165019 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd" Oct 31 13:13:38.259236 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:38.259188 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94" event={"ID":"b508f7f6-8c58-4f97-bf9b-60ef66ad84db","Type":"ContainerStarted","Data":"730188602af72d5f0f82df444657e3ea6896e22e595a333d23e9b30af8231356"} Oct 31 13:13:38.259460 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:38.259269 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94" Oct 31 13:13:38.259460 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:38.259298 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94" event={"ID":"b508f7f6-8c58-4f97-bf9b-60ef66ad84db","Type":"ContainerStarted","Data":"406a67ce52fbf30a7e9606a8199113880effae2fdf7a85ac8f0be42eb83c3fc9"} Oct 31 13:13:38.294655 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:38.294601 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94" podStartSLOduration=1.813492114 podStartE2EDuration="2.294586882s" totalImagesPullingTime="481.094768ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:38.290406321 +0000 UTC m=+2464.773529222" watchObservedRunningTime="2026-10-31 13:13:38.294586882 +0000 UTC m=+2464.777709818" Oct 31 13:13:38.691728 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:38.691692 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:39.446806 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:39.446772 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:39.649194 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:39.649165 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:39.865197 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:39.865171 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" Oct 31 13:13:40.002116 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.002024 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66f79621-82da-431c-8c50-12eb405fc80e-kube-api-access-6qz4z\" (UniqueName: \"kubernetes.io/projected/66f79621-82da-431c-8c50-12eb405fc80e-kube-api-access-6qz4z\") pod \"66f79621-82da-431c-8c50-12eb405fc80e\" (UID: \"66f79621-82da-431c-8c50-12eb405fc80e\") " Oct 31 13:13:40.002116 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.002068 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66f79621-82da-431c-8c50-12eb405fc80e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66f79621-82da-431c-8c50-12eb405fc80e-istio-envoy\") pod \"66f79621-82da-431c-8c50-12eb405fc80e\" (UID: \"66f79621-82da-431c-8c50-12eb405fc80e\") " Oct 31 13:13:40.002309 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.002233 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66f79621-82da-431c-8c50-12eb405fc80e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66f79621-82da-431c-8c50-12eb405fc80e-workload-certs\") pod \"66f79621-82da-431c-8c50-12eb405fc80e\" (UID: \"66f79621-82da-431c-8c50-12eb405fc80e\") " Oct 31 13:13:40.002309 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.002295 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66f79621-82da-431c-8c50-12eb405fc80e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f79621-82da-431c-8c50-12eb405fc80e-credential-socket\") pod \"66f79621-82da-431c-8c50-12eb405fc80e\" (UID: \"66f79621-82da-431c-8c50-12eb405fc80e\") " Oct 31 13:13:40.002395 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.002374 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66f79621-82da-431c-8c50-12eb405fc80e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66f79621-82da-431c-8c50-12eb405fc80e-istio-podinfo\") pod \"66f79621-82da-431c-8c50-12eb405fc80e\" (UID: \"66f79621-82da-431c-8c50-12eb405fc80e\") " Oct 31 13:13:40.002472 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.002456 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66f79621-82da-431c-8c50-12eb405fc80e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f79621-82da-431c-8c50-12eb405fc80e-workload-socket\") pod \"66f79621-82da-431c-8c50-12eb405fc80e\" (UID: \"66f79621-82da-431c-8c50-12eb405fc80e\") " Oct 31 13:13:40.002577 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.002488 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66f79621-82da-431c-8c50-12eb405fc80e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66f79621-82da-431c-8c50-12eb405fc80e-istio-data\") pod \"66f79621-82da-431c-8c50-12eb405fc80e\" (UID: \"66f79621-82da-431c-8c50-12eb405fc80e\") " Oct 31 13:13:40.002577 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.002498 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f79621-82da-431c-8c50-12eb405fc80e-workload-certs" pod "66f79621-82da-431c-8c50-12eb405fc80e" (UID: "66f79621-82da-431c-8c50-12eb405fc80e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:40.002577 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.002524 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f79621-82da-431c-8c50-12eb405fc80e-credential-socket" pod "66f79621-82da-431c-8c50-12eb405fc80e" (UID: "66f79621-82da-431c-8c50-12eb405fc80e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:40.002577 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.002573 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66f79621-82da-431c-8c50-12eb405fc80e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66f79621-82da-431c-8c50-12eb405fc80e-istiod-ca-cert\") pod \"66f79621-82da-431c-8c50-12eb405fc80e\" (UID: \"66f79621-82da-431c-8c50-12eb405fc80e\") " Oct 31 13:13:40.002815 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.002652 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66f79621-82da-431c-8c50-12eb405fc80e-istio-token\" (UniqueName: \"kubernetes.io/projected/66f79621-82da-431c-8c50-12eb405fc80e-istio-token\") pod \"66f79621-82da-431c-8c50-12eb405fc80e\" (UID: \"66f79621-82da-431c-8c50-12eb405fc80e\") " Oct 31 13:13:40.002815 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.002735 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f79621-82da-431c-8c50-12eb405fc80e-istio-data" pod "66f79621-82da-431c-8c50-12eb405fc80e" (UID: "66f79621-82da-431c-8c50-12eb405fc80e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:40.002815 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.002749 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f79621-82da-431c-8c50-12eb405fc80e-workload-socket" pod "66f79621-82da-431c-8c50-12eb405fc80e" (UID: "66f79621-82da-431c-8c50-12eb405fc80e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:40.003038 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.002958 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66f79621-82da-431c-8c50-12eb405fc80e-istiod-ca-cert" pod "66f79621-82da-431c-8c50-12eb405fc80e" (UID: "66f79621-82da-431c-8c50-12eb405fc80e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:13:40.003357 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.003339 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66f79621-82da-431c-8c50-12eb405fc80e-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:40.003357 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.003359 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f79621-82da-431c-8c50-12eb405fc80e-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:40.003531 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.003368 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f79621-82da-431c-8c50-12eb405fc80e-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:40.003531 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.003377 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66f79621-82da-431c-8c50-12eb405fc80e-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:40.003531 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.003386 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66f79621-82da-431c-8c50-12eb405fc80e-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:40.004501 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.004475 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f79621-82da-431c-8c50-12eb405fc80e-istio-envoy" pod "66f79621-82da-431c-8c50-12eb405fc80e" (UID: "66f79621-82da-431c-8c50-12eb405fc80e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:40.004795 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.004773 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66f79621-82da-431c-8c50-12eb405fc80e-kube-api-access-6qz4z" pod "66f79621-82da-431c-8c50-12eb405fc80e" (UID: "66f79621-82da-431c-8c50-12eb405fc80e"). InnerVolumeSpecName "kube-api-access-6qz4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:13:40.004973 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.004955 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66f79621-82da-431c-8c50-12eb405fc80e-istio-podinfo" pod "66f79621-82da-431c-8c50-12eb405fc80e" (UID: "66f79621-82da-431c-8c50-12eb405fc80e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:13:40.005081 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.005046 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66f79621-82da-431c-8c50-12eb405fc80e-istio-token" pod "66f79621-82da-431c-8c50-12eb405fc80e" (UID: "66f79621-82da-431c-8c50-12eb405fc80e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:13:40.104764 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.104726 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66f79621-82da-431c-8c50-12eb405fc80e-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:40.104764 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.104761 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qz4z\" (UniqueName: \"kubernetes.io/projected/66f79621-82da-431c-8c50-12eb405fc80e-kube-api-access-6qz4z\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:40.104764 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.104772 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66f79621-82da-431c-8c50-12eb405fc80e-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:40.105130 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.104780 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66f79621-82da-431c-8c50-12eb405fc80e-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:40.314874 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.314784 6841 generic.go:356] "Generic (PLEG): container finished" podID="66f79621-82da-431c-8c50-12eb405fc80e" containerID="11e28327d752bee8b194d902b9a722da8c96e5e59de7a3e66cb238b33c1f1a37" exitCode=0 Oct 31 13:13:40.315084 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.314871 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" Oct 31 13:13:40.315084 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.314873 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" event={"ID":"66f79621-82da-431c-8c50-12eb405fc80e","Type":"ContainerDied","Data":"11e28327d752bee8b194d902b9a722da8c96e5e59de7a3e66cb238b33c1f1a37"} Oct 31 13:13:40.315084 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.314972 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx" event={"ID":"66f79621-82da-431c-8c50-12eb405fc80e","Type":"ContainerDied","Data":"80fb4791d4140e35975193ef8dd0938c53166400e9b1b8dd1870003cc1f2f2ab"} Oct 31 13:13:40.315084 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.315025 6841 scope.go:122] "RemoveContainer" containerID="11e28327d752bee8b194d902b9a722da8c96e5e59de7a3e66cb238b33c1f1a37" Oct 31 13:13:40.328786 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.328762 6841 scope.go:122] "RemoveContainer" containerID="11e28327d752bee8b194d902b9a722da8c96e5e59de7a3e66cb238b33c1f1a37" Oct 31 13:13:40.329073 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:13:40.329045 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11e28327d752bee8b194d902b9a722da8c96e5e59de7a3e66cb238b33c1f1a37\": container with ID starting with 11e28327d752bee8b194d902b9a722da8c96e5e59de7a3e66cb238b33c1f1a37 not found: ID does not exist" containerID="11e28327d752bee8b194d902b9a722da8c96e5e59de7a3e66cb238b33c1f1a37" Oct 31 13:13:40.329181 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.329082 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11e28327d752bee8b194d902b9a722da8c96e5e59de7a3e66cb238b33c1f1a37"} err="failed to get container status \"11e28327d752bee8b194d902b9a722da8c96e5e59de7a3e66cb238b33c1f1a37\": rpc error: code = NotFound desc = could not find container \"11e28327d752bee8b194d902b9a722da8c96e5e59de7a3e66cb238b33c1f1a37\": container with ID starting with 11e28327d752bee8b194d902b9a722da8c96e5e59de7a3e66cb238b33c1f1a37 not found: ID does not exist" Oct 31 13:13:40.376206 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.376170 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx"] Oct 31 13:13:40.401083 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.401051 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-ghnhx"] Oct 31 13:13:40.461502 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.461465 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:40.987166 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:40.987129 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:41.250299 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:41.250212 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:41.348784 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:41.348737 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt"] Oct 31 13:13:41.350159 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:41.350127 6841 cpu_manager.go:414] "containerMap: removing container" podUID="66f79621-82da-431c-8c50-12eb405fc80e" containerName="istio-proxy" Oct 31 13:13:41.350159 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:41.350155 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66f79621-82da-431c-8c50-12eb405fc80e" containerName="istio-proxy" Oct 31 13:13:41.350418 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:41.350401 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="66f79621-82da-431c-8c50-12eb405fc80e" containerName="istio-proxy" Oct 31 13:13:41.356227 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:41.356202 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt" Oct 31 13:13:41.365575 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:41.365548 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt"] Oct 31 13:13:41.370454 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:41.370431 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-fa8c3f-serving-cert\"" Oct 31 13:13:41.524964 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:41.519927 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:41.539356 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:41.539299 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7754c3ef-a2ff-4a52-b371-873053b3d1ad-openshift-service-ca\") pod \"maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt\" (UID: \"7754c3ef-a2ff-4a52-b371-873053b3d1ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt" Oct 31 13:13:41.539696 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:41.539663 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5k7d\" (UniqueName: \"kubernetes.io/projected/7754c3ef-a2ff-4a52-b371-873053b3d1ad-kube-api-access-f5k7d\") pod \"maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt\" (UID: \"7754c3ef-a2ff-4a52-b371-873053b3d1ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt" Oct 31 13:13:41.539834 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:41.539817 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7754c3ef-a2ff-4a52-b371-873053b3d1ad-metrics-certs\") pod \"maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt\" (UID: \"7754c3ef-a2ff-4a52-b371-873053b3d1ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt" Oct 31 13:13:41.540009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:41.539969 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7754c3ef-a2ff-4a52-b371-873053b3d1ad-maas-api-tls\") pod \"maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt\" (UID: \"7754c3ef-a2ff-4a52-b371-873053b3d1ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt" Oct 31 13:13:41.641954 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:41.641922 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7754c3ef-a2ff-4a52-b371-873053b3d1ad-openshift-service-ca\") pod \"maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt\" (UID: \"7754c3ef-a2ff-4a52-b371-873053b3d1ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt" Oct 31 13:13:41.642283 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:41.641998 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5k7d\" (UniqueName: \"kubernetes.io/projected/7754c3ef-a2ff-4a52-b371-873053b3d1ad-kube-api-access-f5k7d\") pod \"maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt\" (UID: \"7754c3ef-a2ff-4a52-b371-873053b3d1ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt" Oct 31 13:13:41.642283 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:41.642030 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7754c3ef-a2ff-4a52-b371-873053b3d1ad-metrics-certs\") pod \"maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt\" (UID: \"7754c3ef-a2ff-4a52-b371-873053b3d1ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt" Oct 31 13:13:41.642283 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:41.642069 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7754c3ef-a2ff-4a52-b371-873053b3d1ad-maas-api-tls\") pod \"maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt\" (UID: \"7754c3ef-a2ff-4a52-b371-873053b3d1ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt" Oct 31 13:13:41.642862 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:41.642840 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7754c3ef-a2ff-4a52-b371-873053b3d1ad-openshift-service-ca\") pod \"maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt\" (UID: \"7754c3ef-a2ff-4a52-b371-873053b3d1ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt" Oct 31 13:13:41.644662 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:41.644642 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7754c3ef-a2ff-4a52-b371-873053b3d1ad-metrics-certs\") pod \"maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt\" (UID: \"7754c3ef-a2ff-4a52-b371-873053b3d1ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt" Oct 31 13:13:41.644857 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:41.644836 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7754c3ef-a2ff-4a52-b371-873053b3d1ad-maas-api-tls\") pod \"maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt\" (UID: \"7754c3ef-a2ff-4a52-b371-873053b3d1ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt" Oct 31 13:13:41.645430 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:41.645409 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66f79621-82da-431c-8c50-12eb405fc80e" path="/var/lib/kubelet/pods/66f79621-82da-431c-8c50-12eb405fc80e/volumes" Oct 31 13:13:41.658531 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:41.658506 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5k7d\" (UniqueName: \"kubernetes.io/projected/7754c3ef-a2ff-4a52-b371-873053b3d1ad-kube-api-access-f5k7d\") pod \"maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt\" (UID: \"7754c3ef-a2ff-4a52-b371-873053b3d1ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt" Oct 31 13:13:41.675083 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:41.675060 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt" Oct 31 13:13:41.767593 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:41.767557 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:42.027008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:42.020943 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt"] Oct 31 13:13:42.213449 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:42.213411 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:42.377932 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:42.377795 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt" event={"ID":"7754c3ef-a2ff-4a52-b371-873053b3d1ad","Type":"ContainerStarted","Data":"4a836a63b0c28677e9109a5d123474ae3a247df1afd8eac938af06c326a2491c"} Oct 31 13:13:42.449327 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:42.449291 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:42.812387 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:42.812339 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:43.185015 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:43.184698 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:43.569251 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:43.566153 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt" event={"ID":"7754c3ef-a2ff-4a52-b371-873053b3d1ad","Type":"ContainerStarted","Data":"f3f7e7267e29c7ef71fcfd1bbf39a8c6d44a85bdb49309d7d89aa7931360ae7f"} Oct 31 13:13:43.572512 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:43.570667 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt" Oct 31 13:13:43.593725 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:43.593675 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt" podStartSLOduration=2.143787044 podStartE2EDuration="2.593658637s" totalImagesPullingTime="449.871593ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:43.592081582 +0000 UTC m=+2470.075204482" watchObservedRunningTime="2026-10-31 13:13:43.593658637 +0000 UTC m=+2470.076781538" Oct 31 13:13:44.061621 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:44.061577 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm"] Oct 31 13:13:44.066449 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:44.066422 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm" Oct 31 13:13:44.077579 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:44.077547 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-29e3a0-serving-cert\"" Oct 31 13:13:44.082091 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:44.082066 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm"] Oct 31 13:13:44.131315 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:44.131276 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2a5911a-8b22-49ae-b976-dc61217c3dc2-metrics-certs\") pod \"maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm\" (UID: \"d2a5911a-8b22-49ae-b976-dc61217c3dc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm" Oct 31 13:13:44.131547 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:44.131324 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljf45\" (UniqueName: \"kubernetes.io/projected/d2a5911a-8b22-49ae-b976-dc61217c3dc2-kube-api-access-ljf45\") pod \"maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm\" (UID: \"d2a5911a-8b22-49ae-b976-dc61217c3dc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm" Oct 31 13:13:44.131639 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:44.131593 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2a5911a-8b22-49ae-b976-dc61217c3dc2-maas-api-tls\") pod \"maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm\" (UID: \"d2a5911a-8b22-49ae-b976-dc61217c3dc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm" Oct 31 13:13:44.131816 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:44.131799 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2a5911a-8b22-49ae-b976-dc61217c3dc2-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm\" (UID: \"d2a5911a-8b22-49ae-b976-dc61217c3dc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm" Oct 31 13:13:44.213742 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:44.213700 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:44.242298 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:44.241876 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2a5911a-8b22-49ae-b976-dc61217c3dc2-metrics-certs\") pod \"maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm\" (UID: \"d2a5911a-8b22-49ae-b976-dc61217c3dc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm" Oct 31 13:13:44.242298 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:44.241937 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljf45\" (UniqueName: \"kubernetes.io/projected/d2a5911a-8b22-49ae-b976-dc61217c3dc2-kube-api-access-ljf45\") pod \"maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm\" (UID: \"d2a5911a-8b22-49ae-b976-dc61217c3dc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm" Oct 31 13:13:44.242298 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:44.241975 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2a5911a-8b22-49ae-b976-dc61217c3dc2-maas-api-tls\") pod \"maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm\" (UID: \"d2a5911a-8b22-49ae-b976-dc61217c3dc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm" Oct 31 13:13:44.242641 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:44.242386 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2a5911a-8b22-49ae-b976-dc61217c3dc2-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm\" (UID: \"d2a5911a-8b22-49ae-b976-dc61217c3dc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm" Oct 31 13:13:44.250831 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:44.243447 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2a5911a-8b22-49ae-b976-dc61217c3dc2-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm\" (UID: \"d2a5911a-8b22-49ae-b976-dc61217c3dc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm" Oct 31 13:13:44.250831 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:13:44.244914 6841 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-29e3a0-serving-cert: secret "maas-api-e2e-extmod-b-29e3a0-serving-cert" not found Oct 31 13:13:44.250831 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:13:44.244998 6841 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2a5911a-8b22-49ae-b976-dc61217c3dc2-maas-api-tls podName:d2a5911a-8b22-49ae-b976-dc61217c3dc2 nodeName:}" failed. No retries permitted until 2026-10-31 13:13:44.744962799 +0000 UTC m=+2471.228085686 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d2a5911a-8b22-49ae-b976-dc61217c3dc2-maas-api-tls") pod "maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm" (UID: "d2a5911a-8b22-49ae-b976-dc61217c3dc2") : secret "maas-api-e2e-extmod-b-29e3a0-serving-cert" not found Oct 31 13:13:44.250831 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:44.249330 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2a5911a-8b22-49ae-b976-dc61217c3dc2-metrics-certs\") pod \"maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm\" (UID: \"d2a5911a-8b22-49ae-b976-dc61217c3dc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm" Oct 31 13:13:44.285373 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:44.285340 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljf45\" (UniqueName: \"kubernetes.io/projected/d2a5911a-8b22-49ae-b976-dc61217c3dc2-kube-api-access-ljf45\") pod \"maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm\" (UID: \"d2a5911a-8b22-49ae-b976-dc61217c3dc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm" Oct 31 13:13:44.292286 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:44.292254 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94" Oct 31 13:13:44.421908 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:44.421812 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:44.664405 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:44.662846 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:44.767069 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:44.763283 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2a5911a-8b22-49ae-b976-dc61217c3dc2-maas-api-tls\") pod \"maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm\" (UID: \"d2a5911a-8b22-49ae-b976-dc61217c3dc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm" Oct 31 13:13:44.779048 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:44.774277 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2a5911a-8b22-49ae-b976-dc61217c3dc2-maas-api-tls\") pod \"maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm\" (UID: \"d2a5911a-8b22-49ae-b976-dc61217c3dc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm" Oct 31 13:13:44.984074 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:44.984041 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm" Oct 31 13:13:45.211059 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:45.210809 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:45.282143 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:45.282112 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm"] Oct 31 13:13:45.282586 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:13:45.282162 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a5911a_8b22_49ae_b976_dc61217c3dc2.slice/crio-f5501312a3422bb6dd7375f874130d9939fb1dc5a37f4fede040bd3a85c191fb WatchSource:0}: Error finding container f5501312a3422bb6dd7375f874130d9939fb1dc5a37f4fede040bd3a85c191fb: Status 404 returned error can't find the container with id f5501312a3422bb6dd7375f874130d9939fb1dc5a37f4fede040bd3a85c191fb Oct 31 13:13:45.582453 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:45.582416 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm" event={"ID":"d2a5911a-8b22-49ae-b976-dc61217c3dc2","Type":"ContainerStarted","Data":"f5501312a3422bb6dd7375f874130d9939fb1dc5a37f4fede040bd3a85c191fb"} Oct 31 13:13:46.114334 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:46.114303 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:46.591239 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:46.591207 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm" event={"ID":"d2a5911a-8b22-49ae-b976-dc61217c3dc2","Type":"ContainerStarted","Data":"b8797fff3993c20d9facd6a09da766a51df4129a6c97e90d0aee028814d9896d"} Oct 31 13:13:46.591752 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:46.591323 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm" Oct 31 13:13:47.294168 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:47.294131 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:48.321180 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:48.321148 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:49.127976 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:49.127938 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:50.030923 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:50.030889 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:50.057560 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:13:50.057527 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:13:50.588486 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:50.588414 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt" Oct 31 13:13:50.623893 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:50.623846 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm" podStartSLOduration=6.241871998 podStartE2EDuration="6.623833721s" totalImagesPullingTime="381.961723ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:46.612827481 +0000 UTC m=+2473.095950383" watchObservedRunningTime="2026-10-31 13:13:50.623833721 +0000 UTC m=+2477.106956621" Oct 31 13:13:50.896702 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:50.894657 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:51.748603 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:51.748573 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:52.571002 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:52.564935 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:52.604284 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:52.604256 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm" Oct 31 13:13:53.483000 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:53.480750 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:54.635395 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:54.635354 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:55.780482 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:55.780447 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:56.085922 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:56.085825 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:56.867918 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:56.867868 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:57.367662 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:57.367630 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:57.735639 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:57.735601 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4"] Oct 31 13:13:57.760318 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:57.760285 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:13:57.782884 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:57.782854 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-0c1fbb-openshift-default-dockercfg-bgnzh\"" Oct 31 13:13:57.795428 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:57.793281 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4"] Oct 31 13:13:57.821609 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:57.821572 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42529d86-63e7-47dd-a03d-3e2ca7737a52-istiod-ca-cert\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4\" (UID: \"42529d86-63e7-47dd-a03d-3e2ca7737a52\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:13:57.821756 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:57.821623 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42529d86-63e7-47dd-a03d-3e2ca7737a52-istio-envoy\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4\" (UID: \"42529d86-63e7-47dd-a03d-3e2ca7737a52\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:13:57.821756 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:57.821711 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8jfg\" (UniqueName: \"kubernetes.io/projected/42529d86-63e7-47dd-a03d-3e2ca7737a52-kube-api-access-s8jfg\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4\" (UID: \"42529d86-63e7-47dd-a03d-3e2ca7737a52\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:13:57.821843 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:57.821806 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42529d86-63e7-47dd-a03d-3e2ca7737a52-workload-socket\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4\" (UID: \"42529d86-63e7-47dd-a03d-3e2ca7737a52\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:13:57.821952 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:57.821935 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42529d86-63e7-47dd-a03d-3e2ca7737a52-credential-socket\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4\" (UID: \"42529d86-63e7-47dd-a03d-3e2ca7737a52\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:13:57.822003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:57.821977 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42529d86-63e7-47dd-a03d-3e2ca7737a52-workload-certs\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4\" (UID: \"42529d86-63e7-47dd-a03d-3e2ca7737a52\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:13:57.822050 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:57.822035 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42529d86-63e7-47dd-a03d-3e2ca7737a52-istio-token\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4\" (UID: \"42529d86-63e7-47dd-a03d-3e2ca7737a52\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:13:57.822145 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:57.822132 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42529d86-63e7-47dd-a03d-3e2ca7737a52-istio-data\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4\" (UID: \"42529d86-63e7-47dd-a03d-3e2ca7737a52\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:13:57.822272 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:57.822247 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42529d86-63e7-47dd-a03d-3e2ca7737a52-istio-podinfo\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4\" (UID: \"42529d86-63e7-47dd-a03d-3e2ca7737a52\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:13:57.873630 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:57.873591 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:57.948007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:57.929194 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42529d86-63e7-47dd-a03d-3e2ca7737a52-istio-data\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4\" (UID: \"42529d86-63e7-47dd-a03d-3e2ca7737a52\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:13:57.949062 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:57.949028 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42529d86-63e7-47dd-a03d-3e2ca7737a52-istio-podinfo\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4\" (UID: \"42529d86-63e7-47dd-a03d-3e2ca7737a52\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:13:57.949861 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:57.949829 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42529d86-63e7-47dd-a03d-3e2ca7737a52-istiod-ca-cert\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4\" (UID: \"42529d86-63e7-47dd-a03d-3e2ca7737a52\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:13:57.973704 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:57.973675 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42529d86-63e7-47dd-a03d-3e2ca7737a52-istio-envoy\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4\" (UID: \"42529d86-63e7-47dd-a03d-3e2ca7737a52\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:13:57.974471 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:57.974440 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8jfg\" (UniqueName: \"kubernetes.io/projected/42529d86-63e7-47dd-a03d-3e2ca7737a52-kube-api-access-s8jfg\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4\" (UID: \"42529d86-63e7-47dd-a03d-3e2ca7737a52\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:13:57.975056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:57.975034 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42529d86-63e7-47dd-a03d-3e2ca7737a52-workload-socket\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4\" (UID: \"42529d86-63e7-47dd-a03d-3e2ca7737a52\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:13:57.975413 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:57.975394 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42529d86-63e7-47dd-a03d-3e2ca7737a52-credential-socket\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4\" (UID: \"42529d86-63e7-47dd-a03d-3e2ca7737a52\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:13:57.975560 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:57.975546 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42529d86-63e7-47dd-a03d-3e2ca7737a52-workload-certs\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4\" (UID: \"42529d86-63e7-47dd-a03d-3e2ca7737a52\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:13:57.975685 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:57.975671 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42529d86-63e7-47dd-a03d-3e2ca7737a52-istio-token\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4\" (UID: \"42529d86-63e7-47dd-a03d-3e2ca7737a52\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:13:57.979999 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:57.945493 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42529d86-63e7-47dd-a03d-3e2ca7737a52-istio-data\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4\" (UID: \"42529d86-63e7-47dd-a03d-3e2ca7737a52\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:13:57.998052 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:57.980480 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42529d86-63e7-47dd-a03d-3e2ca7737a52-workload-socket\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4\" (UID: \"42529d86-63e7-47dd-a03d-3e2ca7737a52\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:13:57.998301 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:57.982365 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42529d86-63e7-47dd-a03d-3e2ca7737a52-credential-socket\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4\" (UID: \"42529d86-63e7-47dd-a03d-3e2ca7737a52\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:13:57.998425 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:57.950892 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42529d86-63e7-47dd-a03d-3e2ca7737a52-istiod-ca-cert\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4\" (UID: \"42529d86-63e7-47dd-a03d-3e2ca7737a52\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:13:57.998523 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:57.982609 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42529d86-63e7-47dd-a03d-3e2ca7737a52-workload-certs\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4\" (UID: \"42529d86-63e7-47dd-a03d-3e2ca7737a52\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:13:58.001384 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:58.001358 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42529d86-63e7-47dd-a03d-3e2ca7737a52-istio-token\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4\" (UID: \"42529d86-63e7-47dd-a03d-3e2ca7737a52\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:13:58.004686 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:58.004632 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8jfg\" (UniqueName: \"kubernetes.io/projected/42529d86-63e7-47dd-a03d-3e2ca7737a52-kube-api-access-s8jfg\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4\" (UID: \"42529d86-63e7-47dd-a03d-3e2ca7737a52\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:13:58.016028 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:58.005347 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42529d86-63e7-47dd-a03d-3e2ca7737a52-istio-podinfo\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4\" (UID: \"42529d86-63e7-47dd-a03d-3e2ca7737a52\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:13:58.028428 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:58.028337 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42529d86-63e7-47dd-a03d-3e2ca7737a52-istio-envoy\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4\" (UID: \"42529d86-63e7-47dd-a03d-3e2ca7737a52\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:13:58.103562 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:58.103292 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:13:58.136900 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:58.136816 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:58.500797 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:58.500768 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4"] Oct 31 13:13:58.501233 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:13:58.501208 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42529d86_63e7_47dd_a03d_3e2ca7737a52.slice/crio-50c7c13fd9d99327dd53fe5547b26d79250f686dac5f222b1c29b6837621a6c3 WatchSource:0}: Error finding container 50c7c13fd9d99327dd53fe5547b26d79250f686dac5f222b1c29b6837621a6c3: Status 404 returned error can't find the container with id 50c7c13fd9d99327dd53fe5547b26d79250f686dac5f222b1c29b6837621a6c3 Oct 31 13:13:58.504360 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:58.504327 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:13:58.504451 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:58.504413 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:13:58.504451 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:58.504441 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:13:58.706838 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:58.706807 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" event={"ID":"42529d86-63e7-47dd-a03d-3e2ca7737a52","Type":"ContainerStarted","Data":"705f880daff07ec87b8b01f01987326bf511cff6093825ebd6f562d4963788e0"} Oct 31 13:13:58.707035 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:58.706859 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" event={"ID":"42529d86-63e7-47dd-a03d-3e2ca7737a52","Type":"ContainerStarted","Data":"50c7c13fd9d99327dd53fe5547b26d79250f686dac5f222b1c29b6837621a6c3"} Oct 31 13:13:58.776789 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:58.776529 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" podStartSLOduration=1.77650837 podStartE2EDuration="1.77650837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:58.74745602 +0000 UTC m=+2485.230578924" watchObservedRunningTime="2026-10-31 13:13:58.77650837 +0000 UTC m=+2485.259631273" Oct 31 13:13:58.931041 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:58.916293 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:13:59.105120 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:59.105015 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:13:59.106471 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:59.106438 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 31 13:13:59.106592 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:59.106501 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" podUID="42529d86-63e7-47dd-a03d-3e2ca7737a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 31 13:13:59.727469 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:13:59.727431 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:00.110199 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:00.110125 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:14:00.771057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:00.771022 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:14:00.774025 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:00.773938 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:14:00.824547 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:00.824158 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:01.386023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:01.385958 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:01.699103 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:01.699071 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:02.200871 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:02.200824 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:02.496946 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:02.496909 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:02.747529 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:02.747446 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:03.384123 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:03.384058 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:03.871993 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:03.871954 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:04.184272 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:04.184179 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht"] Oct 31 13:14:04.204690 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:04.204660 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:14:04.207476 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:04.207445 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht"] Oct 31 13:14:04.214039 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:04.213974 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-c2c98b-openshift-default-dockercfg-pcv65\"" Oct 31 13:14:04.268007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:04.262412 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:04.558022 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:04.531999 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd61135-1d8a-44a5-9771-a66f1c88cd18-credential-socket\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht\" (UID: \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:14:04.651109 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:04.625762 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:04.651109 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:04.639911 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfd61135-1d8a-44a5-9771-a66f1c88cd18-istio-data\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht\" (UID: \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:14:04.651109 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:04.640050 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfd61135-1d8a-44a5-9771-a66f1c88cd18-workload-certs\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht\" (UID: \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:14:04.651109 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:04.640084 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfd61135-1d8a-44a5-9771-a66f1c88cd18-istio-podinfo\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht\" (UID: \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:14:04.651109 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:04.640243 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfd61135-1d8a-44a5-9771-a66f1c88cd18-istio-token\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht\" (UID: \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:14:04.670770 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:04.670731 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfd61135-1d8a-44a5-9771-a66f1c88cd18-istio-envoy\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht\" (UID: \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:14:04.705157 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:04.705099 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb2z4\" (UniqueName: \"kubernetes.io/projected/cfd61135-1d8a-44a5-9771-a66f1c88cd18-kube-api-access-jb2z4\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht\" (UID: \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:14:04.705964 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:04.705943 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfd61135-1d8a-44a5-9771-a66f1c88cd18-istiod-ca-cert\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht\" (UID: \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:14:04.706257 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:04.706130 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd61135-1d8a-44a5-9771-a66f1c88cd18-workload-socket\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht\" (UID: \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:14:04.807456 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:04.807415 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfd61135-1d8a-44a5-9771-a66f1c88cd18-istio-token\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht\" (UID: \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:14:04.807650 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:04.807470 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfd61135-1d8a-44a5-9771-a66f1c88cd18-istio-envoy\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht\" (UID: \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:14:04.807650 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:04.807510 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb2z4\" (UniqueName: \"kubernetes.io/projected/cfd61135-1d8a-44a5-9771-a66f1c88cd18-kube-api-access-jb2z4\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht\" (UID: \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:14:04.807650 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:04.807550 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfd61135-1d8a-44a5-9771-a66f1c88cd18-istiod-ca-cert\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht\" (UID: \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:14:04.807650 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:04.807597 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd61135-1d8a-44a5-9771-a66f1c88cd18-workload-socket\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht\" (UID: \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:14:04.807918 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:04.807854 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd61135-1d8a-44a5-9771-a66f1c88cd18-credential-socket\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht\" (UID: \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:14:04.807918 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:04.807875 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfd61135-1d8a-44a5-9771-a66f1c88cd18-istio-data\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht\" (UID: \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:14:04.807918 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:04.807912 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfd61135-1d8a-44a5-9771-a66f1c88cd18-workload-certs\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht\" (UID: \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:14:04.808142 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:04.807943 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfd61135-1d8a-44a5-9771-a66f1c88cd18-istio-podinfo\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht\" (UID: \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:14:04.809093 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:04.808352 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfd61135-1d8a-44a5-9771-a66f1c88cd18-istio-data\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht\" (UID: \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:14:04.809093 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:04.808616 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd61135-1d8a-44a5-9771-a66f1c88cd18-workload-socket\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht\" (UID: \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:14:04.809093 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:04.808893 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfd61135-1d8a-44a5-9771-a66f1c88cd18-workload-certs\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht\" (UID: \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:14:04.809093 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:04.808951 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd61135-1d8a-44a5-9771-a66f1c88cd18-credential-socket\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht\" (UID: \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:14:04.809093 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:04.808995 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfd61135-1d8a-44a5-9771-a66f1c88cd18-istiod-ca-cert\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht\" (UID: \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:14:04.810319 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:04.810293 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfd61135-1d8a-44a5-9771-a66f1c88cd18-istio-envoy\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht\" (UID: \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:14:04.810972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:04.810953 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfd61135-1d8a-44a5-9771-a66f1c88cd18-istio-podinfo\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht\" (UID: \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:14:04.821475 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:04.821449 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfd61135-1d8a-44a5-9771-a66f1c88cd18-istio-token\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht\" (UID: \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:14:04.821945 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:04.821925 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb2z4\" (UniqueName: \"kubernetes.io/projected/cfd61135-1d8a-44a5-9771-a66f1c88cd18-kube-api-access-jb2z4\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht\" (UID: \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:14:04.828941 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:04.828916 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:14:04.872009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:04.859341 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:05.203975 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:05.197111 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht"] Oct 31 13:14:05.240249 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:05.240037 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:05.260197 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:14:05.245107 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfd61135_1d8a_44a5_9771_a66f1c88cd18.slice/crio-f828a7191bcae5112272097267dc1365d8d8ec188c4a0fa342353e5bf7da663e WatchSource:0}: Error finding container f828a7191bcae5112272097267dc1365d8d8ec188c4a0fa342353e5bf7da663e: Status 404 returned error can't find the container with id f828a7191bcae5112272097267dc1365d8d8ec188c4a0fa342353e5bf7da663e Oct 31 13:14:05.278189 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:05.267350 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:14:05.278189 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:05.267472 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:14:05.278189 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:05.267524 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:14:06.003060 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:05.997965 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" event={"ID":"cfd61135-1d8a-44a5-9771-a66f1c88cd18","Type":"ContainerStarted","Data":"9a1b6eabb4d08a835841e53610bafb1356c9c10fc97dc39635d66a00bd40bf90"} Oct 31 13:14:06.003060 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:05.998056 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" event={"ID":"cfd61135-1d8a-44a5-9771-a66f1c88cd18","Type":"ContainerStarted","Data":"f828a7191bcae5112272097267dc1365d8d8ec188c4a0fa342353e5bf7da663e"} Oct 31 13:14:06.066393 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:06.054880 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" podStartSLOduration=2.05485953 podStartE2EDuration="2.05485953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:14:06.037098099 +0000 UTC m=+2492.520221000" watchObservedRunningTime="2026-10-31 13:14:06.05485953 +0000 UTC m=+2492.537982432" Oct 31 13:14:06.272688 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:06.272599 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:06.829864 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:06.829828 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:14:06.836503 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:06.836471 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:14:07.006286 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:07.006253 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:14:07.017608 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:07.017576 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:14:07.250293 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:07.248469 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:08.265837 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:08.261729 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:14:08.265837 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:08.261802 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" podUID="bdaa9d86-a826-4768-ae0d-471ce6e65e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:08.265837 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:08.265116 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:08.538449 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:08.538347 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:08.781316 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:08.781283 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:09.057942 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:09.057911 6841 generic.go:356] "Generic (PLEG): container finished" podID="bdaa9d86-a826-4768-ae0d-471ce6e65e15" containerID="0b855bab5c92549530dbfaddf0071e0362ee969a7fc687a03a902ac4b17cabcd" exitCode=0 Oct 31 13:14:09.058126 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:09.058016 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" event={"ID":"bdaa9d86-a826-4768-ae0d-471ce6e65e15","Type":"ContainerDied","Data":"0b855bab5c92549530dbfaddf0071e0362ee969a7fc687a03a902ac4b17cabcd"} Oct 31 13:14:09.058706 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:09.058683 6841 scope.go:122] "RemoveContainer" containerID="0b855bab5c92549530dbfaddf0071e0362ee969a7fc687a03a902ac4b17cabcd" Oct 31 13:14:09.060579 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:09.060551 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:14:09.060652 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:09.060628 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:14:09.060690 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:09.060661 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:14:09.290171 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:09.289327 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:09.548502 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:09.548464 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:09.809423 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:09.809334 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb"] Oct 31 13:14:09.814490 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:09.814468 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb" Oct 31 13:14:09.828327 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:09.827486 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-fa8c3f\"/\"kube-root-ca.crt\"" Oct 31 13:14:09.828327 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:09.827563 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-fa8c3f\"/\"openshift-service-ca.crt\"" Oct 31 13:14:09.832373 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:09.832345 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb"] Oct 31 13:14:09.883009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:09.871160 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-fa8c3f\"/\"autores-model-fa8c3f-kserve-self-signed-certs\"" Oct 31 13:14:09.883009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:09.871547 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-fa8c3f\"/\"default-dockercfg-c4xkx\"" Oct 31 13:14:09.884182 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:09.884137 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skvpx\" (UniqueName: \"kubernetes.io/projected/15523312-bf9c-46ab-ab57-e5ce6d128d9f-kube-api-access-skvpx\") pod \"autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb\" (UID: \"15523312-bf9c-46ab-ab57-e5ce6d128d9f\") " pod="ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb" Oct 31 13:14:09.884329 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:09.884211 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15523312-bf9c-46ab-ab57-e5ce6d128d9f-model-cache\") pod \"autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb\" (UID: \"15523312-bf9c-46ab-ab57-e5ce6d128d9f\") " pod="ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb" Oct 31 13:14:09.884329 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:09.884284 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15523312-bf9c-46ab-ab57-e5ce6d128d9f-home\") pod \"autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb\" (UID: \"15523312-bf9c-46ab-ab57-e5ce6d128d9f\") " pod="ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb" Oct 31 13:14:09.884961 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:09.884428 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15523312-bf9c-46ab-ab57-e5ce6d128d9f-tmp-dir\") pod \"autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb\" (UID: \"15523312-bf9c-46ab-ab57-e5ce6d128d9f\") " pod="ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb" Oct 31 13:14:09.884961 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:09.884617 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15523312-bf9c-46ab-ab57-e5ce6d128d9f-tls-certs\") pod \"autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb\" (UID: \"15523312-bf9c-46ab-ab57-e5ce6d128d9f\") " pod="ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb" Oct 31 13:14:09.884961 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:09.884818 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15523312-bf9c-46ab-ab57-e5ce6d128d9f-dshm\") pod \"autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb\" (UID: \"15523312-bf9c-46ab-ab57-e5ce6d128d9f\") " pod="ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb" Oct 31 13:14:09.986001 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:09.985956 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15523312-bf9c-46ab-ab57-e5ce6d128d9f-tls-certs\") pod \"autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb\" (UID: \"15523312-bf9c-46ab-ab57-e5ce6d128d9f\") " pod="ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb" Oct 31 13:14:09.986262 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:09.986169 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15523312-bf9c-46ab-ab57-e5ce6d128d9f-dshm\") pod \"autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb\" (UID: \"15523312-bf9c-46ab-ab57-e5ce6d128d9f\") " pod="ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb" Oct 31 13:14:09.986262 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:09.986216 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skvpx\" (UniqueName: \"kubernetes.io/projected/15523312-bf9c-46ab-ab57-e5ce6d128d9f-kube-api-access-skvpx\") pod \"autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb\" (UID: \"15523312-bf9c-46ab-ab57-e5ce6d128d9f\") " pod="ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb" Oct 31 13:14:09.986496 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:09.986297 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15523312-bf9c-46ab-ab57-e5ce6d128d9f-model-cache\") pod \"autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb\" (UID: \"15523312-bf9c-46ab-ab57-e5ce6d128d9f\") " pod="ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb" Oct 31 13:14:09.986496 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:09.986376 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15523312-bf9c-46ab-ab57-e5ce6d128d9f-home\") pod \"autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb\" (UID: \"15523312-bf9c-46ab-ab57-e5ce6d128d9f\") " pod="ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb" Oct 31 13:14:09.986496 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:09.986475 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15523312-bf9c-46ab-ab57-e5ce6d128d9f-tmp-dir\") pod \"autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb\" (UID: \"15523312-bf9c-46ab-ab57-e5ce6d128d9f\") " pod="ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb" Oct 31 13:14:09.987007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:09.986972 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15523312-bf9c-46ab-ab57-e5ce6d128d9f-tmp-dir\") pod \"autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb\" (UID: \"15523312-bf9c-46ab-ab57-e5ce6d128d9f\") " pod="ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb" Oct 31 13:14:09.990328 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:09.990300 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15523312-bf9c-46ab-ab57-e5ce6d128d9f-model-cache\") pod \"autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb\" (UID: \"15523312-bf9c-46ab-ab57-e5ce6d128d9f\") " pod="ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb" Oct 31 13:14:09.990616 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:09.990582 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15523312-bf9c-46ab-ab57-e5ce6d128d9f-home\") pod \"autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb\" (UID: \"15523312-bf9c-46ab-ab57-e5ce6d128d9f\") " pod="ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb" Oct 31 13:14:09.994617 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:09.994591 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15523312-bf9c-46ab-ab57-e5ce6d128d9f-dshm\") pod \"autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb\" (UID: \"15523312-bf9c-46ab-ab57-e5ce6d128d9f\") " pod="ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb" Oct 31 13:14:09.995244 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:09.994963 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15523312-bf9c-46ab-ab57-e5ce6d128d9f-tls-certs\") pod \"autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb\" (UID: \"15523312-bf9c-46ab-ab57-e5ce6d128d9f\") " pod="ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb" Oct 31 13:14:10.000122 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:09.999231 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:10.012251 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:10.012224 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skvpx\" (UniqueName: \"kubernetes.io/projected/15523312-bf9c-46ab-ab57-e5ce6d128d9f-kube-api-access-skvpx\") pod \"autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb\" (UID: \"15523312-bf9c-46ab-ab57-e5ce6d128d9f\") " pod="ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb" Oct 31 13:14:10.089142 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:10.089055 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" event={"ID":"bdaa9d86-a826-4768-ae0d-471ce6e65e15","Type":"ContainerStarted","Data":"b8d8d040a18028bc816d2714d48c2058d0b268a7082d0d5c7fdf915039f8ec72"} Oct 31 13:14:10.133617 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:10.133575 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb" Oct 31 13:14:10.388253 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:10.379192 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:10.618885 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:10.618854 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:10.842380 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:10.842206 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:10.880199 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:10.880170 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb"] Oct 31 13:14:10.880831 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:14:10.880801 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15523312_bf9c_46ab_ab57_e5ce6d128d9f.slice/crio-8fe18b4df2376c74d8fadcee4c1fc19b5a8e4b70515a70ff32528a09ea9e9816 WatchSource:0}: Error finding container 8fe18b4df2376c74d8fadcee4c1fc19b5a8e4b70515a70ff32528a09ea9e9816: Status 404 returned error can't find the container with id 8fe18b4df2376c74d8fadcee4c1fc19b5a8e4b70515a70ff32528a09ea9e9816 Oct 31 13:14:10.979040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:10.955934 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:14:10.979040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:10.956206 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 31 13:14:10.979040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:10.956263 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" podUID="bdaa9d86-a826-4768-ae0d-471ce6e65e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 31 13:14:11.068448 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:11.056672 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:11.101142 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:11.101075 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb" event={"ID":"15523312-bf9c-46ab-ab57-e5ce6d128d9f","Type":"ContainerStarted","Data":"8fe18b4df2376c74d8fadcee4c1fc19b5a8e4b70515a70ff32528a09ea9e9816"} Oct 31 13:14:11.425115 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:11.425024 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:11.817400 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:11.817363 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x"] Oct 31 13:14:11.822833 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:11.822808 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" Oct 31 13:14:11.882043 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:11.881890 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x"] Oct 31 13:14:11.886307 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:11.886270 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd82005-b57c-4b49-9bf5-b9a77c624c69-istio-envoy\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x\" (UID: \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" Oct 31 13:14:11.886471 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:11.886321 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd82005-b57c-4b49-9bf5-b9a77c624c69-workload-certs\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x\" (UID: \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" Oct 31 13:14:11.886471 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:11.886420 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd82005-b57c-4b49-9bf5-b9a77c624c69-istiod-ca-cert\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x\" (UID: \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" Oct 31 13:14:11.886620 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:11.886468 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd82005-b57c-4b49-9bf5-b9a77c624c69-workload-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x\" (UID: \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" Oct 31 13:14:11.886620 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:11.886507 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd82005-b57c-4b49-9bf5-b9a77c624c69-credential-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x\" (UID: \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" Oct 31 13:14:11.886744 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:11.886617 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd82005-b57c-4b49-9bf5-b9a77c624c69-istio-podinfo\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x\" (UID: \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" Oct 31 13:14:11.886744 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:11.886657 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd82005-b57c-4b49-9bf5-b9a77c624c69-istio-data\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x\" (UID: \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" Oct 31 13:14:11.886858 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:11.886781 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztb2c\" (UniqueName: \"kubernetes.io/projected/5bd82005-b57c-4b49-9bf5-b9a77c624c69-kube-api-access-ztb2c\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x\" (UID: \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" Oct 31 13:14:11.886858 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:11.886808 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd82005-b57c-4b49-9bf5-b9a77c624c69-istio-token\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x\" (UID: \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" Oct 31 13:14:11.957680 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:11.951310 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 31 13:14:11.957680 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:11.951369 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" podUID="bdaa9d86-a826-4768-ae0d-471ce6e65e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 31 13:14:11.969532 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:11.969502 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:11.988087 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:11.988054 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd82005-b57c-4b49-9bf5-b9a77c624c69-istio-podinfo\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x\" (UID: \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" Oct 31 13:14:11.988266 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:11.988103 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd82005-b57c-4b49-9bf5-b9a77c624c69-istio-data\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x\" (UID: \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" Oct 31 13:14:11.988266 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:11.988222 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztb2c\" (UniqueName: \"kubernetes.io/projected/5bd82005-b57c-4b49-9bf5-b9a77c624c69-kube-api-access-ztb2c\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x\" (UID: \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" Oct 31 13:14:11.988384 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:11.988263 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd82005-b57c-4b49-9bf5-b9a77c624c69-istio-token\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x\" (UID: \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" Oct 31 13:14:11.988457 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:11.988443 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd82005-b57c-4b49-9bf5-b9a77c624c69-istio-envoy\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x\" (UID: \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" Oct 31 13:14:11.988541 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:11.988476 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd82005-b57c-4b49-9bf5-b9a77c624c69-workload-certs\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x\" (UID: \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" Oct 31 13:14:11.988541 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:11.988523 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd82005-b57c-4b49-9bf5-b9a77c624c69-istio-data\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x\" (UID: \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" Oct 31 13:14:11.988657 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:11.988586 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd82005-b57c-4b49-9bf5-b9a77c624c69-istiod-ca-cert\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x\" (UID: \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" Oct 31 13:14:11.988657 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:11.988630 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd82005-b57c-4b49-9bf5-b9a77c624c69-workload-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x\" (UID: \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" Oct 31 13:14:11.988774 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:11.988658 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd82005-b57c-4b49-9bf5-b9a77c624c69-credential-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x\" (UID: \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" Oct 31 13:14:11.989157 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:11.988901 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd82005-b57c-4b49-9bf5-b9a77c624c69-workload-certs\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x\" (UID: \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" Oct 31 13:14:11.989157 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:11.989039 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd82005-b57c-4b49-9bf5-b9a77c624c69-credential-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x\" (UID: \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" Oct 31 13:14:11.989157 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:11.989127 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd82005-b57c-4b49-9bf5-b9a77c624c69-workload-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x\" (UID: \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" Oct 31 13:14:11.989557 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:11.989538 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd82005-b57c-4b49-9bf5-b9a77c624c69-istiod-ca-cert\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x\" (UID: \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" Oct 31 13:14:11.991111 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:11.991089 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd82005-b57c-4b49-9bf5-b9a77c624c69-istio-envoy\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x\" (UID: \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" Oct 31 13:14:11.992101 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:11.992080 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd82005-b57c-4b49-9bf5-b9a77c624c69-istio-podinfo\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x\" (UID: \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" Oct 31 13:14:12.001003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:12.000960 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd82005-b57c-4b49-9bf5-b9a77c624c69-istio-token\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x\" (UID: \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" Oct 31 13:14:12.002612 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:12.002585 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztb2c\" (UniqueName: \"kubernetes.io/projected/5bd82005-b57c-4b49-9bf5-b9a77c624c69-kube-api-access-ztb2c\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x\" (UID: \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" Oct 31 13:14:12.168056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:12.154762 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" Oct 31 13:14:12.275017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:12.262733 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:12.275017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:12.270061 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb" event={"ID":"15523312-bf9c-46ab-ab57-e5ce6d128d9f","Type":"ContainerStarted","Data":"3149382c2831e0226eb1c62f9481b75d4cf9facdf737550e21516c51a9027db5"} Oct 31 13:14:12.594596 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:12.594453 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:12.744539 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:14:12.744486 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bd82005_b57c_4b49_9bf5_b9a77c624c69.slice/crio-ceab437bc96ff71f521af6678dc9590fc78e24951e42282eff5afd7f5fe0ecac WatchSource:0}: Error finding container ceab437bc96ff71f521af6678dc9590fc78e24951e42282eff5afd7f5fe0ecac: Status 404 returned error can't find the container with id ceab437bc96ff71f521af6678dc9590fc78e24951e42282eff5afd7f5fe0ecac Oct 31 13:14:12.775097 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:12.775037 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb" podStartSLOduration=3.743957394 podStartE2EDuration="3.743957394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:14:12.352196655 +0000 UTC m=+2498.835319561" watchObservedRunningTime="2026-10-31 13:14:12.743957394 +0000 UTC m=+2499.227080296" Oct 31 13:14:12.778594 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:12.778548 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x"] Oct 31 13:14:12.780655 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:12.780606 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:14:12.780926 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:12.780901 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:14:12.781177 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:12.781136 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:14:12.833236 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:12.833207 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:12.955385 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:12.954945 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 31 13:14:12.955385 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:12.955024 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" podUID="bdaa9d86-a826-4768-ae0d-471ce6e65e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 31 13:14:13.142173 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:13.142101 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb" Oct 31 13:14:13.284187 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:13.284158 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" event={"ID":"5bd82005-b57c-4b49-9bf5-b9a77c624c69","Type":"ContainerStarted","Data":"695d4845ce63682aac4c250910b1d13a2c7e6a6bc2d0c6bd326517895539c1a2"} Oct 31 13:14:13.284445 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:13.284424 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" event={"ID":"5bd82005-b57c-4b49-9bf5-b9a77c624c69","Type":"ContainerStarted","Data":"ceab437bc96ff71f521af6678dc9590fc78e24951e42282eff5afd7f5fe0ecac"} Oct 31 13:14:13.311651 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:13.310874 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" podStartSLOduration=2.310859537 podStartE2EDuration="2.310859537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:14:13.309638753 +0000 UTC m=+2499.792761671" watchObservedRunningTime="2026-10-31 13:14:13.310859537 +0000 UTC m=+2499.793982437" Oct 31 13:14:13.880921 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:13.880828 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:13.949008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:13.947412 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 31 13:14:13.949008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:13.947486 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" podUID="bdaa9d86-a826-4768-ae0d-471ce6e65e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 31 13:14:14.169061 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:14.166085 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" Oct 31 13:14:14.169061 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:14.167202 6841 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 31 13:14:14.169061 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:14.167255 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" podUID="5bd82005-b57c-4b49-9bf5-b9a77c624c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 31 13:14:14.836620 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:14.828222 6841 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 31 13:14:14.836620 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:14.828294 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" podUID="de4271dd-8fc1-4768-8dd7-2437ee497f9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 31 13:14:14.872023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:14.871965 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:14.988004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:14.980297 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 31 13:14:14.988004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:14.980355 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" podUID="bdaa9d86-a826-4768-ae0d-471ce6e65e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 31 13:14:15.164008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:15.163393 6841 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 31 13:14:15.164008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:15.163488 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" podUID="5bd82005-b57c-4b49-9bf5-b9a77c624c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 31 13:14:15.315394 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:15.315307 6841 generic.go:356] "Generic (PLEG): container finished" podID="de4271dd-8fc1-4768-8dd7-2437ee497f9b" containerID="7502228c566b3d881de24554dfa7716749434c2266896799063260e4b191daf1" exitCode=0 Oct 31 13:14:15.316486 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:15.316458 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" event={"ID":"de4271dd-8fc1-4768-8dd7-2437ee497f9b","Type":"ContainerDied","Data":"7502228c566b3d881de24554dfa7716749434c2266896799063260e4b191daf1"} Oct 31 13:14:15.316922 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:15.316896 6841 scope.go:122] "RemoveContainer" containerID="7502228c566b3d881de24554dfa7716749434c2266896799063260e4b191daf1" Oct 31 13:14:15.318897 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:15.318843 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:14:15.318994 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:15.318917 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:14:15.318994 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:15.318958 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:14:15.762008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:15.761869 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:15.954008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:15.951635 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 31 13:14:15.954008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:15.951691 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" podUID="bdaa9d86-a826-4768-ae0d-471ce6e65e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 31 13:14:16.158088 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:16.157993 6841 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 31 13:14:16.158703 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:16.158652 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" podUID="5bd82005-b57c-4b49-9bf5-b9a77c624c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 31 13:14:16.332789 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:16.332751 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" event={"ID":"de4271dd-8fc1-4768-8dd7-2437ee497f9b","Type":"ContainerStarted","Data":"7195124b4c67e2f273cfc62efa15afb7d3914a5d9b781101d9f947459bf67750"} Oct 31 13:14:16.880019 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:16.877684 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:16.967008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:16.963307 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:14:16.967008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:16.963430 6841 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 31 13:14:16.967008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:16.963473 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" podUID="de4271dd-8fc1-4768-8dd7-2437ee497f9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 31 13:14:16.974861 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:16.974832 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:14:16.980433 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:16.977382 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:14:16.992607 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:16.992581 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:14:17.158517 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:17.158423 6841 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 31 13:14:17.159127 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:17.159091 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" podUID="5bd82005-b57c-4b49-9bf5-b9a77c624c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 31 13:14:17.953317 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:17.953249 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:17.961815 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:17.961357 6841 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 31 13:14:17.961815 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:17.961415 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" podUID="de4271dd-8fc1-4768-8dd7-2437ee497f9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 31 13:14:18.165295 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:18.165256 6841 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 31 13:14:18.168906 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:18.168090 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" podUID="5bd82005-b57c-4b49-9bf5-b9a77c624c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 31 13:14:18.827346 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:18.827306 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:18.968779 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:18.968339 6841 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 31 13:14:18.968779 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:18.968399 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" podUID="de4271dd-8fc1-4768-8dd7-2437ee497f9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 31 13:14:19.169064 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:19.160384 6841 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 31 13:14:19.169064 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:19.160447 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" podUID="5bd82005-b57c-4b49-9bf5-b9a77c624c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 31 13:14:19.651046 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:19.651011 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:19.962095 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:19.962061 6841 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 31 13:14:19.962388 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:19.962359 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" podUID="de4271dd-8fc1-4768-8dd7-2437ee497f9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 31 13:14:20.036122 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:14:20.036083 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:14:20.134055 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:20.134023 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb" Oct 31 13:14:20.170254 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:20.170213 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" Oct 31 13:14:20.170771 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:20.170750 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" Oct 31 13:14:20.171745 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:20.171719 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" Oct 31 13:14:20.173577 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:20.173557 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb" Oct 31 13:14:20.440301 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:20.440195 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:20.487175 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:20.487147 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb" Oct 31 13:14:20.962007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:20.961818 6841 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 31 13:14:20.962007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:20.961879 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" podUID="de4271dd-8fc1-4768-8dd7-2437ee497f9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 31 13:14:21.261088 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:21.260514 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:21.960293 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:21.960257 6841 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 31 13:14:21.960468 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:21.960309 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" podUID="de4271dd-8fc1-4768-8dd7-2437ee497f9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 31 13:14:22.103052 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:22.101827 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:22.932940 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:22.932898 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:22.965221 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:22.965187 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:14:22.965568 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:22.965549 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:14:22.966323 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:22.966305 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:14:23.708453 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:23.708416 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:24.696677 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:24.696639 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:24.910491 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:24.910457 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:25.686639 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:25.685238 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:25.776123 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:25.776090 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb"] Oct 31 13:14:25.924779 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:25.924732 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:26.420142 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:26.420107 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:26.646729 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:26.646690 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:27.262260 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:27.262227 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:28.038900 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:28.038864 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:28.808689 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:28.808635 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:29.820007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:29.816556 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:31.295960 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:31.295898 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:31.561088 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:31.561001 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:32.230599 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:32.227158 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:32.435950 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:32.435879 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:33.166011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:33.164695 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:34.289942 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:34.289907 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:35.190841 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:35.190752 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:36.107507 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:36.107456 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:36.881533 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:36.881133 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:37.821304 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:37.821272 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:38.034400 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:38.034354 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:38.573937 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:38.573902 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:39.440231 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:39.438705 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:40.219156 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:40.219099 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:40.799576 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:40.799527 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb" podUID="15523312-bf9c-46ab-ab57-e5ce6d128d9f" containerName="main" containerID="cri-o://3149382c2831e0226eb1c62f9481b75d4cf9facdf737550e21516c51a9027db5" gracePeriod=45 Oct 31 13:14:40.983886 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:40.983826 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:41.136026 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:41.135972 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb" Oct 31 13:14:41.192358 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:41.192143 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15523312-bf9c-46ab-ab57-e5ce6d128d9f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/15523312-bf9c-46ab-ab57-e5ce6d128d9f-dshm\") pod \"15523312-bf9c-46ab-ab57-e5ce6d128d9f\" (UID: \"15523312-bf9c-46ab-ab57-e5ce6d128d9f\") " Oct 31 13:14:41.192358 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:41.192230 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15523312-bf9c-46ab-ab57-e5ce6d128d9f-kube-api-access-skvpx\" (UniqueName: \"kubernetes.io/projected/15523312-bf9c-46ab-ab57-e5ce6d128d9f-kube-api-access-skvpx\") pod \"15523312-bf9c-46ab-ab57-e5ce6d128d9f\" (UID: \"15523312-bf9c-46ab-ab57-e5ce6d128d9f\") " Oct 31 13:14:41.192358 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:41.192266 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15523312-bf9c-46ab-ab57-e5ce6d128d9f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15523312-bf9c-46ab-ab57-e5ce6d128d9f-model-cache\") pod \"15523312-bf9c-46ab-ab57-e5ce6d128d9f\" (UID: \"15523312-bf9c-46ab-ab57-e5ce6d128d9f\") " Oct 31 13:14:41.192645 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:41.192512 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15523312-bf9c-46ab-ab57-e5ce6d128d9f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15523312-bf9c-46ab-ab57-e5ce6d128d9f-tmp-dir\") pod \"15523312-bf9c-46ab-ab57-e5ce6d128d9f\" (UID: \"15523312-bf9c-46ab-ab57-e5ce6d128d9f\") " Oct 31 13:14:41.192645 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:41.192545 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15523312-bf9c-46ab-ab57-e5ce6d128d9f-home\" (UniqueName: \"kubernetes.io/empty-dir/15523312-bf9c-46ab-ab57-e5ce6d128d9f-home\") pod \"15523312-bf9c-46ab-ab57-e5ce6d128d9f\" (UID: \"15523312-bf9c-46ab-ab57-e5ce6d128d9f\") " Oct 31 13:14:41.192645 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:41.192588 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15523312-bf9c-46ab-ab57-e5ce6d128d9f-tls-certs\" (UniqueName: \"kubernetes.io/secret/15523312-bf9c-46ab-ab57-e5ce6d128d9f-tls-certs\") pod \"15523312-bf9c-46ab-ab57-e5ce6d128d9f\" (UID: \"15523312-bf9c-46ab-ab57-e5ce6d128d9f\") " Oct 31 13:14:41.193205 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:41.193180 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15523312-bf9c-46ab-ab57-e5ce6d128d9f-tmp-dir" pod "15523312-bf9c-46ab-ab57-e5ce6d128d9f" (UID: "15523312-bf9c-46ab-ab57-e5ce6d128d9f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:14:41.193486 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:41.193468 6841 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15523312-bf9c-46ab-ab57-e5ce6d128d9f-tmp-dir\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:14:41.193602 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:41.193579 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15523312-bf9c-46ab-ab57-e5ce6d128d9f-home" pod "15523312-bf9c-46ab-ab57-e5ce6d128d9f" (UID: "15523312-bf9c-46ab-ab57-e5ce6d128d9f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:14:41.193675 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:41.193645 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15523312-bf9c-46ab-ab57-e5ce6d128d9f-model-cache" pod "15523312-bf9c-46ab-ab57-e5ce6d128d9f" (UID: "15523312-bf9c-46ab-ab57-e5ce6d128d9f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:14:41.195422 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:41.195392 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15523312-bf9c-46ab-ab57-e5ce6d128d9f-kube-api-access-skvpx" pod "15523312-bf9c-46ab-ab57-e5ce6d128d9f" (UID: "15523312-bf9c-46ab-ab57-e5ce6d128d9f"). InnerVolumeSpecName "kube-api-access-skvpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:14:41.195675 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:41.195643 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15523312-bf9c-46ab-ab57-e5ce6d128d9f-tls-certs" pod "15523312-bf9c-46ab-ab57-e5ce6d128d9f" (UID: "15523312-bf9c-46ab-ab57-e5ce6d128d9f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:14:41.195853 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:41.195833 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15523312-bf9c-46ab-ab57-e5ce6d128d9f-dshm" pod "15523312-bf9c-46ab-ab57-e5ce6d128d9f" (UID: "15523312-bf9c-46ab-ab57-e5ce6d128d9f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:14:41.295451 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:41.295408 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skvpx\" (UniqueName: \"kubernetes.io/projected/15523312-bf9c-46ab-ab57-e5ce6d128d9f-kube-api-access-skvpx\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:14:41.295451 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:41.295442 6841 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15523312-bf9c-46ab-ab57-e5ce6d128d9f-model-cache\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:14:41.295451 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:41.295456 6841 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15523312-bf9c-46ab-ab57-e5ce6d128d9f-home\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:14:41.295681 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:41.295469 6841 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15523312-bf9c-46ab-ab57-e5ce6d128d9f-tls-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:14:41.295681 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:41.295478 6841 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15523312-bf9c-46ab-ab57-e5ce6d128d9f-dshm\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:14:41.772246 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:41.772213 6841 generic.go:356] "Generic (PLEG): container finished" podID="a8494c61-00b4-41e6-b8ed-f4c2081f8944" containerID="3019597717a794c14594995799ffde8fde8055320e4d2d73d5c079cb751c1deb" exitCode=0 Oct 31 13:14:41.772442 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:41.772293 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" event={"ID":"a8494c61-00b4-41e6-b8ed-f4c2081f8944","Type":"ContainerDied","Data":"3019597717a794c14594995799ffde8fde8055320e4d2d73d5c079cb751c1deb"} Oct 31 13:14:41.772814 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:41.772790 6841 scope.go:122] "RemoveContainer" containerID="3019597717a794c14594995799ffde8fde8055320e4d2d73d5c079cb751c1deb" Oct 31 13:14:41.773962 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:41.773937 6841 generic.go:356] "Generic (PLEG): container finished" podID="15523312-bf9c-46ab-ab57-e5ce6d128d9f" containerID="3149382c2831e0226eb1c62f9481b75d4cf9facdf737550e21516c51a9027db5" exitCode=0 Oct 31 13:14:41.774084 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:41.774062 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb" Oct 31 13:14:41.774150 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:41.774095 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb" event={"ID":"15523312-bf9c-46ab-ab57-e5ce6d128d9f","Type":"ContainerDied","Data":"3149382c2831e0226eb1c62f9481b75d4cf9facdf737550e21516c51a9027db5"} Oct 31 13:14:41.774150 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:41.774129 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb" event={"ID":"15523312-bf9c-46ab-ab57-e5ce6d128d9f","Type":"ContainerDied","Data":"8fe18b4df2376c74d8fadcee4c1fc19b5a8e4b70515a70ff32528a09ea9e9816"} Oct 31 13:14:41.774267 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:41.774151 6841 scope.go:122] "RemoveContainer" containerID="3149382c2831e0226eb1c62f9481b75d4cf9facdf737550e21516c51a9027db5" Oct 31 13:14:41.775229 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:41.775196 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:14:41.775343 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:41.775286 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:14:41.775343 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:41.775334 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:14:41.884942 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:41.884884 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:41.908473 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:41.908438 6841 scope.go:122] "RemoveContainer" containerID="3149382c2831e0226eb1c62f9481b75d4cf9facdf737550e21516c51a9027db5" Oct 31 13:14:41.908917 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:14:41.908875 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3149382c2831e0226eb1c62f9481b75d4cf9facdf737550e21516c51a9027db5\": container with ID starting with 3149382c2831e0226eb1c62f9481b75d4cf9facdf737550e21516c51a9027db5 not found: ID does not exist" containerID="3149382c2831e0226eb1c62f9481b75d4cf9facdf737550e21516c51a9027db5" Oct 31 13:14:41.909048 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:41.908917 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3149382c2831e0226eb1c62f9481b75d4cf9facdf737550e21516c51a9027db5"} err="failed to get container status \"3149382c2831e0226eb1c62f9481b75d4cf9facdf737550e21516c51a9027db5\": rpc error: code = NotFound desc = could not find container \"3149382c2831e0226eb1c62f9481b75d4cf9facdf737550e21516c51a9027db5\": container with ID starting with 3149382c2831e0226eb1c62f9481b75d4cf9facdf737550e21516c51a9027db5 not found: ID does not exist" Oct 31 13:14:41.909048 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:41.909035 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb"] Oct 31 13:14:41.914008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:41.913968 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-fa8c3f/autores-model-fa8c3f-kserve-6b4c7f544f-wd8wb"] Oct 31 13:14:42.786199 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:42.786163 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" event={"ID":"a8494c61-00b4-41e6-b8ed-f4c2081f8944","Type":"ContainerStarted","Data":"a781f06ced084f339b1e8309d5f678b5d425c791a0aecfb2e04476bbcfca8eee"} Oct 31 13:14:42.841620 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:42.838283 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:43.587006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:43.586823 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:43.669934 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:43.669897 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15523312-bf9c-46ab-ab57-e5ce6d128d9f" path="/var/lib/kubelet/pods/15523312-bf9c-46ab-ab57-e5ce6d128d9f/volumes" Oct 31 13:14:43.766147 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:43.766116 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:14:43.766685 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:43.766650 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 31 13:14:43.766821 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:43.766711 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" podUID="a8494c61-00b4-41e6-b8ed-f4c2081f8944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 31 13:14:44.331689 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:44.331648 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:44.762159 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:44.762117 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 31 13:14:44.762765 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:44.762731 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" podUID="a8494c61-00b4-41e6-b8ed-f4c2081f8944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 31 13:14:45.222688 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:45.222649 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:45.763059 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:45.762373 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 31 13:14:45.763059 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:45.762433 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" podUID="a8494c61-00b4-41e6-b8ed-f4c2081f8944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 31 13:14:46.118173 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:46.118085 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:46.766483 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:46.766442 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 31 13:14:46.767063 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:46.766521 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" podUID="a8494c61-00b4-41e6-b8ed-f4c2081f8944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 31 13:14:46.951121 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:46.950824 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:47.762001 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:47.759534 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 31 13:14:47.762001 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:47.759597 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" podUID="a8494c61-00b4-41e6-b8ed-f4c2081f8944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 31 13:14:47.927265 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:47.927215 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:48.758265 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:48.758222 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 31 13:14:48.758502 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:48.758277 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" podUID="a8494c61-00b4-41e6-b8ed-f4c2081f8944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 31 13:14:49.067098 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:49.062665 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:49.764424 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:49.763370 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:14:49.764424 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:49.764057 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:14:49.764762 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:49.764743 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:14:50.093320 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:14:50.093201 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:14:50.116374 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:50.116341 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:51.079005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:51.077854 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:51.860008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:51.854644 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:52.330884 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:52.330854 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:52.808524 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:52.808487 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:53.434438 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:53.434402 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:53.700711 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:53.700675 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:53.936844 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:53.936811 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:54.168104 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:54.168017 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:54.399583 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:54.399545 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:54.708645 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:54.708609 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:54.939378 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:54.932462 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:55.417032 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:55.416956 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:55.992006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:55.991858 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:57.101977 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:57.101934 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:58.024842 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:58.024751 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:58.842570 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:58.842523 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:59.430055 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:59.429969 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:14:59.869276 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:14:59.869066 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:00.708936 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:00.708893 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:00.984706 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:00.983520 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:01.229082 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:01.229048 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:01.446946 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:01.446441 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": read tcp 10.134.0.2:51488->10.134.0.194:15021: read: connection reset by peer" start-of-body= Oct 31 13:15:01.446946 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:01.446501 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" podUID="42529d86-63e7-47dd-a03d-3e2ca7737a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": read tcp 10.134.0.2:51488->10.134.0.194:15021: read: connection reset by peer" Oct 31 13:15:01.466580 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:01.466541 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:01.732061 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:01.731673 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:02.144294 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:02.144206 6841 generic.go:356] "Generic (PLEG): container finished" podID="cfd61135-1d8a-44a5-9771-a66f1c88cd18" containerID="9a1b6eabb4d08a835841e53610bafb1356c9c10fc97dc39635d66a00bd40bf90" exitCode=0 Oct 31 13:15:02.144774 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:02.144286 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" event={"ID":"cfd61135-1d8a-44a5-9771-a66f1c88cd18","Type":"ContainerDied","Data":"9a1b6eabb4d08a835841e53610bafb1356c9c10fc97dc39635d66a00bd40bf90"} Oct 31 13:15:02.144877 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:02.144840 6841 scope.go:122] "RemoveContainer" containerID="9a1b6eabb4d08a835841e53610bafb1356c9c10fc97dc39635d66a00bd40bf90" Oct 31 13:15:02.146271 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:02.146248 6841 generic.go:356] "Generic (PLEG): container finished" podID="42529d86-63e7-47dd-a03d-3e2ca7737a52" containerID="705f880daff07ec87b8b01f01987326bf511cff6093825ebd6f562d4963788e0" exitCode=0 Oct 31 13:15:02.146369 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:02.146294 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" event={"ID":"42529d86-63e7-47dd-a03d-3e2ca7737a52","Type":"ContainerDied","Data":"705f880daff07ec87b8b01f01987326bf511cff6093825ebd6f562d4963788e0"} Oct 31 13:15:02.146733 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:02.146717 6841 scope.go:122] "RemoveContainer" containerID="705f880daff07ec87b8b01f01987326bf511cff6093825ebd6f562d4963788e0" Oct 31 13:15:02.147104 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:02.147075 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:15:02.147182 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:02.147162 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:15:02.147230 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:02.147215 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:15:02.148520 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:02.148492 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:15:02.148629 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:02.148594 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:15:02.148696 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:02.148646 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:15:02.178753 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:02.177850 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:02.585899 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:02.583259 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:03.215016 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:03.214600 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:03.272474 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:03.269052 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" event={"ID":"42529d86-63e7-47dd-a03d-3e2ca7737a52","Type":"ContainerStarted","Data":"8baeb9c963c3d70611ee309af6bfe62949a32294f521ce2196f83ac19feb28f7"} Oct 31 13:15:03.279678 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:03.279648 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" event={"ID":"cfd61135-1d8a-44a5-9771-a66f1c88cd18","Type":"ContainerStarted","Data":"a82cf6f5f0155063fd87d7d9d075dc4a06b0e77b0b499a8edd985e802abeac57"} Oct 31 13:15:03.601094 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:03.595885 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:03.854066 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:03.832387 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:15:03.881006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:03.855160 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 31 13:15:03.881006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:03.855255 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" podUID="cfd61135-1d8a-44a5-9771-a66f1c88cd18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 31 13:15:04.104035 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:04.103998 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:15:04.104245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:04.104187 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 31 13:15:04.104245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:04.104233 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" podUID="42529d86-63e7-47dd-a03d-3e2ca7737a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 31 13:15:04.731632 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:04.731593 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:04.830213 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:04.829605 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 31 13:15:04.830213 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:04.829685 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" podUID="cfd61135-1d8a-44a5-9771-a66f1c88cd18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 31 13:15:05.105234 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:05.105132 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 31 13:15:05.105234 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:05.105193 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" podUID="42529d86-63e7-47dd-a03d-3e2ca7737a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 31 13:15:05.701302 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:05.701214 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:05.829873 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:05.829835 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 31 13:15:05.830324 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:05.829900 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" podUID="cfd61135-1d8a-44a5-9771-a66f1c88cd18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 31 13:15:06.104515 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:06.104421 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 31 13:15:06.104515 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:06.104489 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" podUID="42529d86-63e7-47dd-a03d-3e2ca7737a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 31 13:15:06.585638 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:06.585600 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:06.830205 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:06.830163 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 31 13:15:06.830671 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:06.830639 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" podUID="cfd61135-1d8a-44a5-9771-a66f1c88cd18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 31 13:15:07.006618 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:07.006579 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:15:07.104524 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:07.104491 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 31 13:15:07.104708 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:07.104545 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" podUID="42529d86-63e7-47dd-a03d-3e2ca7737a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 31 13:15:07.265351 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:07.265253 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:07.835188 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:07.835152 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 31 13:15:07.836555 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:07.836516 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" podUID="cfd61135-1d8a-44a5-9771-a66f1c88cd18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 31 13:15:07.909341 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:07.909308 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:08.106363 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:08.105454 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 31 13:15:08.106363 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:08.105537 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" podUID="42529d86-63e7-47dd-a03d-3e2ca7737a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 31 13:15:08.161006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:08.157874 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:08.500910 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:08.500870 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:08.829458 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:08.829365 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 31 13:15:08.829458 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:08.829428 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" podUID="cfd61135-1d8a-44a5-9771-a66f1c88cd18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 31 13:15:09.105290 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:09.105214 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 31 13:15:09.105896 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:09.105868 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" podUID="42529d86-63e7-47dd-a03d-3e2ca7737a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 31 13:15:09.590628 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:09.590588 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:10.034462 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:10.034428 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:15:10.036247 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:10.036224 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:15:10.111136 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:10.111104 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:15:10.112397 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:10.112374 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:15:10.113381 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:10.113341 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:15:10.548445 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:10.548408 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:11.480925 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:11.470539 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:11.673869 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:11.673827 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:12.231654 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:12.231604 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:13.073852 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:13.073819 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:13.828420 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:13.828386 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:14.529290 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:14.529249 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:15.322837 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:15.322807 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:16.047414 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:16.047380 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:16.539026 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:16.538956 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt"] Oct 31 13:15:16.540279 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:16.540237 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt" podUID="7754c3ef-a2ff-4a52-b371-873053b3d1ad" containerName="maas-api" containerID="cri-o://f3f7e7267e29c7ef71fcfd1bbf39a8c6d44a85bdb49309d7d89aa7931360ae7f" gracePeriod=30 Oct 31 13:15:16.694759 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:16.694726 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:17.215433 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:17.215406 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt" Oct 31 13:15:17.319692 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:17.319662 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7754c3ef-a2ff-4a52-b371-873053b3d1ad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7754c3ef-a2ff-4a52-b371-873053b3d1ad-openshift-service-ca\") pod \"7754c3ef-a2ff-4a52-b371-873053b3d1ad\" (UID: \"7754c3ef-a2ff-4a52-b371-873053b3d1ad\") " Oct 31 13:15:17.319893 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:17.319703 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7754c3ef-a2ff-4a52-b371-873053b3d1ad-kube-api-access-f5k7d\" (UniqueName: \"kubernetes.io/projected/7754c3ef-a2ff-4a52-b371-873053b3d1ad-kube-api-access-f5k7d\") pod \"7754c3ef-a2ff-4a52-b371-873053b3d1ad\" (UID: \"7754c3ef-a2ff-4a52-b371-873053b3d1ad\") " Oct 31 13:15:17.319893 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:17.319758 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7754c3ef-a2ff-4a52-b371-873053b3d1ad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7754c3ef-a2ff-4a52-b371-873053b3d1ad-metrics-certs\") pod \"7754c3ef-a2ff-4a52-b371-873053b3d1ad\" (UID: \"7754c3ef-a2ff-4a52-b371-873053b3d1ad\") " Oct 31 13:15:17.319893 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:17.319789 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7754c3ef-a2ff-4a52-b371-873053b3d1ad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7754c3ef-a2ff-4a52-b371-873053b3d1ad-maas-api-tls\") pod \"7754c3ef-a2ff-4a52-b371-873053b3d1ad\" (UID: \"7754c3ef-a2ff-4a52-b371-873053b3d1ad\") " Oct 31 13:15:17.320317 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:17.320293 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7754c3ef-a2ff-4a52-b371-873053b3d1ad-openshift-service-ca" pod "7754c3ef-a2ff-4a52-b371-873053b3d1ad" (UID: "7754c3ef-a2ff-4a52-b371-873053b3d1ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:17.321172 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:17.321132 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7754c3ef-a2ff-4a52-b371-873053b3d1ad-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:17.323117 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:17.323087 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7754c3ef-a2ff-4a52-b371-873053b3d1ad-kube-api-access-f5k7d" pod "7754c3ef-a2ff-4a52-b371-873053b3d1ad" (UID: "7754c3ef-a2ff-4a52-b371-873053b3d1ad"). InnerVolumeSpecName "kube-api-access-f5k7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:17.323216 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:17.323113 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7754c3ef-a2ff-4a52-b371-873053b3d1ad-metrics-certs" pod "7754c3ef-a2ff-4a52-b371-873053b3d1ad" (UID: "7754c3ef-a2ff-4a52-b371-873053b3d1ad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:15:17.323216 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:17.323170 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7754c3ef-a2ff-4a52-b371-873053b3d1ad-maas-api-tls" pod "7754c3ef-a2ff-4a52-b371-873053b3d1ad" (UID: "7754c3ef-a2ff-4a52-b371-873053b3d1ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:15:17.355422 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:17.355391 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:17.422660 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:17.422628 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5k7d\" (UniqueName: \"kubernetes.io/projected/7754c3ef-a2ff-4a52-b371-873053b3d1ad-kube-api-access-f5k7d\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:17.422660 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:17.422659 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7754c3ef-a2ff-4a52-b371-873053b3d1ad-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:17.422869 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:17.422668 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7754c3ef-a2ff-4a52-b371-873053b3d1ad-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:17.571611 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:17.571578 6841 generic.go:356] "Generic (PLEG): container finished" podID="7754c3ef-a2ff-4a52-b371-873053b3d1ad" containerID="f3f7e7267e29c7ef71fcfd1bbf39a8c6d44a85bdb49309d7d89aa7931360ae7f" exitCode=0 Oct 31 13:15:17.571833 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:17.571614 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt" event={"ID":"7754c3ef-a2ff-4a52-b371-873053b3d1ad","Type":"ContainerDied","Data":"f3f7e7267e29c7ef71fcfd1bbf39a8c6d44a85bdb49309d7d89aa7931360ae7f"} Oct 31 13:15:17.571833 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:17.571640 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt" Oct 31 13:15:17.571833 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:17.571705 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt" event={"ID":"7754c3ef-a2ff-4a52-b371-873053b3d1ad","Type":"ContainerDied","Data":"4a836a63b0c28677e9109a5d123474ae3a247df1afd8eac938af06c326a2491c"} Oct 31 13:15:17.571833 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:17.571730 6841 scope.go:122] "RemoveContainer" containerID="f3f7e7267e29c7ef71fcfd1bbf39a8c6d44a85bdb49309d7d89aa7931360ae7f" Oct 31 13:15:17.585568 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:17.585545 6841 scope.go:122] "RemoveContainer" containerID="f3f7e7267e29c7ef71fcfd1bbf39a8c6d44a85bdb49309d7d89aa7931360ae7f" Oct 31 13:15:17.585840 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:15:17.585823 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3f7e7267e29c7ef71fcfd1bbf39a8c6d44a85bdb49309d7d89aa7931360ae7f\": container with ID starting with f3f7e7267e29c7ef71fcfd1bbf39a8c6d44a85bdb49309d7d89aa7931360ae7f not found: ID does not exist" containerID="f3f7e7267e29c7ef71fcfd1bbf39a8c6d44a85bdb49309d7d89aa7931360ae7f" Oct 31 13:15:17.585906 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:17.585849 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3f7e7267e29c7ef71fcfd1bbf39a8c6d44a85bdb49309d7d89aa7931360ae7f"} err="failed to get container status \"f3f7e7267e29c7ef71fcfd1bbf39a8c6d44a85bdb49309d7d89aa7931360ae7f\": rpc error: code = NotFound desc = could not find container \"f3f7e7267e29c7ef71fcfd1bbf39a8c6d44a85bdb49309d7d89aa7931360ae7f\": container with ID starting with f3f7e7267e29c7ef71fcfd1bbf39a8c6d44a85bdb49309d7d89aa7931360ae7f not found: ID does not exist" Oct 31 13:15:17.610486 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:17.610427 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt"] Oct 31 13:15:17.618437 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:17.618412 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-fa8c3f-6cc678476c-7x5lt"] Oct 31 13:15:17.646375 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:17.646343 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7754c3ef-a2ff-4a52-b371-873053b3d1ad" path="/var/lib/kubelet/pods/7754c3ef-a2ff-4a52-b371-873053b3d1ad/volumes" Oct 31 13:15:18.045011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:18.044545 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:18.893355 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:18.893307 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:19.188303 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:19.188226 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:19.569631 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:19.569598 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:20.141517 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:20.141403 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:20.236415 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:15:20.236381 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:15:20.570357 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:20.570310 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:21.550829 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:21.550703 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:21.898334 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:21.898260 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:22.251218 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.251157 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl"] Oct 31 13:15:22.252140 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.252117 6841 cpu_manager.go:414] "containerMap: removing container" podUID="7754c3ef-a2ff-4a52-b371-873053b3d1ad" containerName="maas-api" Oct 31 13:15:22.252140 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.252140 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7754c3ef-a2ff-4a52-b371-873053b3d1ad" containerName="maas-api" Oct 31 13:15:22.252363 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.252182 6841 cpu_manager.go:414] "containerMap: removing container" podUID="15523312-bf9c-46ab-ab57-e5ce6d128d9f" containerName="main" Oct 31 13:15:22.252363 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.252191 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15523312-bf9c-46ab-ab57-e5ce6d128d9f" containerName="main" Oct 31 13:15:22.252475 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.252371 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="15523312-bf9c-46ab-ab57-e5ce6d128d9f" containerName="main" Oct 31 13:15:22.252475 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.252400 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="7754c3ef-a2ff-4a52-b371-873053b3d1ad" containerName="maas-api" Oct 31 13:15:22.257265 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.257246 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:15:22.282328 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.282295 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl"] Oct 31 13:15:22.328939 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.328906 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-2ef686-openshift-default-dockercfg-mzw2z\"" Oct 31 13:15:22.441533 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.441495 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:22.522253 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.522108 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djfv6\" (UniqueName: \"kubernetes.io/projected/6e2a46f8-aab8-482f-8edb-67800fa4960e-kube-api-access-djfv6\") pod \"e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl\" (UID: \"6e2a46f8-aab8-482f-8edb-67800fa4960e\") " pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:15:22.522530 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.522507 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e2a46f8-aab8-482f-8edb-67800fa4960e-istio-podinfo\") pod \"e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl\" (UID: \"6e2a46f8-aab8-482f-8edb-67800fa4960e\") " pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:15:22.522926 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.522905 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e2a46f8-aab8-482f-8edb-67800fa4960e-istiod-ca-cert\") pod \"e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl\" (UID: \"6e2a46f8-aab8-482f-8edb-67800fa4960e\") " pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:15:22.523177 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.523158 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e2a46f8-aab8-482f-8edb-67800fa4960e-istio-token\") pod \"e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl\" (UID: \"6e2a46f8-aab8-482f-8edb-67800fa4960e\") " pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:15:22.523398 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.523379 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e2a46f8-aab8-482f-8edb-67800fa4960e-workload-certs\") pod \"e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl\" (UID: \"6e2a46f8-aab8-482f-8edb-67800fa4960e\") " pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:15:22.523595 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.523575 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e2a46f8-aab8-482f-8edb-67800fa4960e-istio-envoy\") pod \"e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl\" (UID: \"6e2a46f8-aab8-482f-8edb-67800fa4960e\") " pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:15:22.523852 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.523835 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e2a46f8-aab8-482f-8edb-67800fa4960e-istio-data\") pod \"e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl\" (UID: \"6e2a46f8-aab8-482f-8edb-67800fa4960e\") " pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:15:22.524206 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.524183 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e2a46f8-aab8-482f-8edb-67800fa4960e-workload-socket\") pod \"e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl\" (UID: \"6e2a46f8-aab8-482f-8edb-67800fa4960e\") " pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:15:22.524447 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.524428 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e2a46f8-aab8-482f-8edb-67800fa4960e-credential-socket\") pod \"e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl\" (UID: \"6e2a46f8-aab8-482f-8edb-67800fa4960e\") " pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:15:22.650033 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.635757 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e2a46f8-aab8-482f-8edb-67800fa4960e-istio-data\") pod \"e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl\" (UID: \"6e2a46f8-aab8-482f-8edb-67800fa4960e\") " pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:15:22.650033 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.635938 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e2a46f8-aab8-482f-8edb-67800fa4960e-workload-socket\") pod \"e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl\" (UID: \"6e2a46f8-aab8-482f-8edb-67800fa4960e\") " pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:15:22.650033 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.636071 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e2a46f8-aab8-482f-8edb-67800fa4960e-credential-socket\") pod \"e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl\" (UID: \"6e2a46f8-aab8-482f-8edb-67800fa4960e\") " pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:15:22.650033 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.636154 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djfv6\" (UniqueName: \"kubernetes.io/projected/6e2a46f8-aab8-482f-8edb-67800fa4960e-kube-api-access-djfv6\") pod \"e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl\" (UID: \"6e2a46f8-aab8-482f-8edb-67800fa4960e\") " pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:15:22.650033 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.636196 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e2a46f8-aab8-482f-8edb-67800fa4960e-istio-podinfo\") pod \"e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl\" (UID: \"6e2a46f8-aab8-482f-8edb-67800fa4960e\") " pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:15:22.650033 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.636364 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e2a46f8-aab8-482f-8edb-67800fa4960e-istiod-ca-cert\") pod \"e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl\" (UID: \"6e2a46f8-aab8-482f-8edb-67800fa4960e\") " pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:15:22.650033 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.636420 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e2a46f8-aab8-482f-8edb-67800fa4960e-istio-token\") pod \"e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl\" (UID: \"6e2a46f8-aab8-482f-8edb-67800fa4960e\") " pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:15:22.650033 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.636453 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e2a46f8-aab8-482f-8edb-67800fa4960e-workload-certs\") pod \"e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl\" (UID: \"6e2a46f8-aab8-482f-8edb-67800fa4960e\") " pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:15:22.650033 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.636484 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e2a46f8-aab8-482f-8edb-67800fa4960e-istio-envoy\") pod \"e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl\" (UID: \"6e2a46f8-aab8-482f-8edb-67800fa4960e\") " pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:15:22.652395 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.651921 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e2a46f8-aab8-482f-8edb-67800fa4960e-workload-socket\") pod \"e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl\" (UID: \"6e2a46f8-aab8-482f-8edb-67800fa4960e\") " pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:15:22.652395 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.652210 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e2a46f8-aab8-482f-8edb-67800fa4960e-istio-data\") pod \"e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl\" (UID: \"6e2a46f8-aab8-482f-8edb-67800fa4960e\") " pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:15:22.652589 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.652437 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e2a46f8-aab8-482f-8edb-67800fa4960e-credential-socket\") pod \"e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl\" (UID: \"6e2a46f8-aab8-482f-8edb-67800fa4960e\") " pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:15:22.676015 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.652896 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e2a46f8-aab8-482f-8edb-67800fa4960e-workload-certs\") pod \"e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl\" (UID: \"6e2a46f8-aab8-482f-8edb-67800fa4960e\") " pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:15:22.676408 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.653154 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e2a46f8-aab8-482f-8edb-67800fa4960e-istiod-ca-cert\") pod \"e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl\" (UID: \"6e2a46f8-aab8-482f-8edb-67800fa4960e\") " pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:15:22.681012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.677265 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e2a46f8-aab8-482f-8edb-67800fa4960e-istio-envoy\") pod \"e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl\" (UID: \"6e2a46f8-aab8-482f-8edb-67800fa4960e\") " pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:15:22.681012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.677539 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:22.681012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.680537 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e2a46f8-aab8-482f-8edb-67800fa4960e-istio-podinfo\") pod \"e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl\" (UID: \"6e2a46f8-aab8-482f-8edb-67800fa4960e\") " pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:15:22.718018 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.711950 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djfv6\" (UniqueName: \"kubernetes.io/projected/6e2a46f8-aab8-482f-8edb-67800fa4960e-kube-api-access-djfv6\") pod \"e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl\" (UID: \"6e2a46f8-aab8-482f-8edb-67800fa4960e\") " pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:15:22.738006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.719856 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e2a46f8-aab8-482f-8edb-67800fa4960e-istio-token\") pod \"e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl\" (UID: \"6e2a46f8-aab8-482f-8edb-67800fa4960e\") " pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:15:22.872741 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.872648 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:15:22.904566 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:22.904525 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:23.176359 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:23.176326 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl"] Oct 31 13:15:23.177064 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:15:23.177040 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e2a46f8_aab8_482f_8edb_67800fa4960e.slice/crio-8a1b3987a4d7905568a71973d1d878e67cec57d3366efd086d9de4f815f3ad65 WatchSource:0}: Error finding container 8a1b3987a4d7905568a71973d1d878e67cec57d3366efd086d9de4f815f3ad65: Status 404 returned error can't find the container with id 8a1b3987a4d7905568a71973d1d878e67cec57d3366efd086d9de4f815f3ad65 Oct 31 13:15:23.180093 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:23.180062 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:15:23.180174 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:23.180124 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:15:23.180174 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:23.180152 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:15:23.663927 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:23.663567 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" event={"ID":"6e2a46f8-aab8-482f-8edb-67800fa4960e","Type":"ContainerStarted","Data":"fec8706920e9bd7d9213ad585b2dc602732f62dc3d5afb107b76d2ed2303e143"} Oct 31 13:15:23.663927 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:23.663636 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" event={"ID":"6e2a46f8-aab8-482f-8edb-67800fa4960e","Type":"ContainerStarted","Data":"8a1b3987a4d7905568a71973d1d878e67cec57d3366efd086d9de4f815f3ad65"} Oct 31 13:15:23.690174 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:23.690132 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" podStartSLOduration=1.690119552 podStartE2EDuration="1.690119552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:23.688351033 +0000 UTC m=+2570.171473933" watchObservedRunningTime="2026-10-31 13:15:23.690119552 +0000 UTC m=+2570.173242452" Oct 31 13:15:23.876004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:23.867170 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:23.876326 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:23.876300 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:15:24.878793 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:24.878711 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:15:25.156945 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:25.156846 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:25.679894 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:25.679859 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:15:25.680802 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:25.680780 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:15:26.289169 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:26.289127 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:26.578471 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:26.576652 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:27.674860 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:27.674821 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:28.130576 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:28.130536 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:30.093509 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:30.093474 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:31.136253 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:31.136219 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:31.460082 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:31.460042 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:31.752419 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:31.752336 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:32.330961 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:32.330930 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:32.751355 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:32.751306 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:33.167871 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:33.167792 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:33.439911 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:33.439826 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:34.067132 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:34.067090 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:34.444329 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:34.444301 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:34.759969 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:34.759874 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:35.112069 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:35.078612 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:36.233146 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:36.233110 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:37.056485 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:37.056452 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:37.863839 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:37.863768 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:38.118860 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:38.118780 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:38.727333 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:38.727279 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:39.565177 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:39.565141 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:40.363450 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:40.363413 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:41.245783 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:41.245732 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:41.967005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:41.966023 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:42.733493 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:42.733457 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:43.483955 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:43.483921 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:44.348580 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:44.348546 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:45.161233 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:45.161163 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:45.898153 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:45.898109 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:46.725468 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:46.725407 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:48.083697 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:48.083660 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:48.514205 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:48.514171 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:49.322445 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:49.321764 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:49.987286 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:49.987245 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc"] Oct 31 13:15:49.995197 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:49.995160 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc" Oct 31 13:15:50.007746 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:50.007702 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-2ef686-serving-cert\"" Oct 31 13:15:50.009201 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:50.009173 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc"] Oct 31 13:15:50.021746 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:50.021716 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5khnl\" (UniqueName: \"kubernetes.io/projected/aa093405-83f7-4d3e-b984-bb188e5171d7-kube-api-access-5khnl\") pod \"maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc\" (UID: \"aa093405-83f7-4d3e-b984-bb188e5171d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc" Oct 31 13:15:50.021935 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:50.021913 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa093405-83f7-4d3e-b984-bb188e5171d7-metrics-certs\") pod \"maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc\" (UID: \"aa093405-83f7-4d3e-b984-bb188e5171d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc" Oct 31 13:15:50.022217 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:50.022195 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa093405-83f7-4d3e-b984-bb188e5171d7-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc\" (UID: \"aa093405-83f7-4d3e-b984-bb188e5171d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc" Oct 31 13:15:50.022310 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:50.022237 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa093405-83f7-4d3e-b984-bb188e5171d7-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc\" (UID: \"aa093405-83f7-4d3e-b984-bb188e5171d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc" Oct 31 13:15:50.044762 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:15:50.044682 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:15:50.173203 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:50.156014 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa093405-83f7-4d3e-b984-bb188e5171d7-metrics-certs\") pod \"maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc\" (UID: \"aa093405-83f7-4d3e-b984-bb188e5171d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc" Oct 31 13:15:50.222754 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:50.222578 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa093405-83f7-4d3e-b984-bb188e5171d7-metrics-certs\") pod \"maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc\" (UID: \"aa093405-83f7-4d3e-b984-bb188e5171d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc" Oct 31 13:15:50.225527 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:50.225495 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:50.226488 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:50.226212 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa093405-83f7-4d3e-b984-bb188e5171d7-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc\" (UID: \"aa093405-83f7-4d3e-b984-bb188e5171d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc" Oct 31 13:15:50.227859 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:50.227837 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa093405-83f7-4d3e-b984-bb188e5171d7-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc\" (UID: \"aa093405-83f7-4d3e-b984-bb188e5171d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc" Oct 31 13:15:50.228062 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:50.226837 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa093405-83f7-4d3e-b984-bb188e5171d7-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc\" (UID: \"aa093405-83f7-4d3e-b984-bb188e5171d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc" Oct 31 13:15:50.228164 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:50.228144 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5khnl\" (UniqueName: \"kubernetes.io/projected/aa093405-83f7-4d3e-b984-bb188e5171d7-kube-api-access-5khnl\") pod \"maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc\" (UID: \"aa093405-83f7-4d3e-b984-bb188e5171d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc" Oct 31 13:15:50.231374 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:50.231353 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa093405-83f7-4d3e-b984-bb188e5171d7-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc\" (UID: \"aa093405-83f7-4d3e-b984-bb188e5171d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc" Oct 31 13:15:50.241017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:50.240947 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5khnl\" (UniqueName: \"kubernetes.io/projected/aa093405-83f7-4d3e-b984-bb188e5171d7-kube-api-access-5khnl\") pod \"maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc\" (UID: \"aa093405-83f7-4d3e-b984-bb188e5171d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc" Oct 31 13:15:50.317336 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:50.317312 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc" Oct 31 13:15:50.468712 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:50.468675 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:50.617421 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:15:50.617389 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa093405_83f7_4d3e_b984_bb188e5171d7.slice/crio-7c32c1cd30a5eeacdedd803583bf4ed99e293f18966496116415467de6c076a1 WatchSource:0}: Error finding container 7c32c1cd30a5eeacdedd803583bf4ed99e293f18966496116415467de6c076a1: Status 404 returned error can't find the container with id 7c32c1cd30a5eeacdedd803583bf4ed99e293f18966496116415467de6c076a1 Oct 31 13:15:50.619905 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:50.619886 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc"] Oct 31 13:15:51.006117 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:51.005977 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc" event={"ID":"aa093405-83f7-4d3e-b984-bb188e5171d7","Type":"ContainerStarted","Data":"7c32c1cd30a5eeacdedd803583bf4ed99e293f18966496116415467de6c076a1"} Oct 31 13:15:51.186707 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:51.186668 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:52.015993 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:52.015943 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc" event={"ID":"aa093405-83f7-4d3e-b984-bb188e5171d7","Type":"ContainerStarted","Data":"f79fb0a090f0b78c1b70fd48f5bc621166ea07367abf3b293858afb4a714097b"} Oct 31 13:15:52.016493 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:52.016028 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc" Oct 31 13:15:52.040006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:52.039946 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc" podStartSLOduration=2.318765455 podStartE2EDuration="3.03993383s" totalImagesPullingTime="721.168375ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:52.036723483 +0000 UTC m=+2598.519846396" watchObservedRunningTime="2026-10-31 13:15:52.03993383 +0000 UTC m=+2598.523056730" Oct 31 13:15:52.256434 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:52.253133 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:53.125530 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:53.125493 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:54.001658 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:54.001623 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:54.764219 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:54.764130 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:55.558755 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:55.558720 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:55.813880 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:55.813770 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:56.430369 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:56.430330 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:57.286028 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:57.285936 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:58.086628 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:58.084314 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc" Oct 31 13:15:58.086628 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:58.085951 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:58.834999 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:58.834945 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:15:59.529904 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:15:59.529870 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:00.424004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:00.423828 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:01.261368 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:01.261332 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:02.026673 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:02.026633 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:02.719293 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:02.719242 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:03.631638 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:03.631601 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:04.385158 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:04.385127 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:05.171684 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:05.171654 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:05.922335 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:05.922299 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:06.621005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:06.617531 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:07.349193 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:07.349140 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:08.081274 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:08.081238 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:08.761004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:08.755418 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:09.538363 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:09.538328 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:10.209719 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:10.209684 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:11.282621 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:11.282585 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:12.005019 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:12.002844 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:12.207393 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:12.207354 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:12.807140 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:12.807105 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:13.344473 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:13.344440 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:13.616453 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:13.616380 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh"] Oct 31 13:16:13.620363 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:13.620337 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:13.621556 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:13.621541 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:16:13.626938 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:13.626913 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-5a9e45-openshift-default-dockercfg-szwmr\"" Oct 31 13:16:13.659764 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:13.659733 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh"] Oct 31 13:16:13.682074 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:13.682038 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8f62f2e-7128-4109-af26-c1d6955b83bf-istio-token\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh\" (UID: \"a8f62f2e-7128-4109-af26-c1d6955b83bf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:16:13.682306 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:13.682251 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhpvl\" (UniqueName: \"kubernetes.io/projected/a8f62f2e-7128-4109-af26-c1d6955b83bf-kube-api-access-xhpvl\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh\" (UID: \"a8f62f2e-7128-4109-af26-c1d6955b83bf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:16:13.682386 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:13.682295 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8f62f2e-7128-4109-af26-c1d6955b83bf-istio-envoy\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh\" (UID: \"a8f62f2e-7128-4109-af26-c1d6955b83bf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:16:13.682386 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:13.682367 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f62f2e-7128-4109-af26-c1d6955b83bf-credential-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh\" (UID: \"a8f62f2e-7128-4109-af26-c1d6955b83bf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:16:13.682506 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:13.682434 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8f62f2e-7128-4109-af26-c1d6955b83bf-istio-data\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh\" (UID: \"a8f62f2e-7128-4109-af26-c1d6955b83bf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:16:13.682506 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:13.682481 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8f62f2e-7128-4109-af26-c1d6955b83bf-istiod-ca-cert\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh\" (UID: \"a8f62f2e-7128-4109-af26-c1d6955b83bf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:16:13.682613 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:13.682569 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f62f2e-7128-4109-af26-c1d6955b83bf-workload-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh\" (UID: \"a8f62f2e-7128-4109-af26-c1d6955b83bf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:16:13.682676 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:13.682649 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8f62f2e-7128-4109-af26-c1d6955b83bf-workload-certs\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh\" (UID: \"a8f62f2e-7128-4109-af26-c1d6955b83bf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:16:13.682732 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:13.682671 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8f62f2e-7128-4109-af26-c1d6955b83bf-istio-podinfo\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh\" (UID: \"a8f62f2e-7128-4109-af26-c1d6955b83bf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:16:13.858642 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:13.850352 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhpvl\" (UniqueName: \"kubernetes.io/projected/a8f62f2e-7128-4109-af26-c1d6955b83bf-kube-api-access-xhpvl\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh\" (UID: \"a8f62f2e-7128-4109-af26-c1d6955b83bf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:16:13.858642 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:13.850408 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8f62f2e-7128-4109-af26-c1d6955b83bf-istio-envoy\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh\" (UID: \"a8f62f2e-7128-4109-af26-c1d6955b83bf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:16:13.858642 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:13.850443 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f62f2e-7128-4109-af26-c1d6955b83bf-credential-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh\" (UID: \"a8f62f2e-7128-4109-af26-c1d6955b83bf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:16:13.858642 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:13.850541 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8f62f2e-7128-4109-af26-c1d6955b83bf-istio-data\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh\" (UID: \"a8f62f2e-7128-4109-af26-c1d6955b83bf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:16:13.858642 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:13.850604 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8f62f2e-7128-4109-af26-c1d6955b83bf-istiod-ca-cert\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh\" (UID: \"a8f62f2e-7128-4109-af26-c1d6955b83bf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:16:13.858642 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:13.850727 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f62f2e-7128-4109-af26-c1d6955b83bf-workload-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh\" (UID: \"a8f62f2e-7128-4109-af26-c1d6955b83bf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:16:13.858642 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:13.850831 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8f62f2e-7128-4109-af26-c1d6955b83bf-workload-certs\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh\" (UID: \"a8f62f2e-7128-4109-af26-c1d6955b83bf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:16:13.858642 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:13.850852 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8f62f2e-7128-4109-af26-c1d6955b83bf-istio-podinfo\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh\" (UID: \"a8f62f2e-7128-4109-af26-c1d6955b83bf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:16:13.858642 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:13.851055 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8f62f2e-7128-4109-af26-c1d6955b83bf-istio-token\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh\" (UID: \"a8f62f2e-7128-4109-af26-c1d6955b83bf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:16:13.858642 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:13.853343 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f62f2e-7128-4109-af26-c1d6955b83bf-credential-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh\" (UID: \"a8f62f2e-7128-4109-af26-c1d6955b83bf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:16:13.858642 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:13.853942 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8f62f2e-7128-4109-af26-c1d6955b83bf-istio-data\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh\" (UID: \"a8f62f2e-7128-4109-af26-c1d6955b83bf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:16:13.858642 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:13.854693 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f62f2e-7128-4109-af26-c1d6955b83bf-workload-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh\" (UID: \"a8f62f2e-7128-4109-af26-c1d6955b83bf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:16:13.858642 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:13.855275 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8f62f2e-7128-4109-af26-c1d6955b83bf-istiod-ca-cert\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh\" (UID: \"a8f62f2e-7128-4109-af26-c1d6955b83bf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:16:13.858642 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:13.855813 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8f62f2e-7128-4109-af26-c1d6955b83bf-workload-certs\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh\" (UID: \"a8f62f2e-7128-4109-af26-c1d6955b83bf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:16:13.868830 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:13.868647 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:13.938061 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:13.907903 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8f62f2e-7128-4109-af26-c1d6955b83bf-istio-podinfo\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh\" (UID: \"a8f62f2e-7128-4109-af26-c1d6955b83bf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:16:13.953069 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:13.941931 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8f62f2e-7128-4109-af26-c1d6955b83bf-istio-envoy\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh\" (UID: \"a8f62f2e-7128-4109-af26-c1d6955b83bf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:16:13.953069 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:13.946747 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhpvl\" (UniqueName: \"kubernetes.io/projected/a8f62f2e-7128-4109-af26-c1d6955b83bf-kube-api-access-xhpvl\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh\" (UID: \"a8f62f2e-7128-4109-af26-c1d6955b83bf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:16:13.953069 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:13.947184 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8f62f2e-7128-4109-af26-c1d6955b83bf-istio-token\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh\" (UID: \"a8f62f2e-7128-4109-af26-c1d6955b83bf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:16:13.977016 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:13.976122 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:16:14.083466 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:14.083371 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:14.469005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:14.450523 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:14.478009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:14.476105 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh"] Oct 31 13:16:14.487176 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:14.487138 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:16:14.487281 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:14.487210 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:16:14.487281 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:14.487242 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:16:15.200694 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:15.200661 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:15.257397 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:15.257363 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" event={"ID":"a8f62f2e-7128-4109-af26-c1d6955b83bf","Type":"ContainerStarted","Data":"d90cb5a9db1c02f1985e9a1dd8da4877aaa655169c80036ffea2598c4bd755e8"} Oct 31 13:16:15.257585 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:15.257423 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" event={"ID":"a8f62f2e-7128-4109-af26-c1d6955b83bf","Type":"ContainerStarted","Data":"9ec31381a20a79c08f38f19cde663cfdce95eb6bea80a1e9c8981e8d35963537"} Oct 31 13:16:15.288845 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:15.288792 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" podStartSLOduration=2.288776575 podStartE2EDuration="2.288776575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:16:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:16:15.286741344 +0000 UTC m=+2621.769864246" watchObservedRunningTime="2026-10-31 13:16:15.288776575 +0000 UTC m=+2621.771899475" Oct 31 13:16:16.011606 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:16.011378 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:16:16.033512 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:16.033477 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:16.050647 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:16.050612 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:16:16.266450 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:16.266359 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:16:16.267367 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:16.267346 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:16:17.005004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:17.003285 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:17.753547 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:17.753510 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:17.994683 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:17.994650 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:18.220117 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:18.220023 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:19.077723 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:19.077688 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:19.412054 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:19.411951 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:19.622462 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:19.622412 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:20.036166 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:16:20.036128 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:16:20.128702 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:20.127948 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:20.895141 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:20.895104 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:21.793447 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:21.793410 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:22.737002 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:22.732186 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:23.568004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:23.567042 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:24.311656 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:24.311620 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:24.540805 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:24.540754 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:24.765231 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:24.765194 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:25.553864 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:25.553827 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:26.038055 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:26.038014 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:26.348089 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:26.346161 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:27.753922 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:27.753889 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:28.095431 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:28.085560 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:28.800009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:28.796476 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:29.516784 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:29.516748 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:30.299039 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:30.298721 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:31.167800 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:31.167763 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:31.440770 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:31.440666 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:31.640400 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:31.640370 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:32.238997 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:32.238948 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:33.029764 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:33.029722 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:33.830345 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:33.829858 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:34.733048 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:34.732977 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:35.517263 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:35.517221 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:36.231001 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:36.225842 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:36.885331 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:36.885221 6841 generic.go:356] "Generic (PLEG): container finished" podID="25d7ee65-7958-458c-a202-693bd09893c7" containerID="e838473111f333938c845e3e045d0bcac8a456dbcaa3563781125ba6980845cb" exitCode=0 Oct 31 13:16:36.885331 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:36.885296 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" event={"ID":"25d7ee65-7958-458c-a202-693bd09893c7","Type":"ContainerDied","Data":"e838473111f333938c845e3e045d0bcac8a456dbcaa3563781125ba6980845cb"} Oct 31 13:16:36.885790 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:36.885772 6841 scope.go:122] "RemoveContainer" containerID="e838473111f333938c845e3e045d0bcac8a456dbcaa3563781125ba6980845cb" Oct 31 13:16:36.888029 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:36.887995 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:16:36.888173 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:36.888068 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:16:36.888173 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:36.888103 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:16:37.072052 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:37.070132 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:37.818240 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:37.816945 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:37.907601 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:37.907567 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" event={"ID":"25d7ee65-7958-458c-a202-693bd09893c7","Type":"ContainerStarted","Data":"33207ea1fd72f6e21d98e41f9682d2611d714ace7b906bb3ddce49bb96306f69"} Oct 31 13:16:38.140330 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:38.140241 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:38.311732 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:38.311701 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:16:38.313262 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:38.313232 6841 patch_prober.go:29] interesting pod/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 31 13:16:38.313398 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:38.313294 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" podUID="25d7ee65-7958-458c-a202-693bd09893c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 31 13:16:38.570779 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:38.570741 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:38.607314 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:38.607279 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:16:39.318447 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:39.318404 6841 patch_prober.go:29] interesting pod/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 31 13:16:39.319171 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:39.319138 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" podUID="25d7ee65-7958-458c-a202-693bd09893c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 31 13:16:39.352073 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:39.352038 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:40.080144 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:40.080109 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm"] Oct 31 13:16:40.086566 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:40.086538 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm" Oct 31 13:16:40.088068 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:40.088045 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm"] Oct 31 13:16:40.091614 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:40.091590 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-5a9e45-serving-cert\"" Oct 31 13:16:40.202708 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:40.202667 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/481838f4-af98-4e2e-8f4e-2db1897925c4-metrics-certs\") pod \"maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm\" (UID: \"481838f4-af98-4e2e-8f4e-2db1897925c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm" Oct 31 13:16:40.231999 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:40.231950 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/481838f4-af98-4e2e-8f4e-2db1897925c4-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm\" (UID: \"481838f4-af98-4e2e-8f4e-2db1897925c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm" Oct 31 13:16:40.232487 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:40.232458 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb8kd\" (UniqueName: \"kubernetes.io/projected/481838f4-af98-4e2e-8f4e-2db1897925c4-kube-api-access-xb8kd\") pod \"maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm\" (UID: \"481838f4-af98-4e2e-8f4e-2db1897925c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm" Oct 31 13:16:40.255656 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:40.255617 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/481838f4-af98-4e2e-8f4e-2db1897925c4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm\" (UID: \"481838f4-af98-4e2e-8f4e-2db1897925c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm" Oct 31 13:16:40.256142 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:40.231669 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:40.312138 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:40.312111 6841 patch_prober.go:29] interesting pod/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 31 13:16:40.312390 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:40.312362 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" podUID="25d7ee65-7958-458c-a202-693bd09893c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 31 13:16:40.365051 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:40.364087 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb8kd\" (UniqueName: \"kubernetes.io/projected/481838f4-af98-4e2e-8f4e-2db1897925c4-kube-api-access-xb8kd\") pod \"maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm\" (UID: \"481838f4-af98-4e2e-8f4e-2db1897925c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm" Oct 31 13:16:40.365051 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:40.364293 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/481838f4-af98-4e2e-8f4e-2db1897925c4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm\" (UID: \"481838f4-af98-4e2e-8f4e-2db1897925c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm" Oct 31 13:16:40.365051 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:40.364366 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/481838f4-af98-4e2e-8f4e-2db1897925c4-metrics-certs\") pod \"maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm\" (UID: \"481838f4-af98-4e2e-8f4e-2db1897925c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm" Oct 31 13:16:40.365051 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:40.364417 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/481838f4-af98-4e2e-8f4e-2db1897925c4-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm\" (UID: \"481838f4-af98-4e2e-8f4e-2db1897925c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm" Oct 31 13:16:40.365703 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:40.365424 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/481838f4-af98-4e2e-8f4e-2db1897925c4-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm\" (UID: \"481838f4-af98-4e2e-8f4e-2db1897925c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm" Oct 31 13:16:40.369151 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:40.369123 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/481838f4-af98-4e2e-8f4e-2db1897925c4-metrics-certs\") pod \"maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm\" (UID: \"481838f4-af98-4e2e-8f4e-2db1897925c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm" Oct 31 13:16:40.369466 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:40.369443 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/481838f4-af98-4e2e-8f4e-2db1897925c4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm\" (UID: \"481838f4-af98-4e2e-8f4e-2db1897925c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm" Oct 31 13:16:40.378744 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:40.378706 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb8kd\" (UniqueName: \"kubernetes.io/projected/481838f4-af98-4e2e-8f4e-2db1897925c4-kube-api-access-xb8kd\") pod \"maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm\" (UID: \"481838f4-af98-4e2e-8f4e-2db1897925c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm" Oct 31 13:16:40.402633 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:40.402432 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm" Oct 31 13:16:40.470469 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:40.470425 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:40.725648 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:40.725619 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm"] Oct 31 13:16:41.008147 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:41.007916 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm" event={"ID":"481838f4-af98-4e2e-8f4e-2db1897925c4","Type":"ContainerStarted","Data":"88a86332e00ca4d79baa69250cb4a3c2748965c9ab4a3eb6ebbe395d071f9721"} Oct 31 13:16:41.128519 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:41.128459 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:41.311805 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:41.311779 6841 patch_prober.go:29] interesting pod/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 31 13:16:41.311923 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:41.311837 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" podUID="25d7ee65-7958-458c-a202-693bd09893c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 31 13:16:41.913786 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:41.913752 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:42.017944 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:42.017907 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm" event={"ID":"481838f4-af98-4e2e-8f4e-2db1897925c4","Type":"ContainerStarted","Data":"cc4ac0333c956e5a4b8d57f2f0cedde2b11235a4030515db7bfdbeaeefa1054f"} Oct 31 13:16:42.018139 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:42.017968 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm" Oct 31 13:16:42.040500 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:42.040455 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm" podStartSLOduration=1.5420316349999998 podStartE2EDuration="2.04044149s" totalImagesPullingTime="498.409855ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:16:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:16:42.038946168 +0000 UTC m=+2648.522069065" watchObservedRunningTime="2026-10-31 13:16:42.04044149 +0000 UTC m=+2648.523564391" Oct 31 13:16:42.311551 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:42.311516 6841 patch_prober.go:29] interesting pod/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 31 13:16:42.311738 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:42.311583 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" podUID="25d7ee65-7958-458c-a202-693bd09893c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 31 13:16:42.834682 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:42.833770 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:43.286548 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:43.286515 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:43.311462 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:43.311425 6841 patch_prober.go:29] interesting pod/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 31 13:16:43.311650 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:43.311503 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" podUID="25d7ee65-7958-458c-a202-693bd09893c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 31 13:16:43.594058 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:43.583531 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:44.023546 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:44.023508 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:44.313291 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:44.308958 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:44.337383 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:44.337348 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:16:44.338947 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:44.338922 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:16:45.145471 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:45.130707 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:46.025002 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:46.024734 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:46.770233 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:46.769014 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:47.811496 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:47.811456 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:48.033330 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:48.033301 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm" Oct 31 13:16:48.696456 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:48.696350 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:49.164813 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:49.164706 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:49.855339 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:49.855306 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:50.036508 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:16:50.036478 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:16:50.979891 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:50.979853 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:53.085883 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:53.085851 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:53.298538 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:53.298490 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:53.938131 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:53.938000 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:54.747861 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:54.747822 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:55.209727 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:55.209697 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:55.653052 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:55.652936 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:56.429866 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:56.429832 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:57.220339 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:57.220298 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:58.137204 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:58.137165 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:58.992889 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:58.992842 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:59.229329 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:59.229292 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:59.545649 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:59.545615 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:16:59.959152 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:16:59.959119 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:00.308618 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:00.308487 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:00.743017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:00.742951 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:01.408717 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:01.408683 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:01.626213 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:01.621904 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:01.930004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:01.929052 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:03.039742 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:03.039690 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:03.440242 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:03.440143 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:03.926186 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:03.926146 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:04.737327 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:04.737289 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:05.917154 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:05.912930 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:06.863435 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:06.863398 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:07.655011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:07.651787 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:08.697271 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:08.696789 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:08.901261 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:08.901226 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:09.526536 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:09.526498 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:09.787388 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:09.787307 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:10.034576 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:10.034545 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:10.335719 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:10.335684 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:10.618430 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:10.618344 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:10.878251 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:10.878174 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:11.403015 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:11.402946 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:11.697581 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:11.697548 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:12.284559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:12.284505 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:13.225267 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:13.225230 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:14.142557 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:14.142092 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:14.869635 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:14.869600 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:15.374023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:15.369435 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:15.591242 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:15.591199 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:15.915741 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:15.915707 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:16.695239 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:16.695207 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:17.452476 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:17.452430 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:17.659402 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:17.659365 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:18.335968 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:18.335930 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:18.572471 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:18.572440 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:19.109918 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:19.109879 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:19.334667 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:19.334630 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:20.059592 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:17:20.059504 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:17:20.256645 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:20.256611 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:21.808974 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:21.808878 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:22.640859 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:22.640802 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:23.387976 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:23.387934 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:23.974303 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:23.974263 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:24.392804 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:24.392727 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:24.754723 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:24.754687 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:25.049744 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:25.049655 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:25.292445 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:25.292412 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:26.011005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:26.008925 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:26.434292 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:26.434204 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:26.871012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:26.869363 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:28.041153 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:28.041115 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:28.940502 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:28.940467 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:29.185962 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:29.185928 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:29.427146 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:29.427115 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:29.911037 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:29.910947 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:30.895971 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:30.895935 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:31.846427 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:31.846389 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:32.118435 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:32.118352 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:32.488006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:32.486713 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:33.673335 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:33.673300 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:33.963971 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:33.963943 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:35.060601 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:35.060564 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:35.925082 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:35.925034 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:36.377217 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:36.377177 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:36.683081 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:36.681876 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:36.952434 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:36.952402 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:37.253506 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:37.253241 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:37.474265 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:37.474225 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:37.899271 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:37.899239 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:38.163057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:38.161159 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:38.392960 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:38.392922 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:38.888212 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:38.888167 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:39.360003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:39.355587 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:40.237804 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:40.237701 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:40.512946 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:40.512740 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:40.729172 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:40.729140 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:41.517966 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:41.517932 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:42.419802 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:42.419754 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:43.227123 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:43.227076 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:44.032295 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:44.032192 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:45.094565 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:45.094531 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:45.801133 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:45.801053 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:46.354825 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:46.354785 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:46.938696 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:46.938599 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:47.501750 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:47.501715 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:47.826264 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:47.826165 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:48.409083 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:48.408961 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:49.289076 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:49.289037 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:49.501201 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:49.501166 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:50.041957 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:17:50.041914 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:17:50.224100 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:50.224004 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:50.927383 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:50.927349 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:51.710033 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:51.709998 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:51.720019 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:51.718426 6841 generic.go:356] "Generic (PLEG): container finished" podID="6e2a46f8-aab8-482f-8edb-67800fa4960e" containerID="fec8706920e9bd7d9213ad585b2dc602732f62dc3d5afb107b76d2ed2303e143" exitCode=0 Oct 31 13:17:51.720019 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:51.718820 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" event={"ID":"6e2a46f8-aab8-482f-8edb-67800fa4960e","Type":"ContainerDied","Data":"fec8706920e9bd7d9213ad585b2dc602732f62dc3d5afb107b76d2ed2303e143"} Oct 31 13:17:51.720019 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:51.719410 6841 scope.go:122] "RemoveContainer" containerID="fec8706920e9bd7d9213ad585b2dc602732f62dc3d5afb107b76d2ed2303e143" Oct 31 13:17:51.724034 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:51.723456 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:17:51.724034 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:51.723561 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:17:51.724034 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:51.723613 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:17:52.578121 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:52.578086 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:52.729049 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:52.729016 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" event={"ID":"6e2a46f8-aab8-482f-8edb-67800fa4960e","Type":"ContainerStarted","Data":"87d1573007990af25b8140be7754ad1af13dd956627c21cb9771456fc377791a"} Oct 31 13:17:52.873376 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:52.873288 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:17:52.874830 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:52.874801 6841 patch_prober.go:29] interesting pod/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 31 13:17:52.874977 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:52.874853 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" podUID="6e2a46f8-aab8-482f-8edb-67800fa4960e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 31 13:17:53.366565 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:53.366527 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:53.873635 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:53.873599 6841 patch_prober.go:29] interesting pod/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 31 13:17:53.874063 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:53.873650 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" podUID="6e2a46f8-aab8-482f-8edb-67800fa4960e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 31 13:17:54.141064 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:54.130896 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:54.783842 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:54.783805 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:54.873904 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:54.873864 6841 patch_prober.go:29] interesting pod/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 31 13:17:54.874343 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:54.873928 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" podUID="6e2a46f8-aab8-482f-8edb-67800fa4960e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 31 13:17:55.357136 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:55.357088 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:55.679924 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:55.679829 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:17:55.820633 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:55.820577 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:55.877045 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:55.877002 6841 patch_prober.go:29] interesting pod/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 31 13:17:55.877573 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:55.877067 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" podUID="6e2a46f8-aab8-482f-8edb-67800fa4960e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 31 13:17:56.058002 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:56.057948 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:56.873316 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:56.873279 6841 patch_prober.go:29] interesting pod/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 31 13:17:56.873545 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:56.873337 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" podUID="6e2a46f8-aab8-482f-8edb-67800fa4960e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 31 13:17:57.036225 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:57.036136 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:57.566964 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:57.566928 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:57.796008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:57.787429 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:57.876058 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:57.873166 6841 patch_prober.go:29] interesting pod/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 31 13:17:57.876058 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:57.873262 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" podUID="6e2a46f8-aab8-482f-8edb-67800fa4960e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 31 13:17:58.267660 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:58.267607 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:58.507960 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:58.507921 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:17:58.878447 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:58.878414 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:17:58.879715 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:58.879689 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:17:59.126077 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:17:59.126000 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:00.041574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:00.041539 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:00.859156 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:00.859124 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:01.641685 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:01.641650 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:01.848512 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:01.848466 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:02.061422 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:02.061390 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:02.508159 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:02.508066 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:03.241242 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:03.241207 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:04.118308 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:04.118268 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:04.939275 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:04.939189 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:05.730529 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:05.730471 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:05.993939 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:05.993855 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:06.214960 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:06.214920 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:06.564501 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:06.564468 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:07.352036 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:07.343470 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:08.140054 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:08.139574 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:08.970633 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:08.970596 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:09.744555 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:09.744507 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:10.593082 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:10.592903 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:11.583258 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:11.583225 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:12.320226 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:12.320193 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:13.144947 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:13.144913 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:13.523968 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:13.523925 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:13.848088 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:13.847961 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:14.068359 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:14.068313 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:14.287287 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:14.287250 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:14.856973 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:14.856937 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:15.096046 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:15.096009 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:15.800895 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:15.800864 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:16.058343 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:16.058264 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:16.910399 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:16.910322 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:17.727137 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:17.727094 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:18.552002 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:18.551899 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:19.364973 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:19.364932 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:20.036333 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:18:20.036298 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:18:20.432206 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:20.432107 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:21.780175 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:21.780141 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:22.623670 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:22.623636 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:23.501319 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:23.501285 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:24.323418 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:24.323361 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:25.125101 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:25.125063 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:25.907311 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:25.907275 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:26.718004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:26.717213 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:27.543657 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:27.543607 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:28.305788 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:28.305749 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:29.049218 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:29.049123 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:29.697596 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:29.697562 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:29.975612 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:29.975526 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:30.192870 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:30.192767 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:30.575517 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:30.575483 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:31.104220 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:31.104184 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:31.816278 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:31.816243 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:32.675424 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:32.675392 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:33.491828 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:33.491780 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:34.327805 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:34.327766 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:35.025204 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:35.025152 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:35.263792 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:35.263755 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:35.536638 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:35.536605 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:35.751392 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:35.751354 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:36.038409 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:36.038364 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:36.252498 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:36.252463 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:36.495797 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:36.495757 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:36.720382 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:36.720347 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:37.093279 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:37.093243 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:37.319249 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:37.317670 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:37.813783 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:37.813754 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:38.027717 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:38.027071 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:38.434927 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:38.434891 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:38.760537 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:38.760502 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:39.037600 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:39.037500 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:39.306241 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:39.306140 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:39.621053 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:39.620597 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:39.846083 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:39.846043 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:40.188266 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:40.188074 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:41.237570 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:41.237536 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:42.061814 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:42.061680 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:42.813008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:42.800405 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:43.626023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:43.623911 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:44.477504 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:44.477468 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:45.372685 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:45.372648 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:46.105137 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:46.105104 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:46.352266 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:46.352232 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:46.601558 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:46.601519 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:46.895065 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:46.894838 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:47.195131 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:47.195100 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:47.660959 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:47.660659 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:48.649649 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:48.649618 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:49.573734 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:49.573700 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:50.036254 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:18:50.036218 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:18:50.085426 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:50.085392 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:50.371063 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:50.370330 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:51.196077 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:51.196046 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:52.072531 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:52.072485 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:53.341849 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:53.341816 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:54.280458 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.280377 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:54.299685 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.299649 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg"] Oct 31 13:18:54.320412 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.320342 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" Oct 31 13:18:54.326432 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.326400 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-842ac9-openshift-default-dockercfg-tdrsf\"" Oct 31 13:18:54.327952 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.327918 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg"] Oct 31 13:18:54.489133 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.489098 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-istio-envoy\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg\" (UID: \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" Oct 31 13:18:54.489539 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.489143 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-istio-podinfo\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg\" (UID: \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" Oct 31 13:18:54.489539 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.489227 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-credential-socket\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg\" (UID: \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" Oct 31 13:18:54.489539 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.489283 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-istio-data\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg\" (UID: \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" Oct 31 13:18:54.489539 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.489330 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh874\" (UniqueName: \"kubernetes.io/projected/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-kube-api-access-nh874\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg\" (UID: \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" Oct 31 13:18:54.489698 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.489529 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-workload-socket\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg\" (UID: \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" Oct 31 13:18:54.489698 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.489614 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-workload-certs\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg\" (UID: \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" Oct 31 13:18:54.489768 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.489722 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-istio-token\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg\" (UID: \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" Oct 31 13:18:54.489804 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.489778 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-istiod-ca-cert\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg\" (UID: \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" Oct 31 13:18:54.591788 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.591687 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-workload-certs\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg\" (UID: \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" Oct 31 13:18:54.591976 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.591828 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-istio-token\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg\" (UID: \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" Oct 31 13:18:54.591976 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.591866 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-istiod-ca-cert\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg\" (UID: \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" Oct 31 13:18:54.591976 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.591928 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-istio-envoy\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg\" (UID: \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" Oct 31 13:18:54.591976 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.591965 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-istio-podinfo\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg\" (UID: \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" Oct 31 13:18:54.592260 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.592033 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-credential-socket\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg\" (UID: \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" Oct 31 13:18:54.592260 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.592098 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-istio-data\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg\" (UID: \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" Oct 31 13:18:54.592260 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.592127 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh874\" (UniqueName: \"kubernetes.io/projected/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-kube-api-access-nh874\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg\" (UID: \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" Oct 31 13:18:54.592260 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.592234 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-workload-certs\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg\" (UID: \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" Oct 31 13:18:54.592497 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.592346 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-workload-socket\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg\" (UID: \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" Oct 31 13:18:54.592497 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.592449 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-credential-socket\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg\" (UID: \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" Oct 31 13:18:54.592595 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.592517 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-istio-data\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg\" (UID: \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" Oct 31 13:18:54.592728 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.592706 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-workload-socket\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg\" (UID: \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" Oct 31 13:18:54.592786 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.592761 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-istiod-ca-cert\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg\" (UID: \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" Oct 31 13:18:54.594479 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.594454 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-istio-envoy\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg\" (UID: \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" Oct 31 13:18:54.594799 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.594776 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-istio-podinfo\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg\" (UID: \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" Oct 31 13:18:54.610831 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.610802 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-istio-token\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg\" (UID: \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" Oct 31 13:18:54.611200 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.611179 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh874\" (UniqueName: \"kubernetes.io/projected/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-kube-api-access-nh874\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg\" (UID: \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" Oct 31 13:18:54.655726 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.655689 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" Oct 31 13:18:54.965435 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.965402 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg"] Oct 31 13:18:54.965606 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:18:54.965460 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc74bb4_ddf5_4a5f_836d_e206e6ba6020.slice/crio-e54a0c518c78e081ca73b9ddf69035c4c88b90413701895fae39709fe849ae17 WatchSource:0}: Error finding container e54a0c518c78e081ca73b9ddf69035c4c88b90413701895fae39709fe849ae17: Status 404 returned error can't find the container with id e54a0c518c78e081ca73b9ddf69035c4c88b90413701895fae39709fe849ae17 Oct 31 13:18:54.968654 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.968625 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:18:54.968727 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.968689 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:18:54.968727 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:54.968719 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:18:55.134205 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:55.133904 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:55.164311 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:55.164213 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" event={"ID":"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020","Type":"ContainerStarted","Data":"594b02f4d01b830cf8211dafef85ac2010c3e9dc37a8a37645934bc8c8f7d7c1"} Oct 31 13:18:55.164311 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:55.164276 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" event={"ID":"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020","Type":"ContainerStarted","Data":"e54a0c518c78e081ca73b9ddf69035c4c88b90413701895fae39709fe849ae17"} Oct 31 13:18:55.195544 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:55.195494 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" podStartSLOduration=1.19548087 podStartE2EDuration="1.19548087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:18:55.190385515 +0000 UTC m=+2781.673508418" watchObservedRunningTime="2026-10-31 13:18:55.19548087 +0000 UTC m=+2781.678603771" Oct 31 13:18:55.581007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:55.575189 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:55.656241 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:55.656207 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" Oct 31 13:18:55.816638 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:55.816600 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:56.578008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:56.577053 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:56.661007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:56.660950 6841 patch_prober.go:29] interesting pod/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Oct 31 13:18:56.661513 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:56.661032 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" podUID="bbc74bb4-ddf5-4a5f-836d-e206e6ba6020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Oct 31 13:18:57.688737 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:57.688694 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:57.689395 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:57.689246 6841 patch_prober.go:29] interesting pod/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Oct 31 13:18:57.689395 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:57.689325 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" podUID="bbc74bb4-ddf5-4a5f-836d-e206e6ba6020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Oct 31 13:18:57.913798 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:57.913734 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:58.545792 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:58.545757 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:58.659667 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:58.659628 6841 patch_prober.go:29] interesting pod/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Oct 31 13:18:58.659954 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:58.659922 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" podUID="bbc74bb4-ddf5-4a5f-836d-e206e6ba6020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Oct 31 13:18:59.418660 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:59.418623 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:18:59.671080 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:59.664090 6841 patch_prober.go:29] interesting pod/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Oct 31 13:18:59.671080 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:18:59.664176 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" podUID="bbc74bb4-ddf5-4a5f-836d-e206e6ba6020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Oct 31 13:19:00.665006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:00.657893 6841 patch_prober.go:29] interesting pod/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Oct 31 13:19:00.665006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:00.657969 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" podUID="bbc74bb4-ddf5-4a5f-836d-e206e6ba6020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Oct 31 13:19:00.670923 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:00.667678 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:01.469557 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:01.469453 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:01.668216 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:01.668175 6841 patch_prober.go:29] interesting pod/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Oct 31 13:19:01.668605 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:01.668257 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" podUID="bbc74bb4-ddf5-4a5f-836d-e206e6ba6020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Oct 31 13:19:02.361343 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:02.361301 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:02.662474 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:02.662400 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" Oct 31 13:19:02.662890 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:02.662871 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" Oct 31 13:19:02.663588 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:02.663568 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" Oct 31 13:19:03.172491 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:03.172457 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:04.117502 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:04.117465 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:04.937566 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:04.937521 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:05.296496 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:05.296457 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:05.792738 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:05.792702 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:06.669830 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:06.669770 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:07.533935 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:07.533866 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:08.386913 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:08.386872 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:08.622310 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:08.622279 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:09.741091 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:09.740771 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:10.413005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:10.412391 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:11.229972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:11.229937 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:12.006577 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:12.006539 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:12.870568 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:12.870526 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:13.744040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:13.743816 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:14.536063 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:14.536028 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:15.305963 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:15.305927 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:16.102823 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:16.102732 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:16.800034 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:16.799122 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:17.732958 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:17.732924 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:18.636476 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:18.636422 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:19.453802 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:19.453760 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:20.035802 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:19:20.035765 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:19:20.213008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:20.204583 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:21.031015 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:21.030957 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:21.810802 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:21.810769 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:22.547116 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:22.546252 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:23.218423 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:23.217594 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:23.934326 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:23.934267 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:24.748053 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:24.748015 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:25.710055 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:25.702518 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:26.448870 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:26.448775 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:27.300779 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:27.300579 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:28.137152 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:28.134474 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:28.875282 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:28.875248 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:29.767732 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:29.767701 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:30.652541 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:30.652501 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:31.473495 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:31.473462 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:32.239696 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:32.239659 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:32.965552 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:32.965514 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:33.765927 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:33.765894 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:34.520847 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:34.520803 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:35.234317 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:35.234144 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:36.080005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:36.075027 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:37.251447 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:37.251409 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:38.066032 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:38.065945 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:38.541036 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:38.540964 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:38.849754 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:38.849610 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:40.006129 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:40.005831 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:40.778613 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:40.778577 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:41.968017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:41.967967 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:42.837003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:42.833767 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:43.532952 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:43.532884 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:44.334058 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:44.333962 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:45.129163 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:45.129125 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:45.876552 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:45.876519 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:46.578226 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:46.578188 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:47.362622 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:47.362586 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:48.083275 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:48.083141 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:48.794836 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:48.794742 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:49.582789 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:49.582754 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:50.035777 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:19:50.035742 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:19:50.447758 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:50.444711 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:50.801434 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:50.801401 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:51.013321 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:51.013286 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:51.221593 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:51.221470 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:51.459018 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:51.438444 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:52.026275 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:52.026239 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:52.962037 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:52.959006 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:53.890544 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:53.890508 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:54.747010 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:54.735961 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:55.435666 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:55.435628 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:56.254113 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:56.254071 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:56.958155 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:56.958120 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:57.900363 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:57.900323 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:58.729898 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:58.729867 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:59.137563 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:59.137475 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:59.353651 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:59.353616 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:19:59.585489 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:19:59.585455 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:00.229014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:00.225878 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:01.024328 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:01.024292 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:01.837039 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:01.836956 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:02.627049 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:02.625655 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:03.517619 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:03.517581 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:03.938619 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:03.938539 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:04.245246 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:04.245212 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:04.522539 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:04.522436 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:04.959049 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:04.958928 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:05.703961 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:05.703912 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:05.954188 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:05.954095 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:06.220373 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:06.220286 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:06.531426 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:06.531341 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:06.782703 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:06.782623 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:07.134451 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:07.134358 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:07.452169 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:07.452131 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:08.147797 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:08.147761 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:08.375161 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:08.375119 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:08.739640 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:08.737067 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:09.544797 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:09.544763 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:09.754764 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:09.754729 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:10.394538 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:10.394500 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:11.241147 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:11.241109 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:11.453389 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:11.453354 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:11.662908 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:11.662820 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:12.035713 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:12.035677 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:12.284217 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:12.284181 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:12.529004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:12.528636 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:12.958105 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:12.958067 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:13.508071 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:13.508026 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:13.713386 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:13.713344 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:14.052632 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:14.052597 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:14.268885 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:14.268850 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:14.642503 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:14.642459 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:15.351437 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:15.351402 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:16.134543 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:16.134508 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:16.905401 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:16.905370 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:17.815216 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:17.812552 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:18.520166 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:18.520129 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:18.725444 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:18.725411 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:19.293048 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:19.293008 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:19.628339 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:19.628251 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:19.852010 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:19.845871 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:20.189026 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:20:20.188744 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:20:20.381143 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:20.381107 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:20.673003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:20.672085 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:21.762627 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:21.762596 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:22.045954 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:22.045862 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:22.833005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:22.831877 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:23.583359 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:23.583298 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:24.408626 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:24.408593 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:25.129967 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:25.129933 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:25.837006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:25.833621 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:26.631521 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:26.631489 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:27.347008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:27.344145 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:28.233040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:28.233005 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:28.747859 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:28.747824 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:28.980709 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:28.980672 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:29.234201 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:29.234172 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:29.932100 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:29.932061 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:30.166504 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:30.166472 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:31.009008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:31.002561 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:32.118161 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:32.118127 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:32.937718 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:32.936969 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:33.616191 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:33.616160 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:33.902545 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:33.902436 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:34.117681 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:34.117643 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:34.972400 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:34.972365 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:35.717401 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:35.717366 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:36.418645 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:36.418609 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:37.146547 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:37.146512 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:37.926192 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:37.926150 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:38.607834 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:38.607796 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:38.814147 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:38.814110 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:39.623887 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:39.623826 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:40.362556 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:40.362517 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:41.125142 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:41.124914 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:41.816803 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:41.813509 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:42.701351 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:42.701313 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:43.430658 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:43.430564 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:44.240639 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:44.239518 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:45.227593 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:45.227496 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:45.964357 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:45.963958 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:46.855624 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:46.855585 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:47.727376 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:47.727345 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:48.516252 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:48.516221 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:49.271449 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:49.271403 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:50.030947 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:50.030901 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:50.067133 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:20:50.067101 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:20:50.847339 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:50.847261 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:51.764003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:51.762469 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:52.626013 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:52.623145 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:53.465410 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:53.465360 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:54.258230 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:54.258194 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:55.154614 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:55.154517 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:55.952541 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:55.952505 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:57.691776 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:57.691741 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:57.962350 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:57.962320 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:58.209004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:58.196623 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:58.451835 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:58.451800 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:58.676187 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:58.676145 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:59.153842 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:59.153805 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:59.375138 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:59.375103 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:20:59.958374 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:20:59.958340 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:00.198771 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:00.198734 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:00.437145 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:00.435581 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:01.432254 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:01.432212 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:02.596059 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:02.595974 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:03.777750 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:03.777642 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:04.855900 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:04.855733 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:06.071072 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:06.071038 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:06.914110 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:06.914074 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:07.121331 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:07.121291 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:07.409091 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:07.409053 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:08.749683 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:08.749647 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:08.971847 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:08.971804 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:09.186694 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:09.186605 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:10.145629 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:10.145590 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:10.360005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:10.359864 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:11.516126 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:11.516067 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:12.424006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:12.421035 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:13.129114 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:13.129058 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:13.387215 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:13.387114 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:13.627804 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:13.627767 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:13.837287 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:13.837237 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:14.345435 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:14.345394 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:14.654783 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:14.651793 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:14.893866 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:14.893829 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:15.513880 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:15.513827 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:16.140696 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:16.140651 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:16.376048 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:16.373798 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:17.335172 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:17.335137 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:18.075042 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:18.074957 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:18.935312 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:18.935274 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:19.744447 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:19.744411 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:20.036175 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:21:20.036093 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:21:20.651635 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:20.651602 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:21.057498 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:21.057429 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:21.272879 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:21.272751 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:21.503035 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:21.502975 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:22.083306 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:22.083256 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:23.232717 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:23.232672 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:23.477651 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:23.477604 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:23.695911 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:23.695879 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:23.904180 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:23.904136 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:24.160130 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:24.160048 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:24.511241 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:24.511209 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:24.735116 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:24.735026 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:25.278582 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:25.278549 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:25.543177 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:25.543091 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:26.406665 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:26.406628 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:27.209003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:27.201739 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:27.828964 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:27.828926 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:28.530934 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:28.530866 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:29.320421 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:29.320385 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:30.144310 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:30.144279 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:31.008505 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:31.008466 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:31.775612 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:31.775570 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:32.623311 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:32.623269 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:33.335565 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:33.335350 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:34.043743 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:34.043655 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:34.893341 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:34.893302 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:35.579397 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:35.579362 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd"] Oct 31 13:21:35.584970 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:35.584945 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" Oct 31 13:21:35.619873 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:35.619832 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd"] Oct 31 13:21:35.720004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:35.712295 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-s84vd\" (UID: \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" Oct 31 13:21:35.720004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:35.712415 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-s84vd\" (UID: \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" Oct 31 13:21:35.720004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:35.712608 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-s84vd\" (UID: \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" Oct 31 13:21:35.720004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:35.712675 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-s84vd\" (UID: \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" Oct 31 13:21:35.720004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:35.712715 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-s84vd\" (UID: \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" Oct 31 13:21:35.720004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:35.712882 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-s84vd\" (UID: \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" Oct 31 13:21:35.720004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:35.713077 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbn6s\" (UniqueName: \"kubernetes.io/projected/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-kube-api-access-rbn6s\") pod \"maas-default-gateway-openshift-default-fc8c56547-s84vd\" (UID: \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" Oct 31 13:21:35.720004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:35.713200 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-s84vd\" (UID: \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" Oct 31 13:21:35.720004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:35.713233 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-s84vd\" (UID: \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" Oct 31 13:21:35.758314 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:35.757651 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:35.815150 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:35.815028 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-s84vd\" (UID: \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" Oct 31 13:21:35.815150 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:35.815072 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-s84vd\" (UID: \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" Oct 31 13:21:35.815503 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:35.815211 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-s84vd\" (UID: \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" Oct 31 13:21:35.815503 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:35.815268 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbn6s\" (UniqueName: \"kubernetes.io/projected/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-kube-api-access-rbn6s\") pod \"maas-default-gateway-openshift-default-fc8c56547-s84vd\" (UID: \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" Oct 31 13:21:35.815503 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:35.815380 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-s84vd\" (UID: \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" Oct 31 13:21:35.815503 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:35.815409 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-s84vd\" (UID: \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" Oct 31 13:21:35.815503 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:35.815478 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-s84vd\" (UID: \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" Oct 31 13:21:35.815503 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:35.815505 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-s84vd\" (UID: \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" Oct 31 13:21:35.815860 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:35.815575 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-s84vd\" (UID: \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" Oct 31 13:21:35.815860 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:35.815681 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-s84vd\" (UID: \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" Oct 31 13:21:35.815860 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:35.815819 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-s84vd\" (UID: \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" Oct 31 13:21:35.816026 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:35.815875 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-s84vd\" (UID: \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" Oct 31 13:21:35.816026 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:35.815950 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-s84vd\" (UID: \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" Oct 31 13:21:35.816473 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:35.816435 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-s84vd\" (UID: \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" Oct 31 13:21:35.817769 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:35.817739 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-s84vd\" (UID: \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" Oct 31 13:21:35.818030 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:35.818008 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-s84vd\" (UID: \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" Oct 31 13:21:35.828379 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:35.828354 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbn6s\" (UniqueName: \"kubernetes.io/projected/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-kube-api-access-rbn6s\") pod \"maas-default-gateway-openshift-default-fc8c56547-s84vd\" (UID: \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" Oct 31 13:21:35.828634 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:35.828617 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-s84vd\" (UID: \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" Oct 31 13:21:35.905951 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:35.905873 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" Oct 31 13:21:36.214593 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:21:36.214556 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd90d4670_3be8_4966_b1b5_cbe007b9a8ff.slice/crio-7e3692154f93d7159df81e17079d98680aa4b40d3de2fb4c61e3172486535e88 WatchSource:0}: Error finding container 7e3692154f93d7159df81e17079d98680aa4b40d3de2fb4c61e3172486535e88: Status 404 returned error can't find the container with id 7e3692154f93d7159df81e17079d98680aa4b40d3de2fb4c61e3172486535e88 Oct 31 13:21:36.215457 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:36.215430 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd"] Oct 31 13:21:36.217687 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:36.217656 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:21:36.217781 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:36.217720 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:21:36.217781 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:36.217752 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:21:36.430644 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:36.430601 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" event={"ID":"d90d4670-3be8-4966-b1b5-cbe007b9a8ff","Type":"ContainerStarted","Data":"dc45023f3a87b31fa665dc43c1b003b845817952e88faa5d1c1701fdd259ec7e"} Oct 31 13:21:36.430844 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:36.430658 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" event={"ID":"d90d4670-3be8-4966-b1b5-cbe007b9a8ff","Type":"ContainerStarted","Data":"7e3692154f93d7159df81e17079d98680aa4b40d3de2fb4c61e3172486535e88"} Oct 31 13:21:36.457158 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:36.457100 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" podStartSLOduration=1.457086265 podStartE2EDuration="1.457086265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:21:36.453233809 +0000 UTC m=+2942.936356712" watchObservedRunningTime="2026-10-31 13:21:36.457086265 +0000 UTC m=+2942.940209166" Oct 31 13:21:36.542709 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:36.542627 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:36.906074 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:36.905997 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" Oct 31 13:21:37.469187 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:37.469061 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:37.908436 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:37.908349 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-s84vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 31 13:21:37.908888 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:37.908423 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" podUID="d90d4670-3be8-4966-b1b5-cbe007b9a8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 31 13:21:38.412373 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:38.412338 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:38.738242 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:38.738205 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:38.907725 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:38.907685 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-s84vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 31 13:21:38.907920 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:38.907746 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" podUID="d90d4670-3be8-4966-b1b5-cbe007b9a8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 31 13:21:39.232264 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:39.232215 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:39.907340 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:39.907289 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-s84vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 31 13:21:39.907529 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:39.907358 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" podUID="d90d4670-3be8-4966-b1b5-cbe007b9a8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 31 13:21:40.155701 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:40.155665 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:40.910013 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:40.907457 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-s84vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 31 13:21:40.910013 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:40.907530 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" podUID="d90d4670-3be8-4966-b1b5-cbe007b9a8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 31 13:21:40.942054 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:40.939114 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:41.906236 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:41.906202 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-s84vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 31 13:21:41.906467 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:41.906257 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" podUID="d90d4670-3be8-4966-b1b5-cbe007b9a8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 31 13:21:41.924545 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:41.924512 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:42.204393 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:42.204356 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg"] Oct 31 13:21:42.204624 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:42.204597 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" podUID="0fe9fa73-5f1d-47ea-b303-869b3392ac75" containerName="istio-proxy" containerID="cri-o://8f45944cbcfdd44d257c64794984a7e215891df1dfaba1655c0ecb284ea16ac6" gracePeriod=30 Oct 31 13:21:42.814819 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:42.814765 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:42.906848 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:42.906812 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-s84vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 31 13:21:42.907072 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:42.906867 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" podUID="d90d4670-3be8-4966-b1b5-cbe007b9a8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 31 13:21:43.527006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:43.522664 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:43.781098 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:43.781005 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:43.906228 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:43.906190 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-s84vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 31 13:21:43.906408 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:43.906257 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" podUID="d90d4670-3be8-4966-b1b5-cbe007b9a8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 31 13:21:44.192613 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:44.192221 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:44.500700 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:44.500608 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:44.732370 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:44.732277 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:45.073971 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:45.073922 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:45.632003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:45.631957 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:45.910907 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:45.910823 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" Oct 31 13:21:45.911393 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:45.911360 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" Oct 31 13:21:45.913720 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:45.913694 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" Oct 31 13:21:46.515209 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:46.515176 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:47.542452 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.538700 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:47.601903 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.601873 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:21:47.642293 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.642258 6841 generic.go:356] "Generic (PLEG): container finished" podID="0fe9fa73-5f1d-47ea-b303-869b3392ac75" containerID="8f45944cbcfdd44d257c64794984a7e215891df1dfaba1655c0ecb284ea16ac6" exitCode=0 Oct 31 13:21:47.642496 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.642382 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" Oct 31 13:21:47.650956 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.650923 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" event={"ID":"0fe9fa73-5f1d-47ea-b303-869b3392ac75","Type":"ContainerDied","Data":"8f45944cbcfdd44d257c64794984a7e215891df1dfaba1655c0ecb284ea16ac6"} Oct 31 13:21:47.651247 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.651040 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg" event={"ID":"0fe9fa73-5f1d-47ea-b303-869b3392ac75","Type":"ContainerDied","Data":"7bfd179b550088fdd872d1e00b447d5732b71beaf80e9e8a8341ecabadb3f609"} Oct 31 13:21:47.651247 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.651081 6841 scope.go:122] "RemoveContainer" containerID="8f45944cbcfdd44d257c64794984a7e215891df1dfaba1655c0ecb284ea16ac6" Oct 31 13:21:47.657296 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.657270 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe9fa73-5f1d-47ea-b303-869b3392ac75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe9fa73-5f1d-47ea-b303-869b3392ac75-workload-socket\") pod \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\" (UID: \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\") " Oct 31 13:21:47.657439 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.657320 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0fe9fa73-5f1d-47ea-b303-869b3392ac75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fe9fa73-5f1d-47ea-b303-869b3392ac75-istio-podinfo\") pod \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\" (UID: \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\") " Oct 31 13:21:47.657439 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.657352 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe9fa73-5f1d-47ea-b303-869b3392ac75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fe9fa73-5f1d-47ea-b303-869b3392ac75-istio-envoy\") pod \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\" (UID: \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\") " Oct 31 13:21:47.657439 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.657393 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fe9fa73-5f1d-47ea-b303-869b3392ac75-kube-api-access-nnc27\" (UniqueName: \"kubernetes.io/projected/0fe9fa73-5f1d-47ea-b303-869b3392ac75-kube-api-access-nnc27\") pod \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\" (UID: \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\") " Oct 31 13:21:47.657595 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.657565 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe9fa73-5f1d-47ea-b303-869b3392ac75-workload-socket" pod "0fe9fa73-5f1d-47ea-b303-869b3392ac75" (UID: "0fe9fa73-5f1d-47ea-b303-869b3392ac75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:21:47.657595 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.657589 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fe9fa73-5f1d-47ea-b303-869b3392ac75-istio-token\" (UniqueName: \"kubernetes.io/projected/0fe9fa73-5f1d-47ea-b303-869b3392ac75-istio-token\") pod \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\" (UID: \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\") " Oct 31 13:21:47.657709 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.657694 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe9fa73-5f1d-47ea-b303-869b3392ac75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fe9fa73-5f1d-47ea-b303-869b3392ac75-workload-certs\") pod \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\" (UID: \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\") " Oct 31 13:21:47.658023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.657961 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe9fa73-5f1d-47ea-b303-869b3392ac75-workload-certs" pod "0fe9fa73-5f1d-47ea-b303-869b3392ac75" (UID: "0fe9fa73-5f1d-47ea-b303-869b3392ac75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:21:47.658131 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.658115 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe9fa73-5f1d-47ea-b303-869b3392ac75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fe9fa73-5f1d-47ea-b303-869b3392ac75-istio-data\") pod \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\" (UID: \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\") " Oct 31 13:21:47.658245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.658207 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fe9fa73-5f1d-47ea-b303-869b3392ac75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fe9fa73-5f1d-47ea-b303-869b3392ac75-istiod-ca-cert\") pod \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\" (UID: \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\") " Oct 31 13:21:47.658329 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.658256 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe9fa73-5f1d-47ea-b303-869b3392ac75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe9fa73-5f1d-47ea-b303-869b3392ac75-credential-socket\") pod \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\" (UID: \"0fe9fa73-5f1d-47ea-b303-869b3392ac75\") " Oct 31 13:21:47.658962 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.658652 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe9fa73-5f1d-47ea-b303-869b3392ac75-istio-data" pod "0fe9fa73-5f1d-47ea-b303-869b3392ac75" (UID: "0fe9fa73-5f1d-47ea-b303-869b3392ac75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:21:47.658962 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.658933 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fe9fa73-5f1d-47ea-b303-869b3392ac75-istiod-ca-cert" pod "0fe9fa73-5f1d-47ea-b303-869b3392ac75" (UID: "0fe9fa73-5f1d-47ea-b303-869b3392ac75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:21:47.659433 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.659388 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe9fa73-5f1d-47ea-b303-869b3392ac75-credential-socket" pod "0fe9fa73-5f1d-47ea-b303-869b3392ac75" (UID: "0fe9fa73-5f1d-47ea-b303-869b3392ac75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:21:47.661664 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.661642 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fe9fa73-5f1d-47ea-b303-869b3392ac75-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:47.661798 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.661783 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fe9fa73-5f1d-47ea-b303-869b3392ac75-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:47.661895 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.661884 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fe9fa73-5f1d-47ea-b303-869b3392ac75-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:47.662161 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.662147 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe9fa73-5f1d-47ea-b303-869b3392ac75-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:47.662297 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.662285 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe9fa73-5f1d-47ea-b303-869b3392ac75-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:47.662412 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.662108 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe9fa73-5f1d-47ea-b303-869b3392ac75-kube-api-access-nnc27" pod "0fe9fa73-5f1d-47ea-b303-869b3392ac75" (UID: "0fe9fa73-5f1d-47ea-b303-869b3392ac75"). InnerVolumeSpecName "kube-api-access-nnc27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:21:47.663450 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.663142 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe9fa73-5f1d-47ea-b303-869b3392ac75-istio-envoy" pod "0fe9fa73-5f1d-47ea-b303-869b3392ac75" (UID: "0fe9fa73-5f1d-47ea-b303-869b3392ac75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:21:47.666800 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.665794 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe9fa73-5f1d-47ea-b303-869b3392ac75-istio-token" pod "0fe9fa73-5f1d-47ea-b303-869b3392ac75" (UID: "0fe9fa73-5f1d-47ea-b303-869b3392ac75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:21:47.668417 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.668377 6841 scope.go:122] "RemoveContainer" containerID="e5ad7fee3046d5d4b15fd5814289d3b69f57aba3304c813816f26f282ea2c55d" Oct 31 13:21:47.676468 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.676422 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0fe9fa73-5f1d-47ea-b303-869b3392ac75-istio-podinfo" pod "0fe9fa73-5f1d-47ea-b303-869b3392ac75" (UID: "0fe9fa73-5f1d-47ea-b303-869b3392ac75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:21:47.703507 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.703475 6841 scope.go:122] "RemoveContainer" containerID="8f45944cbcfdd44d257c64794984a7e215891df1dfaba1655c0ecb284ea16ac6" Oct 31 13:21:47.703898 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:21:47.703864 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f45944cbcfdd44d257c64794984a7e215891df1dfaba1655c0ecb284ea16ac6\": container with ID starting with 8f45944cbcfdd44d257c64794984a7e215891df1dfaba1655c0ecb284ea16ac6 not found: ID does not exist" containerID="8f45944cbcfdd44d257c64794984a7e215891df1dfaba1655c0ecb284ea16ac6" Oct 31 13:21:47.704015 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.703911 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f45944cbcfdd44d257c64794984a7e215891df1dfaba1655c0ecb284ea16ac6"} err="failed to get container status \"8f45944cbcfdd44d257c64794984a7e215891df1dfaba1655c0ecb284ea16ac6\": rpc error: code = NotFound desc = could not find container \"8f45944cbcfdd44d257c64794984a7e215891df1dfaba1655c0ecb284ea16ac6\": container with ID starting with 8f45944cbcfdd44d257c64794984a7e215891df1dfaba1655c0ecb284ea16ac6 not found: ID does not exist" Oct 31 13:21:47.704015 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.703939 6841 scope.go:122] "RemoveContainer" containerID="e5ad7fee3046d5d4b15fd5814289d3b69f57aba3304c813816f26f282ea2c55d" Oct 31 13:21:47.704304 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:21:47.704281 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5ad7fee3046d5d4b15fd5814289d3b69f57aba3304c813816f26f282ea2c55d\": container with ID starting with e5ad7fee3046d5d4b15fd5814289d3b69f57aba3304c813816f26f282ea2c55d not found: ID does not exist" containerID="e5ad7fee3046d5d4b15fd5814289d3b69f57aba3304c813816f26f282ea2c55d" Oct 31 13:21:47.704395 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.704315 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5ad7fee3046d5d4b15fd5814289d3b69f57aba3304c813816f26f282ea2c55d"} err="failed to get container status \"e5ad7fee3046d5d4b15fd5814289d3b69f57aba3304c813816f26f282ea2c55d\": rpc error: code = NotFound desc = could not find container \"e5ad7fee3046d5d4b15fd5814289d3b69f57aba3304c813816f26f282ea2c55d\": container with ID starting with e5ad7fee3046d5d4b15fd5814289d3b69f57aba3304c813816f26f282ea2c55d not found: ID does not exist" Oct 31 13:21:47.765223 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.765194 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fe9fa73-5f1d-47ea-b303-869b3392ac75-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:47.765223 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.765222 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fe9fa73-5f1d-47ea-b303-869b3392ac75-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:47.765223 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.765234 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fe9fa73-5f1d-47ea-b303-869b3392ac75-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:47.765478 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.765259 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnc27\" (UniqueName: \"kubernetes.io/projected/0fe9fa73-5f1d-47ea-b303-869b3392ac75-kube-api-access-nnc27\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:47.972854 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.972820 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg"] Oct 31 13:21:47.977094 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:47.977069 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-7h5xg"] Oct 31 13:21:48.233538 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:48.228442 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:48.977003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:48.975273 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:49.644350 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:49.644320 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fe9fa73-5f1d-47ea-b303-869b3392ac75" path="/var/lib/kubelet/pods/0fe9fa73-5f1d-47ea-b303-869b3392ac75/volumes" Oct 31 13:21:49.823725 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:49.823668 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:50.036460 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:21:50.036414 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:21:50.562192 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:50.559858 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:51.456012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:51.455031 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:51.970813 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:51.970781 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:52.209735 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:52.209696 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:52.469659 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:52.469627 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:53.022216 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:53.022177 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:54.308319 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:54.308274 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:54.824175 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:54.824120 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:55.718682 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:55.718648 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:56.420034 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:56.419994 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:57.281682 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:57.281644 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:58.061294 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:58.061257 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:58.912756 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:58.912708 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:59.137396 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:59.137346 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:59.711756 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:59.711721 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:21:59.933959 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:21:59.933921 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:00.160062 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:00.159916 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:00.889722 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:00.889682 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:01.596721 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:01.592976 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:02.335119 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:02.335083 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:03.151462 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:03.151428 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:04.023188 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:04.023153 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:05.227015 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:05.226958 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:06.254017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:06.244134 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:07.574009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:07.567703 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:08.899818 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:08.899761 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:09.738002 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:09.737875 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:10.688464 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:10.688432 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:11.425002 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:11.424954 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:12.226005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:12.225718 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:13.025090 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:13.024539 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:13.727582 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:13.727537 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:14.487634 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:14.487594 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:15.326251 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:15.326218 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:16.083129 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:16.082528 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:16.872008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:16.866904 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:17.735940 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:17.735908 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:18.561279 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:18.561244 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:19.324583 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:19.324472 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:20.058019 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:20.057885 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:20.084283 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:22:20.084248 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:22:20.914286 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:20.914250 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:21.715534 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:21.715500 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:22.612070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:22.610717 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:23.418518 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:23.418482 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:24.225140 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:24.225106 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:25.236017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:25.234602 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:26.159869 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:26.159828 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:27.047007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:27.046094 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fk2tx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:22:27.047007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:27.046186 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" podUID="aa7e0a48-9dc5-4735-b1ae-cafd28ff982f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:22:27.213068 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:27.210921 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:27.928445 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:27.928414 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:28.166778 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:28.166742 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:29.034685 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:29.034642 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:29.760711 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:29.760672 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:30.479916 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:30.479879 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:31.520641 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:31.520566 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:32.327582 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:32.327545 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:32.562839 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:32.562587 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:33.400818 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:33.400788 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:33.619008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:33.618521 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:34.146278 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:34.146243 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:34.606752 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:34.606708 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:35.450023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:35.448551 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:36.310023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:36.305135 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:37.021532 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:37.021499 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:37.737747 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:37.737705 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:38.530160 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:38.530126 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:39.040925 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:39.040885 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:39.323651 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:39.323298 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:40.055547 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:40.055516 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:40.841612 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:40.841579 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:41.355489 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:41.355429 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:41.586341 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:41.586294 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:41.852164 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:41.852130 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:42.583387 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:42.583352 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:43.341397 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:43.341362 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:44.100005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:44.099955 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:44.829494 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:44.829461 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:45.798138 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:45.798095 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:46.847840 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:46.847791 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:47.749078 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:47.749044 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:48.441629 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:48.441595 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:48.993240 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:48.992852 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:49.282078 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:49.281549 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:49.519686 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:49.519652 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:49.883696 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:49.883659 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:50.036080 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:22:50.036046 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:22:50.574591 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:50.574550 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:50.805090 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:50.805012 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:51.652029 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:51.651964 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:51.986502 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:51.986451 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:52.287080 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:52.278536 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:53.310629 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:53.310595 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:54.188818 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:54.188758 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:54.924219 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:54.924168 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:55.821357 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:55.821263 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:56.603354 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:56.603321 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:57.265884 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:57.265827 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:57.468782 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:57.468740 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:57.492654 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:57.492621 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-93e2ca/e2e-usage-sim-e84206-869bc5dff9-2qgqm"] Oct 31 13:22:57.493688 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:57.493663 6841 cpu_manager.go:414] "containerMap: removing container" podUID="0fe9fa73-5f1d-47ea-b303-869b3392ac75" containerName="istio-proxy" Oct 31 13:22:57.493688 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:57.493686 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fe9fa73-5f1d-47ea-b303-869b3392ac75" containerName="istio-proxy" Oct 31 13:22:57.493888 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:57.493713 6841 cpu_manager.go:414] "containerMap: removing container" podUID="0fe9fa73-5f1d-47ea-b303-869b3392ac75" containerName="istio-proxy" Oct 31 13:22:57.493888 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:57.493719 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fe9fa73-5f1d-47ea-b303-869b3392ac75" containerName="istio-proxy" Oct 31 13:22:57.493991 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:57.493953 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fe9fa73-5f1d-47ea-b303-869b3392ac75" containerName="istio-proxy" Oct 31 13:22:57.498285 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:57.498266 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-93e2ca/e2e-usage-sim-e84206-869bc5dff9-2qgqm" Oct 31 13:22:57.505214 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:57.505171 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-93e2ca\"/\"default-dockercfg-blkb5\"" Oct 31 13:22:57.505214 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:57.505171 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-93e2ca\"/\"openshift-service-ca.crt\"" Oct 31 13:22:57.505214 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:57.505202 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-93e2ca\"/\"e2e-usage-sim-e84206-cert\"" Oct 31 13:22:57.505465 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:57.505264 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-93e2ca\"/\"kube-root-ca.crt\"" Oct 31 13:22:57.505828 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:57.505808 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-93e2ca/e2e-usage-sim-e84206-869bc5dff9-2qgqm"] Oct 31 13:22:57.586459 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:57.586383 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp8fb\" (UniqueName: \"kubernetes.io/projected/e2f041ef-d532-4840-936f-895135d81e19-kube-api-access-mp8fb\") pod \"e2e-usage-sim-e84206-869bc5dff9-2qgqm\" (UID: \"e2f041ef-d532-4840-936f-895135d81e19\") " pod="e2e-extmod-shr-user-93e2ca/e2e-usage-sim-e84206-869bc5dff9-2qgqm" Oct 31 13:22:57.586611 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:57.586517 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e2f041ef-d532-4840-936f-895135d81e19-tls\") pod \"e2e-usage-sim-e84206-869bc5dff9-2qgqm\" (UID: \"e2f041ef-d532-4840-936f-895135d81e19\") " pod="e2e-extmod-shr-user-93e2ca/e2e-usage-sim-e84206-869bc5dff9-2qgqm" Oct 31 13:22:57.687684 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:57.687650 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp8fb\" (UniqueName: \"kubernetes.io/projected/e2f041ef-d532-4840-936f-895135d81e19-kube-api-access-mp8fb\") pod \"e2e-usage-sim-e84206-869bc5dff9-2qgqm\" (UID: \"e2f041ef-d532-4840-936f-895135d81e19\") " pod="e2e-extmod-shr-user-93e2ca/e2e-usage-sim-e84206-869bc5dff9-2qgqm" Oct 31 13:22:57.688140 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:57.687761 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e2f041ef-d532-4840-936f-895135d81e19-tls\") pod \"e2e-usage-sim-e84206-869bc5dff9-2qgqm\" (UID: \"e2f041ef-d532-4840-936f-895135d81e19\") " pod="e2e-extmod-shr-user-93e2ca/e2e-usage-sim-e84206-869bc5dff9-2qgqm" Oct 31 13:22:57.690420 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:57.690395 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e2f041ef-d532-4840-936f-895135d81e19-tls\") pod \"e2e-usage-sim-e84206-869bc5dff9-2qgqm\" (UID: \"e2f041ef-d532-4840-936f-895135d81e19\") " pod="e2e-extmod-shr-user-93e2ca/e2e-usage-sim-e84206-869bc5dff9-2qgqm" Oct 31 13:22:57.703200 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:57.703172 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp8fb\" (UniqueName: \"kubernetes.io/projected/e2f041ef-d532-4840-936f-895135d81e19-kube-api-access-mp8fb\") pod \"e2e-usage-sim-e84206-869bc5dff9-2qgqm\" (UID: \"e2f041ef-d532-4840-936f-895135d81e19\") " pod="e2e-extmod-shr-user-93e2ca/e2e-usage-sim-e84206-869bc5dff9-2qgqm" Oct 31 13:22:57.813581 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:57.813536 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-93e2ca/e2e-usage-sim-e84206-869bc5dff9-2qgqm" Oct 31 13:22:58.103454 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:58.103423 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-93e2ca/e2e-usage-sim-e84206-869bc5dff9-2qgqm"] Oct 31 13:22:58.103846 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:22:58.103822 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2f041ef_d532_4840_936f_895135d81e19.slice/crio-17b297fa001833a572d06790442adf03309e0915eb6bc6a8dd88f8b0b69e6e4b WatchSource:0}: Error finding container 17b297fa001833a572d06790442adf03309e0915eb6bc6a8dd88f8b0b69e6e4b: Status 404 returned error can't find the container with id 17b297fa001833a572d06790442adf03309e0915eb6bc6a8dd88f8b0b69e6e4b Oct 31 13:22:58.252755 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:58.252653 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:59.086558 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:59.086506 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-93e2ca/e2e-usage-sim-e84206-869bc5dff9-2qgqm" event={"ID":"e2f041ef-d532-4840-936f-895135d81e19","Type":"ContainerStarted","Data":"5a5f866245fca7a38642bec2b963c458fd0b786f266ee7f9888ea3c40373bf7e"} Oct 31 13:22:59.087051 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:59.086567 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-93e2ca/e2e-usage-sim-e84206-869bc5dff9-2qgqm" event={"ID":"e2f041ef-d532-4840-936f-895135d81e19","Type":"ContainerStarted","Data":"17b297fa001833a572d06790442adf03309e0915eb6bc6a8dd88f8b0b69e6e4b"} Oct 31 13:22:59.087051 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:59.086587 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-93e2ca/e2e-usage-sim-e84206-869bc5dff9-2qgqm" Oct 31 13:22:59.091428 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:59.091401 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-93e2ca/e2e-usage-sim-e84206-869bc5dff9-2qgqm" Oct 31 13:22:59.092745 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:59.092721 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:22:59.110875 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:59.110826 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-93e2ca/e2e-usage-sim-e84206-869bc5dff9-2qgqm" podStartSLOduration=2.110814679 podStartE2EDuration="2.110814679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:22:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:22:59.108119058 +0000 UTC m=+3025.591241958" watchObservedRunningTime="2026-10-31 13:22:59.110814679 +0000 UTC m=+3025.593937580" Oct 31 13:22:59.850803 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:22:59.850759 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:00.578707 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:00.578669 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:01.320079 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:01.320043 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:02.046765 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:02.046675 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:02.863006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:02.862944 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:03.534072 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:03.534042 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:03.757006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:03.756324 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:04.988968 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:04.988927 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:05.757004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:05.756066 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:06.523662 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:06.523617 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:07.228005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:07.227232 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:08.147206 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:08.147176 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:09.050014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:09.047462 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:09.925333 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:09.925296 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:10.711853 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:10.711814 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:11.433725 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:11.433687 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:12.176796 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:12.176721 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:12.911029 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:12.910946 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:13.675199 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:13.675161 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:14.502774 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:14.502741 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:15.298334 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:15.296743 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:16.059298 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:16.059260 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:16.926443 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:16.926403 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:17.689632 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:17.686789 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:18.442325 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:18.442292 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:19.326241 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:19.326123 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:20.036752 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:23:20.036716 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:23:20.205821 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:20.205724 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:20.933655 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:20.933620 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:22.132528 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:22.132491 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:23.362013 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:23.345601 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:24.444394 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:24.444356 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:25.585957 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:25.585818 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:26.853226 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:26.853191 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:27.907113 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:27.907044 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:28.962547 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:28.962511 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:29.179348 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:29.179307 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:29.401813 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:29.401726 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:29.788648 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:29.788612 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:30.278483 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:30.277502 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:31.053594 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:31.053554 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:31.725004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:31.706575 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:32.518524 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:32.518488 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:33.234843 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:33.234224 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:33.533883 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:33.533797 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:34.183639 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:34.183558 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:34.851798 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:34.851758 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:35.725156 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:35.725117 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:36.521682 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:36.521575 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:37.217845 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:37.217812 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:37.976514 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:37.976427 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:38.878224 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:38.878185 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:39.086598 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:39.086519 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:39.310003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:39.308246 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:39.602197 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:39.602078 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:39.807380 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:39.807345 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:40.056007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:40.051233 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:40.350809 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:40.350727 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:41.137727 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:41.137690 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:41.374178 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:41.374098 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:41.628116 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:41.628082 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:41.918064 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:41.910928 6841 generic.go:356] "Generic (PLEG): container finished" podID="a8f62f2e-7128-4109-af26-c1d6955b83bf" containerID="d90cb5a9db1c02f1985e9a1dd8da4877aaa655169c80036ffea2598c4bd755e8" exitCode=0 Oct 31 13:23:41.918064 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:41.911100 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" event={"ID":"a8f62f2e-7128-4109-af26-c1d6955b83bf","Type":"ContainerDied","Data":"d90cb5a9db1c02f1985e9a1dd8da4877aaa655169c80036ffea2598c4bd755e8"} Oct 31 13:23:41.918064 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:41.912090 6841 scope.go:122] "RemoveContainer" containerID="d90cb5a9db1c02f1985e9a1dd8da4877aaa655169c80036ffea2598c4bd755e8" Oct 31 13:23:41.930024 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:41.928845 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:23:41.930024 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:41.928973 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:23:41.930024 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:41.929051 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:23:42.071394 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:42.070554 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:42.307586 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:42.307550 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:42.524357 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:42.524271 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:42.735005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:42.734469 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:42.998139 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:42.997329 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" event={"ID":"a8f62f2e-7128-4109-af26-c1d6955b83bf","Type":"ContainerStarted","Data":"80d722ca5500ec6b228191657f45924adc6a807c1bc0f77d0d3754736a97bbaa"} Oct 31 13:23:43.455626 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:43.455583 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:43.995596 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:43.995243 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:23:43.997469 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:43.996662 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 31 13:23:43.998334 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:43.997492 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" podUID="a8f62f2e-7128-4109-af26-c1d6955b83bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 31 13:23:44.057899 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:44.057862 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:44.422060 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:44.416958 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:45.012073 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:45.012034 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 31 13:23:45.036694 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:45.036634 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" podUID="a8f62f2e-7128-4109-af26-c1d6955b83bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 31 13:23:45.070730 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:45.060509 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:45.270909 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:45.270817 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:45.489398 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:45.489367 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:45.756006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:45.755888 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:45.991376 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:45.991170 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 31 13:23:45.991628 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:45.991595 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" podUID="a8f62f2e-7128-4109-af26-c1d6955b83bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 31 13:23:46.018066 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:46.017957 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:46.271202 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:46.271087 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:23:46.467638 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:46.467603 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:46.991275 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:46.991239 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 31 13:23:46.991462 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:46.991292 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" podUID="a8f62f2e-7128-4109-af26-c1d6955b83bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 31 13:23:47.327430 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:47.327348 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:47.562330 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:47.556689 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:47.990415 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:47.990385 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 31 13:23:47.990718 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:47.990657 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" podUID="a8f62f2e-7128-4109-af26-c1d6955b83bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 31 13:23:48.417964 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:48.417868 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:48.994348 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:48.990504 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 31 13:23:48.994348 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:48.990584 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" podUID="a8f62f2e-7128-4109-af26-c1d6955b83bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 31 13:23:49.336323 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:49.336230 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:50.038618 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:23:50.038580 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:23:50.087805 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:50.087771 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:23:50.089099 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:50.089073 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:23:50.238623 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:50.238581 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:51.124676 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:51.124632 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:52.019641 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:52.019603 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:52.439528 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:52.439437 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:52.638242 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:52.638199 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:52.696264 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:52.696229 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h"] Oct 31 13:23:52.696508 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:52.696484 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" podUID="2e3d6506-3350-4dd0-bdf1-efb951353ba5" containerName="istio-proxy" containerID="cri-o://96d3e66485c0b36a610ac6bcdba9a79e8d7b6fc9fb5b0d16d6a1ae50290dc257" gracePeriod=30 Oct 31 13:23:52.851866 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:52.850716 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:53.102598 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:53.102524 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:53.650039 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:53.649802 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:54.110787 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:54.110747 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:54.321133 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:54.321073 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:54.829490 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:54.828600 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:55.162528 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:55.162109 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v"] Oct 31 13:23:55.162711 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:55.162549 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" podUID="2975edb7-e9cc-4793-acac-baa30b53613c" containerName="istio-proxy" containerID="cri-o://46cf4036f285105781d6f462426889bdd0ea24f5562afb22cbf3cf61a9276f95" gracePeriod=30 Oct 31 13:23:55.710351 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:55.710315 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:56.716879 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:56.716841 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:57.357142 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:57.357099 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:57.597573 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:57.593707 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:57.817651 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:57.817622 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:58.002110 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.002081 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" Oct 31 13:23:58.149640 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.149603 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e3d6506-3350-4dd0-bdf1-efb951353ba5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e3d6506-3350-4dd0-bdf1-efb951353ba5-workload-certs\") pod \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\" (UID: \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\") " Oct 31 13:23:58.149818 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.149659 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e3d6506-3350-4dd0-bdf1-efb951353ba5-istio-token\" (UniqueName: \"kubernetes.io/projected/2e3d6506-3350-4dd0-bdf1-efb951353ba5-istio-token\") pod \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\" (UID: \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\") " Oct 31 13:23:58.149818 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.149717 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e3d6506-3350-4dd0-bdf1-efb951353ba5-kube-api-access-q9kcl\" (UniqueName: \"kubernetes.io/projected/2e3d6506-3350-4dd0-bdf1-efb951353ba5-kube-api-access-q9kcl\") pod \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\" (UID: \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\") " Oct 31 13:23:58.149818 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.149810 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e3d6506-3350-4dd0-bdf1-efb951353ba5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3d6506-3350-4dd0-bdf1-efb951353ba5-credential-socket\") pod \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\" (UID: \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\") " Oct 31 13:23:58.149953 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.149838 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3d6506-3350-4dd0-bdf1-efb951353ba5-workload-certs" pod "2e3d6506-3350-4dd0-bdf1-efb951353ba5" (UID: "2e3d6506-3350-4dd0-bdf1-efb951353ba5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:23:58.149953 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.149912 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e3d6506-3350-4dd0-bdf1-efb951353ba5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e3d6506-3350-4dd0-bdf1-efb951353ba5-istiod-ca-cert\") pod \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\" (UID: \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\") " Oct 31 13:23:58.150058 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.149965 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e3d6506-3350-4dd0-bdf1-efb951353ba5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3d6506-3350-4dd0-bdf1-efb951353ba5-workload-socket\") pod \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\" (UID: \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\") " Oct 31 13:23:58.150106 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.150073 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3d6506-3350-4dd0-bdf1-efb951353ba5-credential-socket" pod "2e3d6506-3350-4dd0-bdf1-efb951353ba5" (UID: "2e3d6506-3350-4dd0-bdf1-efb951353ba5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:23:58.150106 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.150090 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e3d6506-3350-4dd0-bdf1-efb951353ba5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e3d6506-3350-4dd0-bdf1-efb951353ba5-istio-podinfo\") pod \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\" (UID: \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\") " Oct 31 13:23:58.150231 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.150124 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e3d6506-3350-4dd0-bdf1-efb951353ba5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e3d6506-3350-4dd0-bdf1-efb951353ba5-istio-envoy\") pod \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\" (UID: \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\") " Oct 31 13:23:58.150339 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.150311 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3d6506-3350-4dd0-bdf1-efb951353ba5-workload-socket" pod "2e3d6506-3350-4dd0-bdf1-efb951353ba5" (UID: "2e3d6506-3350-4dd0-bdf1-efb951353ba5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:23:58.150414 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.150363 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e3d6506-3350-4dd0-bdf1-efb951353ba5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e3d6506-3350-4dd0-bdf1-efb951353ba5-istio-data\") pod \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\" (UID: \"2e3d6506-3350-4dd0-bdf1-efb951353ba5\") " Oct 31 13:23:58.150414 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.150379 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e3d6506-3350-4dd0-bdf1-efb951353ba5-istiod-ca-cert" pod "2e3d6506-3350-4dd0-bdf1-efb951353ba5" (UID: "2e3d6506-3350-4dd0-bdf1-efb951353ba5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:23:58.150674 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.150652 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3d6506-3350-4dd0-bdf1-efb951353ba5-istio-data" pod "2e3d6506-3350-4dd0-bdf1-efb951353ba5" (UID: "2e3d6506-3350-4dd0-bdf1-efb951353ba5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:23:58.151170 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.151146 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e3d6506-3350-4dd0-bdf1-efb951353ba5-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:58.151170 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.151169 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3d6506-3350-4dd0-bdf1-efb951353ba5-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:58.151336 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.151183 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e3d6506-3350-4dd0-bdf1-efb951353ba5-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:58.151336 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.151196 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3d6506-3350-4dd0-bdf1-efb951353ba5-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:58.151336 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.151208 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e3d6506-3350-4dd0-bdf1-efb951353ba5-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:58.152375 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.152337 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e3d6506-3350-4dd0-bdf1-efb951353ba5-istio-token" pod "2e3d6506-3350-4dd0-bdf1-efb951353ba5" (UID: "2e3d6506-3350-4dd0-bdf1-efb951353ba5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:23:58.152726 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.152701 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e3d6506-3350-4dd0-bdf1-efb951353ba5-kube-api-access-q9kcl" pod "2e3d6506-3350-4dd0-bdf1-efb951353ba5" (UID: "2e3d6506-3350-4dd0-bdf1-efb951353ba5"). InnerVolumeSpecName "kube-api-access-q9kcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:23:58.152922 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.152886 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3d6506-3350-4dd0-bdf1-efb951353ba5-istio-envoy" pod "2e3d6506-3350-4dd0-bdf1-efb951353ba5" (UID: "2e3d6506-3350-4dd0-bdf1-efb951353ba5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:23:58.153107 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.153084 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e3d6506-3350-4dd0-bdf1-efb951353ba5-istio-podinfo" pod "2e3d6506-3350-4dd0-bdf1-efb951353ba5" (UID: "2e3d6506-3350-4dd0-bdf1-efb951353ba5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:23:58.252429 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.252355 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e3d6506-3350-4dd0-bdf1-efb951353ba5-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:58.252429 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.252385 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9kcl\" (UniqueName: \"kubernetes.io/projected/2e3d6506-3350-4dd0-bdf1-efb951353ba5-kube-api-access-q9kcl\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:58.252429 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.252395 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e3d6506-3350-4dd0-bdf1-efb951353ba5-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:58.252429 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.252406 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e3d6506-3350-4dd0-bdf1-efb951353ba5-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:58.454860 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.454824 6841 generic.go:356] "Generic (PLEG): container finished" podID="2e3d6506-3350-4dd0-bdf1-efb951353ba5" containerID="96d3e66485c0b36a610ac6bcdba9a79e8d7b6fc9fb5b0d16d6a1ae50290dc257" exitCode=0 Oct 31 13:23:58.455055 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.454872 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" Oct 31 13:23:58.455055 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.454906 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" event={"ID":"2e3d6506-3350-4dd0-bdf1-efb951353ba5","Type":"ContainerDied","Data":"96d3e66485c0b36a610ac6bcdba9a79e8d7b6fc9fb5b0d16d6a1ae50290dc257"} Oct 31 13:23:58.455055 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.454960 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h" event={"ID":"2e3d6506-3350-4dd0-bdf1-efb951353ba5","Type":"ContainerDied","Data":"8cc8fdb7d1216d879075fcde4fe03b078a267fab744b2ca5717a3eb60d903a08"} Oct 31 13:23:58.455055 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.454994 6841 scope.go:122] "RemoveContainer" containerID="96d3e66485c0b36a610ac6bcdba9a79e8d7b6fc9fb5b0d16d6a1ae50290dc257" Oct 31 13:23:58.467938 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.467914 6841 scope.go:122] "RemoveContainer" containerID="96d3e66485c0b36a610ac6bcdba9a79e8d7b6fc9fb5b0d16d6a1ae50290dc257" Oct 31 13:23:58.468243 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:23:58.468223 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96d3e66485c0b36a610ac6bcdba9a79e8d7b6fc9fb5b0d16d6a1ae50290dc257\": container with ID starting with 96d3e66485c0b36a610ac6bcdba9a79e8d7b6fc9fb5b0d16d6a1ae50290dc257 not found: ID does not exist" containerID="96d3e66485c0b36a610ac6bcdba9a79e8d7b6fc9fb5b0d16d6a1ae50290dc257" Oct 31 13:23:58.468299 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.468254 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96d3e66485c0b36a610ac6bcdba9a79e8d7b6fc9fb5b0d16d6a1ae50290dc257"} err="failed to get container status \"96d3e66485c0b36a610ac6bcdba9a79e8d7b6fc9fb5b0d16d6a1ae50290dc257\": rpc error: code = NotFound desc = could not find container \"96d3e66485c0b36a610ac6bcdba9a79e8d7b6fc9fb5b0d16d6a1ae50290dc257\": container with ID starting with 96d3e66485c0b36a610ac6bcdba9a79e8d7b6fc9fb5b0d16d6a1ae50290dc257 not found: ID does not exist" Oct 31 13:23:58.499221 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.499193 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h"] Oct 31 13:23:58.526352 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.526284 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7wd2h"] Oct 31 13:23:58.704157 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:58.704108 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:59.088002 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:59.084428 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:23:59.644046 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:59.644014 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e3d6506-3350-4dd0-bdf1-efb951353ba5" path="/var/lib/kubelet/pods/2e3d6506-3350-4dd0-bdf1-efb951353ba5/volumes" Oct 31 13:23:59.798648 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:23:59.798614 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:00.452428 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.452400 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 13:24:00.473723 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.473690 6841 generic.go:356] "Generic (PLEG): container finished" podID="2975edb7-e9cc-4793-acac-baa30b53613c" containerID="46cf4036f285105781d6f462426889bdd0ea24f5562afb22cbf3cf61a9276f95" exitCode=0 Oct 31 13:24:00.473888 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.473743 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" Oct 31 13:24:00.473888 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.473768 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" event={"ID":"2975edb7-e9cc-4793-acac-baa30b53613c","Type":"ContainerDied","Data":"46cf4036f285105781d6f462426889bdd0ea24f5562afb22cbf3cf61a9276f95"} Oct 31 13:24:00.473888 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.473821 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v" event={"ID":"2975edb7-e9cc-4793-acac-baa30b53613c","Type":"ContainerDied","Data":"14ef2df8dc3d761b1bc376fb59614dc958c4ed94dd062a83f6e4f9a5deab082e"} Oct 31 13:24:00.473888 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.473845 6841 scope.go:122] "RemoveContainer" containerID="46cf4036f285105781d6f462426889bdd0ea24f5562afb22cbf3cf61a9276f95" Oct 31 13:24:00.488389 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.488370 6841 scope.go:122] "RemoveContainer" containerID="0541c2e1998aca3a1d4d3ca605532a1e2e984bcfa0bd581f092d11f2fe0e4c3b" Oct 31 13:24:00.503550 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.503521 6841 scope.go:122] "RemoveContainer" containerID="46cf4036f285105781d6f462426889bdd0ea24f5562afb22cbf3cf61a9276f95" Oct 31 13:24:00.507152 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:24:00.507116 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46cf4036f285105781d6f462426889bdd0ea24f5562afb22cbf3cf61a9276f95\": container with ID starting with 46cf4036f285105781d6f462426889bdd0ea24f5562afb22cbf3cf61a9276f95 not found: ID does not exist" containerID="46cf4036f285105781d6f462426889bdd0ea24f5562afb22cbf3cf61a9276f95" Oct 31 13:24:00.507299 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.507162 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46cf4036f285105781d6f462426889bdd0ea24f5562afb22cbf3cf61a9276f95"} err="failed to get container status \"46cf4036f285105781d6f462426889bdd0ea24f5562afb22cbf3cf61a9276f95\": rpc error: code = NotFound desc = could not find container \"46cf4036f285105781d6f462426889bdd0ea24f5562afb22cbf3cf61a9276f95\": container with ID starting with 46cf4036f285105781d6f462426889bdd0ea24f5562afb22cbf3cf61a9276f95 not found: ID does not exist" Oct 31 13:24:00.507299 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.507191 6841 scope.go:122] "RemoveContainer" containerID="0541c2e1998aca3a1d4d3ca605532a1e2e984bcfa0bd581f092d11f2fe0e4c3b" Oct 31 13:24:00.507526 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:24:00.507496 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0541c2e1998aca3a1d4d3ca605532a1e2e984bcfa0bd581f092d11f2fe0e4c3b\": container with ID starting with 0541c2e1998aca3a1d4d3ca605532a1e2e984bcfa0bd581f092d11f2fe0e4c3b not found: ID does not exist" containerID="0541c2e1998aca3a1d4d3ca605532a1e2e984bcfa0bd581f092d11f2fe0e4c3b" Oct 31 13:24:00.507617 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.507534 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0541c2e1998aca3a1d4d3ca605532a1e2e984bcfa0bd581f092d11f2fe0e4c3b"} err="failed to get container status \"0541c2e1998aca3a1d4d3ca605532a1e2e984bcfa0bd581f092d11f2fe0e4c3b\": rpc error: code = NotFound desc = could not find container \"0541c2e1998aca3a1d4d3ca605532a1e2e984bcfa0bd581f092d11f2fe0e4c3b\": container with ID starting with 0541c2e1998aca3a1d4d3ca605532a1e2e984bcfa0bd581f092d11f2fe0e4c3b not found: ID does not exist" Oct 31 13:24:00.573441 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.573409 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2975edb7-e9cc-4793-acac-baa30b53613c-istio-token\" (UniqueName: \"kubernetes.io/projected/2975edb7-e9cc-4793-acac-baa30b53613c-istio-token\") pod \"2975edb7-e9cc-4793-acac-baa30b53613c\" (UID: \"2975edb7-e9cc-4793-acac-baa30b53613c\") " Oct 31 13:24:00.573639 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.573476 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2975edb7-e9cc-4793-acac-baa30b53613c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2975edb7-e9cc-4793-acac-baa30b53613c-istio-data\") pod \"2975edb7-e9cc-4793-acac-baa30b53613c\" (UID: \"2975edb7-e9cc-4793-acac-baa30b53613c\") " Oct 31 13:24:00.573639 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.573557 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2975edb7-e9cc-4793-acac-baa30b53613c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2975edb7-e9cc-4793-acac-baa30b53613c-workload-certs\") pod \"2975edb7-e9cc-4793-acac-baa30b53613c\" (UID: \"2975edb7-e9cc-4793-acac-baa30b53613c\") " Oct 31 13:24:00.573639 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.573602 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2975edb7-e9cc-4793-acac-baa30b53613c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2975edb7-e9cc-4793-acac-baa30b53613c-istio-podinfo\") pod \"2975edb7-e9cc-4793-acac-baa30b53613c\" (UID: \"2975edb7-e9cc-4793-acac-baa30b53613c\") " Oct 31 13:24:00.573837 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.573723 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2975edb7-e9cc-4793-acac-baa30b53613c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2975edb7-e9cc-4793-acac-baa30b53613c-istio-envoy\") pod \"2975edb7-e9cc-4793-acac-baa30b53613c\" (UID: \"2975edb7-e9cc-4793-acac-baa30b53613c\") " Oct 31 13:24:00.573837 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.573797 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2975edb7-e9cc-4793-acac-baa30b53613c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2975edb7-e9cc-4793-acac-baa30b53613c-istiod-ca-cert\") pod \"2975edb7-e9cc-4793-acac-baa30b53613c\" (UID: \"2975edb7-e9cc-4793-acac-baa30b53613c\") " Oct 31 13:24:00.573961 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.573877 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2975edb7-e9cc-4793-acac-baa30b53613c-kube-api-access-4dnhj\" (UniqueName: \"kubernetes.io/projected/2975edb7-e9cc-4793-acac-baa30b53613c-kube-api-access-4dnhj\") pod \"2975edb7-e9cc-4793-acac-baa30b53613c\" (UID: \"2975edb7-e9cc-4793-acac-baa30b53613c\") " Oct 31 13:24:00.573961 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.573792 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2975edb7-e9cc-4793-acac-baa30b53613c-istio-data" pod "2975edb7-e9cc-4793-acac-baa30b53613c" (UID: "2975edb7-e9cc-4793-acac-baa30b53613c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:24:00.574100 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.573852 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2975edb7-e9cc-4793-acac-baa30b53613c-workload-certs" pod "2975edb7-e9cc-4793-acac-baa30b53613c" (UID: "2975edb7-e9cc-4793-acac-baa30b53613c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:24:00.574100 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.573955 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2975edb7-e9cc-4793-acac-baa30b53613c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2975edb7-e9cc-4793-acac-baa30b53613c-workload-socket\") pod \"2975edb7-e9cc-4793-acac-baa30b53613c\" (UID: \"2975edb7-e9cc-4793-acac-baa30b53613c\") " Oct 31 13:24:00.574214 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.574139 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2975edb7-e9cc-4793-acac-baa30b53613c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2975edb7-e9cc-4793-acac-baa30b53613c-credential-socket\") pod \"2975edb7-e9cc-4793-acac-baa30b53613c\" (UID: \"2975edb7-e9cc-4793-acac-baa30b53613c\") " Oct 31 13:24:00.574272 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.574143 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2975edb7-e9cc-4793-acac-baa30b53613c-workload-socket" pod "2975edb7-e9cc-4793-acac-baa30b53613c" (UID: "2975edb7-e9cc-4793-acac-baa30b53613c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:24:00.574388 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.574174 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2975edb7-e9cc-4793-acac-baa30b53613c-istiod-ca-cert" pod "2975edb7-e9cc-4793-acac-baa30b53613c" (UID: "2975edb7-e9cc-4793-acac-baa30b53613c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:24:00.574470 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.574344 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2975edb7-e9cc-4793-acac-baa30b53613c-credential-socket" pod "2975edb7-e9cc-4793-acac-baa30b53613c" (UID: "2975edb7-e9cc-4793-acac-baa30b53613c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:24:00.575189 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.574975 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2975edb7-e9cc-4793-acac-baa30b53613c-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:24:00.575314 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.575196 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2975edb7-e9cc-4793-acac-baa30b53613c-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:24:00.575314 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.575215 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2975edb7-e9cc-4793-acac-baa30b53613c-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:24:00.575314 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.575230 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2975edb7-e9cc-4793-acac-baa30b53613c-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:24:00.575314 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.575244 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2975edb7-e9cc-4793-acac-baa30b53613c-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:24:00.576062 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.576036 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2975edb7-e9cc-4793-acac-baa30b53613c-istio-token" pod "2975edb7-e9cc-4793-acac-baa30b53613c" (UID: "2975edb7-e9cc-4793-acac-baa30b53613c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:24:00.576184 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.576147 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2975edb7-e9cc-4793-acac-baa30b53613c-istio-podinfo" pod "2975edb7-e9cc-4793-acac-baa30b53613c" (UID: "2975edb7-e9cc-4793-acac-baa30b53613c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:24:00.576590 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.576569 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2975edb7-e9cc-4793-acac-baa30b53613c-istio-envoy" pod "2975edb7-e9cc-4793-acac-baa30b53613c" (UID: "2975edb7-e9cc-4793-acac-baa30b53613c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:24:00.576656 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.576588 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2975edb7-e9cc-4793-acac-baa30b53613c-kube-api-access-4dnhj" pod "2975edb7-e9cc-4793-acac-baa30b53613c" (UID: "2975edb7-e9cc-4793-acac-baa30b53613c"). InnerVolumeSpecName "kube-api-access-4dnhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:24:00.686006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.681703 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2975edb7-e9cc-4793-acac-baa30b53613c-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:24:00.686006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.681748 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2975edb7-e9cc-4793-acac-baa30b53613c-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:24:00.686006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.681765 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2975edb7-e9cc-4793-acac-baa30b53613c-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:24:00.686006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.681780 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dnhj\" (UniqueName: \"kubernetes.io/projected/2975edb7-e9cc-4793-acac-baa30b53613c-kube-api-access-4dnhj\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:24:00.701438 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.701405 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:00.804072 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.804036 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v"] Oct 31 13:24:00.807892 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:00.807868 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-7296v"] Oct 31 13:24:01.406112 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:01.406075 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:01.644211 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:01.644176 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2975edb7-e9cc-4793-acac-baa30b53613c" path="/var/lib/kubelet/pods/2975edb7-e9cc-4793-acac-baa30b53613c/volumes" Oct 31 13:24:02.122520 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:02.122441 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:02.184337 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:02.184296 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk"] Oct 31 13:24:02.184538 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:02.184514 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" podUID="4f4b3219-9e15-407f-9715-9865e01ea8f7" containerName="istio-proxy" containerID="cri-o://d7e43ad29f8df3b5b72f3717a607d985c30d9adffae3f992bd0b244f420e3c3f" gracePeriod=30 Oct 31 13:24:03.001204 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:03.001165 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:03.869627 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:03.869590 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:04.693461 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:04.693424 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:05.752222 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:05.752183 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:06.794424 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:06.793615 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:07.547626 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.547601 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" Oct 31 13:24:07.564724 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.564692 6841 generic.go:356] "Generic (PLEG): container finished" podID="4f4b3219-9e15-407f-9715-9865e01ea8f7" containerID="d7e43ad29f8df3b5b72f3717a607d985c30d9adffae3f992bd0b244f420e3c3f" exitCode=0 Oct 31 13:24:07.564880 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.564743 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" Oct 31 13:24:07.564880 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.564742 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" event={"ID":"4f4b3219-9e15-407f-9715-9865e01ea8f7","Type":"ContainerDied","Data":"d7e43ad29f8df3b5b72f3717a607d985c30d9adffae3f992bd0b244f420e3c3f"} Oct 31 13:24:07.565067 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.564886 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk" event={"ID":"4f4b3219-9e15-407f-9715-9865e01ea8f7","Type":"ContainerDied","Data":"3672deae979ec0109af66537f6e47895efd7862044dc9eca09116cc916c385a3"} Oct 31 13:24:07.565067 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.564914 6841 scope.go:122] "RemoveContainer" containerID="d7e43ad29f8df3b5b72f3717a607d985c30d9adffae3f992bd0b244f420e3c3f" Oct 31 13:24:07.590226 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.589149 6841 scope.go:122] "RemoveContainer" containerID="d7e43ad29f8df3b5b72f3717a607d985c30d9adffae3f992bd0b244f420e3c3f" Oct 31 13:24:07.591329 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:24:07.591303 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7e43ad29f8df3b5b72f3717a607d985c30d9adffae3f992bd0b244f420e3c3f\": container with ID starting with d7e43ad29f8df3b5b72f3717a607d985c30d9adffae3f992bd0b244f420e3c3f not found: ID does not exist" containerID="d7e43ad29f8df3b5b72f3717a607d985c30d9adffae3f992bd0b244f420e3c3f" Oct 31 13:24:07.591505 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.591481 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7e43ad29f8df3b5b72f3717a607d985c30d9adffae3f992bd0b244f420e3c3f"} err="failed to get container status \"d7e43ad29f8df3b5b72f3717a607d985c30d9adffae3f992bd0b244f420e3c3f\": rpc error: code = NotFound desc = could not find container \"d7e43ad29f8df3b5b72f3717a607d985c30d9adffae3f992bd0b244f420e3c3f\": container with ID starting with d7e43ad29f8df3b5b72f3717a607d985c30d9adffae3f992bd0b244f420e3c3f not found: ID does not exist" Oct 31 13:24:07.617427 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.617399 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f4b3219-9e15-407f-9715-9865e01ea8f7-kube-api-access-8xcf5\" (UniqueName: \"kubernetes.io/projected/4f4b3219-9e15-407f-9715-9865e01ea8f7-kube-api-access-8xcf5\") pod \"4f4b3219-9e15-407f-9715-9865e01ea8f7\" (UID: \"4f4b3219-9e15-407f-9715-9865e01ea8f7\") " Oct 31 13:24:07.617562 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.617439 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f4b3219-9e15-407f-9715-9865e01ea8f7-istio-token\" (UniqueName: \"kubernetes.io/projected/4f4b3219-9e15-407f-9715-9865e01ea8f7-istio-token\") pod \"4f4b3219-9e15-407f-9715-9865e01ea8f7\" (UID: \"4f4b3219-9e15-407f-9715-9865e01ea8f7\") " Oct 31 13:24:07.617649 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.617570 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f4b3219-9e15-407f-9715-9865e01ea8f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f4b3219-9e15-407f-9715-9865e01ea8f7-credential-socket\") pod \"4f4b3219-9e15-407f-9715-9865e01ea8f7\" (UID: \"4f4b3219-9e15-407f-9715-9865e01ea8f7\") " Oct 31 13:24:07.617649 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.617595 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f4b3219-9e15-407f-9715-9865e01ea8f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f4b3219-9e15-407f-9715-9865e01ea8f7-istiod-ca-cert\") pod \"4f4b3219-9e15-407f-9715-9865e01ea8f7\" (UID: \"4f4b3219-9e15-407f-9715-9865e01ea8f7\") " Oct 31 13:24:07.617649 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.617641 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f4b3219-9e15-407f-9715-9865e01ea8f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f4b3219-9e15-407f-9715-9865e01ea8f7-istio-envoy\") pod \"4f4b3219-9e15-407f-9715-9865e01ea8f7\" (UID: \"4f4b3219-9e15-407f-9715-9865e01ea8f7\") " Oct 31 13:24:07.617834 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.617716 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f4b3219-9e15-407f-9715-9865e01ea8f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f4b3219-9e15-407f-9715-9865e01ea8f7-workload-certs\") pod \"4f4b3219-9e15-407f-9715-9865e01ea8f7\" (UID: \"4f4b3219-9e15-407f-9715-9865e01ea8f7\") " Oct 31 13:24:07.617834 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.617827 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f4b3219-9e15-407f-9715-9865e01ea8f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f4b3219-9e15-407f-9715-9865e01ea8f7-istio-data\") pod \"4f4b3219-9e15-407f-9715-9865e01ea8f7\" (UID: \"4f4b3219-9e15-407f-9715-9865e01ea8f7\") " Oct 31 13:24:07.617957 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.617869 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f4b3219-9e15-407f-9715-9865e01ea8f7-credential-socket" pod "4f4b3219-9e15-407f-9715-9865e01ea8f7" (UID: "4f4b3219-9e15-407f-9715-9865e01ea8f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:24:07.617957 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.617916 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f4b3219-9e15-407f-9715-9865e01ea8f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f4b3219-9e15-407f-9715-9865e01ea8f7-istio-podinfo\") pod \"4f4b3219-9e15-407f-9715-9865e01ea8f7\" (UID: \"4f4b3219-9e15-407f-9715-9865e01ea8f7\") " Oct 31 13:24:07.618099 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.617968 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f4b3219-9e15-407f-9715-9865e01ea8f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f4b3219-9e15-407f-9715-9865e01ea8f7-workload-socket\") pod \"4f4b3219-9e15-407f-9715-9865e01ea8f7\" (UID: \"4f4b3219-9e15-407f-9715-9865e01ea8f7\") " Oct 31 13:24:07.618099 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.618052 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f4b3219-9e15-407f-9715-9865e01ea8f7-workload-certs" pod "4f4b3219-9e15-407f-9715-9865e01ea8f7" (UID: "4f4b3219-9e15-407f-9715-9865e01ea8f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:24:07.618099 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.618058 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f4b3219-9e15-407f-9715-9865e01ea8f7-istiod-ca-cert" pod "4f4b3219-9e15-407f-9715-9865e01ea8f7" (UID: "4f4b3219-9e15-407f-9715-9865e01ea8f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:24:07.618283 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.618167 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f4b3219-9e15-407f-9715-9865e01ea8f7-istio-data" pod "4f4b3219-9e15-407f-9715-9865e01ea8f7" (UID: "4f4b3219-9e15-407f-9715-9865e01ea8f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:24:07.618380 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.618361 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f4b3219-9e15-407f-9715-9865e01ea8f7-workload-socket" pod "4f4b3219-9e15-407f-9715-9865e01ea8f7" (UID: "4f4b3219-9e15-407f-9715-9865e01ea8f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:24:07.618753 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.618735 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f4b3219-9e15-407f-9715-9865e01ea8f7-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:24:07.618841 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.618758 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f4b3219-9e15-407f-9715-9865e01ea8f7-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:24:07.618841 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.618771 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f4b3219-9e15-407f-9715-9865e01ea8f7-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:24:07.618841 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.618785 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f4b3219-9e15-407f-9715-9865e01ea8f7-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:24:07.618841 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.618799 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f4b3219-9e15-407f-9715-9865e01ea8f7-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:24:07.620287 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.620261 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f4b3219-9e15-407f-9715-9865e01ea8f7-kube-api-access-8xcf5" pod "4f4b3219-9e15-407f-9715-9865e01ea8f7" (UID: "4f4b3219-9e15-407f-9715-9865e01ea8f7"). InnerVolumeSpecName "kube-api-access-8xcf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:24:07.620402 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.620341 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f4b3219-9e15-407f-9715-9865e01ea8f7-istio-token" pod "4f4b3219-9e15-407f-9715-9865e01ea8f7" (UID: "4f4b3219-9e15-407f-9715-9865e01ea8f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:24:07.620470 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.620417 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f4b3219-9e15-407f-9715-9865e01ea8f7-istio-envoy" pod "4f4b3219-9e15-407f-9715-9865e01ea8f7" (UID: "4f4b3219-9e15-407f-9715-9865e01ea8f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:24:07.620580 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.620562 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f4b3219-9e15-407f-9715-9865e01ea8f7-istio-podinfo" pod "4f4b3219-9e15-407f-9715-9865e01ea8f7" (UID: "4f4b3219-9e15-407f-9715-9865e01ea8f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:24:07.720729 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.720696 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f4b3219-9e15-407f-9715-9865e01ea8f7-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:24:07.720729 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.720726 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f4b3219-9e15-407f-9715-9865e01ea8f7-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:24:07.720729 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.720738 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xcf5\" (UniqueName: \"kubernetes.io/projected/4f4b3219-9e15-407f-9715-9865e01ea8f7-kube-api-access-8xcf5\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:24:07.721007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.720747 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f4b3219-9e15-407f-9715-9865e01ea8f7-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:24:07.916056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.915955 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk"] Oct 31 13:24:07.965917 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:07.965872 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-xxmtk"] Oct 31 13:24:08.136224 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:08.136186 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:09.099909 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:09.099872 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:09.644637 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:09.644603 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f4b3219-9e15-407f-9715-9865e01ea8f7" path="/var/lib/kubelet/pods/4f4b3219-9e15-407f-9715-9865e01ea8f7/volumes" Oct 31 13:24:09.994611 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:09.994576 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:10.744411 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:10.744373 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d"] Oct 31 13:24:10.745383 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:10.745359 6841 cpu_manager.go:414] "containerMap: removing container" podUID="2e3d6506-3350-4dd0-bdf1-efb951353ba5" containerName="istio-proxy" Oct 31 13:24:10.745383 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:10.745382 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e3d6506-3350-4dd0-bdf1-efb951353ba5" containerName="istio-proxy" Oct 31 13:24:10.745576 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:10.745413 6841 cpu_manager.go:414] "containerMap: removing container" podUID="2975edb7-e9cc-4793-acac-baa30b53613c" containerName="istio-proxy" Oct 31 13:24:10.745576 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:10.745421 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2975edb7-e9cc-4793-acac-baa30b53613c" containerName="istio-proxy" Oct 31 13:24:10.745576 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:10.745503 6841 cpu_manager.go:414] "containerMap: removing container" podUID="4f4b3219-9e15-407f-9715-9865e01ea8f7" containerName="istio-proxy" Oct 31 13:24:10.745576 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:10.745512 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f4b3219-9e15-407f-9715-9865e01ea8f7" containerName="istio-proxy" Oct 31 13:24:10.745804 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:10.745682 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e3d6506-3350-4dd0-bdf1-efb951353ba5" containerName="istio-proxy" Oct 31 13:24:10.745804 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:10.745701 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="2975edb7-e9cc-4793-acac-baa30b53613c" containerName="istio-proxy" Oct 31 13:24:10.745804 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:10.745715 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fe9fa73-5f1d-47ea-b303-869b3392ac75" containerName="istio-proxy" Oct 31 13:24:10.745804 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:10.745734 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f4b3219-9e15-407f-9715-9865e01ea8f7" containerName="istio-proxy" Oct 31 13:24:10.751271 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:10.751249 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d" Oct 31 13:24:10.760225 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:10.760197 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d"] Oct 31 13:24:10.764355 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:10.764326 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-4023ad-serving-cert\"" Oct 31 13:24:10.914400 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:10.914037 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:10.924940 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:10.924291 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c26c62d4-873b-422e-8740-457b84206ac3-metrics-certs\") pod \"maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d\" (UID: \"c26c62d4-873b-422e-8740-457b84206ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d" Oct 31 13:24:10.924940 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:10.924339 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swzzb\" (UniqueName: \"kubernetes.io/projected/c26c62d4-873b-422e-8740-457b84206ac3-kube-api-access-swzzb\") pod \"maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d\" (UID: \"c26c62d4-873b-422e-8740-457b84206ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d" Oct 31 13:24:10.924940 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:10.924456 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c26c62d4-873b-422e-8740-457b84206ac3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d\" (UID: \"c26c62d4-873b-422e-8740-457b84206ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d" Oct 31 13:24:10.924940 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:10.924550 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c26c62d4-873b-422e-8740-457b84206ac3-maas-api-tls\") pod \"maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d\" (UID: \"c26c62d4-873b-422e-8740-457b84206ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d" Oct 31 13:24:11.026402 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:11.026328 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c26c62d4-873b-422e-8740-457b84206ac3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d\" (UID: \"c26c62d4-873b-422e-8740-457b84206ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d" Oct 31 13:24:11.026565 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:11.026417 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c26c62d4-873b-422e-8740-457b84206ac3-maas-api-tls\") pod \"maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d\" (UID: \"c26c62d4-873b-422e-8740-457b84206ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d" Oct 31 13:24:11.026669 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:11.026655 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c26c62d4-873b-422e-8740-457b84206ac3-metrics-certs\") pod \"maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d\" (UID: \"c26c62d4-873b-422e-8740-457b84206ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d" Oct 31 13:24:11.026717 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:11.026676 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swzzb\" (UniqueName: \"kubernetes.io/projected/c26c62d4-873b-422e-8740-457b84206ac3-kube-api-access-swzzb\") pod \"maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d\" (UID: \"c26c62d4-873b-422e-8740-457b84206ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d" Oct 31 13:24:11.027020 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:11.026975 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c26c62d4-873b-422e-8740-457b84206ac3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d\" (UID: \"c26c62d4-873b-422e-8740-457b84206ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d" Oct 31 13:24:11.028724 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:11.028703 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c26c62d4-873b-422e-8740-457b84206ac3-maas-api-tls\") pod \"maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d\" (UID: \"c26c62d4-873b-422e-8740-457b84206ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d" Oct 31 13:24:11.029051 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:11.029032 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c26c62d4-873b-422e-8740-457b84206ac3-metrics-certs\") pod \"maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d\" (UID: \"c26c62d4-873b-422e-8740-457b84206ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d" Oct 31 13:24:11.039453 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:11.039426 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swzzb\" (UniqueName: \"kubernetes.io/projected/c26c62d4-873b-422e-8740-457b84206ac3-kube-api-access-swzzb\") pod \"maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d\" (UID: \"c26c62d4-873b-422e-8740-457b84206ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d" Oct 31 13:24:11.074301 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:11.074275 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d" Oct 31 13:24:11.139955 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:11.139909 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:11.436747 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:24:11.436714 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc26c62d4_873b_422e_8740_457b84206ac3.slice/crio-ce160b1fbdfa5c797017984c876bb5cdb1d509e6f041fd4594c7ab66c1d5b315 WatchSource:0}: Error finding container ce160b1fbdfa5c797017984c876bb5cdb1d509e6f041fd4594c7ab66c1d5b315: Status 404 returned error can't find the container with id ce160b1fbdfa5c797017984c876bb5cdb1d509e6f041fd4594c7ab66c1d5b315 Oct 31 13:24:11.437093 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:11.437070 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d"] Oct 31 13:24:11.438374 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:11.438355 6841 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 13:24:11.600859 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:11.600822 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d" event={"ID":"c26c62d4-873b-422e-8740-457b84206ac3","Type":"ContainerStarted","Data":"ce160b1fbdfa5c797017984c876bb5cdb1d509e6f041fd4594c7ab66c1d5b315"} Oct 31 13:24:11.741138 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:11.741099 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:12.522642 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:12.522610 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:12.611386 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:12.611347 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d" event={"ID":"c26c62d4-873b-422e-8740-457b84206ac3","Type":"ContainerStarted","Data":"04656f489e162cf216c0feb9c892fd7f785651121b34696a98baf3921771767f"} Oct 31 13:24:12.611572 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:12.611439 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d" Oct 31 13:24:12.639157 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:12.639096 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d" podStartSLOduration=2.09678169 podStartE2EDuration="2.639082761s" totalImagesPullingTime="542.301071ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:12.633978676 +0000 UTC m=+3099.117101580" watchObservedRunningTime="2026-10-31 13:24:12.639082761 +0000 UTC m=+3099.122205662" Oct 31 13:24:13.249104 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:13.249069 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:13.934847 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:13.934793 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:14.585393 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.585359 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w"] Oct 31 13:24:14.586215 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.586195 6841 cpu_manager.go:414] "containerMap: removing container" podUID="2975edb7-e9cc-4793-acac-baa30b53613c" containerName="istio-proxy" Oct 31 13:24:14.586289 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.586216 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2975edb7-e9cc-4793-acac-baa30b53613c" containerName="istio-proxy" Oct 31 13:24:14.586429 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.586408 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="2975edb7-e9cc-4793-acac-baa30b53613c" containerName="istio-proxy" Oct 31 13:24:14.590739 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.590720 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" Oct 31 13:24:14.602647 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.602617 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w"] Oct 31 13:24:14.607990 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.607948 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx"] Oct 31 13:24:14.614414 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.614388 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" Oct 31 13:24:14.633304 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.633279 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx"] Oct 31 13:24:14.653605 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.653568 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9936f700-0761-4889-9f23-b262428f2d2f-workload-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w\" (UID: \"9936f700-0761-4889-9f23-b262428f2d2f\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" Oct 31 13:24:14.653741 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.653704 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ec82c1-2f15-4a66-8211-330a7a5029cf-credential-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx\" (UID: \"41ec82c1-2f15-4a66-8211-330a7a5029cf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" Oct 31 13:24:14.653741 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.653733 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lgdx\" (UniqueName: \"kubernetes.io/projected/9936f700-0761-4889-9f23-b262428f2d2f-kube-api-access-4lgdx\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w\" (UID: \"9936f700-0761-4889-9f23-b262428f2d2f\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" Oct 31 13:24:14.653882 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.653754 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41ec82c1-2f15-4a66-8211-330a7a5029cf-istiod-ca-cert\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx\" (UID: \"41ec82c1-2f15-4a66-8211-330a7a5029cf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" Oct 31 13:24:14.653882 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.653771 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9936f700-0761-4889-9f23-b262428f2d2f-istio-data\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w\" (UID: \"9936f700-0761-4889-9f23-b262428f2d2f\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" Oct 31 13:24:14.653882 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.653845 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9936f700-0761-4889-9f23-b262428f2d2f-istio-envoy\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w\" (UID: \"9936f700-0761-4889-9f23-b262428f2d2f\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" Oct 31 13:24:14.654026 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.653953 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41ec82c1-2f15-4a66-8211-330a7a5029cf-istio-token\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx\" (UID: \"41ec82c1-2f15-4a66-8211-330a7a5029cf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" Oct 31 13:24:14.654026 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.654000 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41ec82c1-2f15-4a66-8211-330a7a5029cf-istio-envoy\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx\" (UID: \"41ec82c1-2f15-4a66-8211-330a7a5029cf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" Oct 31 13:24:14.654026 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.654023 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9936f700-0761-4889-9f23-b262428f2d2f-istio-podinfo\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w\" (UID: \"9936f700-0761-4889-9f23-b262428f2d2f\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" Oct 31 13:24:14.654148 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.654045 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41ec82c1-2f15-4a66-8211-330a7a5029cf-istio-podinfo\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx\" (UID: \"41ec82c1-2f15-4a66-8211-330a7a5029cf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" Oct 31 13:24:14.654148 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.654115 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9936f700-0761-4889-9f23-b262428f2d2f-workload-certs\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w\" (UID: \"9936f700-0761-4889-9f23-b262428f2d2f\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" Oct 31 13:24:14.654148 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.654136 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ec82c1-2f15-4a66-8211-330a7a5029cf-workload-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx\" (UID: \"41ec82c1-2f15-4a66-8211-330a7a5029cf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" Oct 31 13:24:14.654272 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.654154 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9936f700-0761-4889-9f23-b262428f2d2f-istiod-ca-cert\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w\" (UID: \"9936f700-0761-4889-9f23-b262428f2d2f\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" Oct 31 13:24:14.654272 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.654190 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9936f700-0761-4889-9f23-b262428f2d2f-credential-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w\" (UID: \"9936f700-0761-4889-9f23-b262428f2d2f\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" Oct 31 13:24:14.654373 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.654316 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41ec82c1-2f15-4a66-8211-330a7a5029cf-istio-data\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx\" (UID: \"41ec82c1-2f15-4a66-8211-330a7a5029cf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" Oct 31 13:24:14.654416 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.654394 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdcxd\" (UniqueName: \"kubernetes.io/projected/41ec82c1-2f15-4a66-8211-330a7a5029cf-kube-api-access-mdcxd\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx\" (UID: \"41ec82c1-2f15-4a66-8211-330a7a5029cf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" Oct 31 13:24:14.654470 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.654419 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41ec82c1-2f15-4a66-8211-330a7a5029cf-workload-certs\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx\" (UID: \"41ec82c1-2f15-4a66-8211-330a7a5029cf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" Oct 31 13:24:14.654470 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.654456 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9936f700-0761-4889-9f23-b262428f2d2f-istio-token\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w\" (UID: \"9936f700-0761-4889-9f23-b262428f2d2f\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" Oct 31 13:24:14.757813 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.757779 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41ec82c1-2f15-4a66-8211-330a7a5029cf-istio-token\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx\" (UID: \"41ec82c1-2f15-4a66-8211-330a7a5029cf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" Oct 31 13:24:14.758208 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.758174 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41ec82c1-2f15-4a66-8211-330a7a5029cf-istio-envoy\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx\" (UID: \"41ec82c1-2f15-4a66-8211-330a7a5029cf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" Oct 31 13:24:14.758357 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.758219 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9936f700-0761-4889-9f23-b262428f2d2f-istio-podinfo\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w\" (UID: \"9936f700-0761-4889-9f23-b262428f2d2f\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" Oct 31 13:24:14.758357 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.758249 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41ec82c1-2f15-4a66-8211-330a7a5029cf-istio-podinfo\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx\" (UID: \"41ec82c1-2f15-4a66-8211-330a7a5029cf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" Oct 31 13:24:14.758357 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.758283 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9936f700-0761-4889-9f23-b262428f2d2f-workload-certs\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w\" (UID: \"9936f700-0761-4889-9f23-b262428f2d2f\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" Oct 31 13:24:14.758357 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.758308 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ec82c1-2f15-4a66-8211-330a7a5029cf-workload-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx\" (UID: \"41ec82c1-2f15-4a66-8211-330a7a5029cf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" Oct 31 13:24:14.758357 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.758332 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9936f700-0761-4889-9f23-b262428f2d2f-istiod-ca-cert\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w\" (UID: \"9936f700-0761-4889-9f23-b262428f2d2f\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" Oct 31 13:24:14.758636 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.758391 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9936f700-0761-4889-9f23-b262428f2d2f-credential-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w\" (UID: \"9936f700-0761-4889-9f23-b262428f2d2f\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" Oct 31 13:24:14.758636 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.758447 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41ec82c1-2f15-4a66-8211-330a7a5029cf-istio-data\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx\" (UID: \"41ec82c1-2f15-4a66-8211-330a7a5029cf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" Oct 31 13:24:14.758636 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.758535 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdcxd\" (UniqueName: \"kubernetes.io/projected/41ec82c1-2f15-4a66-8211-330a7a5029cf-kube-api-access-mdcxd\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx\" (UID: \"41ec82c1-2f15-4a66-8211-330a7a5029cf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" Oct 31 13:24:14.758636 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.758567 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41ec82c1-2f15-4a66-8211-330a7a5029cf-workload-certs\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx\" (UID: \"41ec82c1-2f15-4a66-8211-330a7a5029cf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" Oct 31 13:24:14.758848 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.758650 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9936f700-0761-4889-9f23-b262428f2d2f-istio-token\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w\" (UID: \"9936f700-0761-4889-9f23-b262428f2d2f\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" Oct 31 13:24:14.759399 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.758934 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9936f700-0761-4889-9f23-b262428f2d2f-workload-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w\" (UID: \"9936f700-0761-4889-9f23-b262428f2d2f\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" Oct 31 13:24:14.759399 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.759134 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ec82c1-2f15-4a66-8211-330a7a5029cf-workload-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx\" (UID: \"41ec82c1-2f15-4a66-8211-330a7a5029cf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" Oct 31 13:24:14.759399 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.759169 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ec82c1-2f15-4a66-8211-330a7a5029cf-credential-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx\" (UID: \"41ec82c1-2f15-4a66-8211-330a7a5029cf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" Oct 31 13:24:14.759399 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.759209 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lgdx\" (UniqueName: \"kubernetes.io/projected/9936f700-0761-4889-9f23-b262428f2d2f-kube-api-access-4lgdx\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w\" (UID: \"9936f700-0761-4889-9f23-b262428f2d2f\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" Oct 31 13:24:14.759399 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.759238 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41ec82c1-2f15-4a66-8211-330a7a5029cf-istiod-ca-cert\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx\" (UID: \"41ec82c1-2f15-4a66-8211-330a7a5029cf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" Oct 31 13:24:14.759399 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.759261 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9936f700-0761-4889-9f23-b262428f2d2f-istio-data\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w\" (UID: \"9936f700-0761-4889-9f23-b262428f2d2f\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" Oct 31 13:24:14.759399 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.759317 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9936f700-0761-4889-9f23-b262428f2d2f-istio-envoy\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w\" (UID: \"9936f700-0761-4889-9f23-b262428f2d2f\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" Oct 31 13:24:14.759841 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.759642 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9936f700-0761-4889-9f23-b262428f2d2f-istio-data\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w\" (UID: \"9936f700-0761-4889-9f23-b262428f2d2f\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" Oct 31 13:24:14.760267 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.760240 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41ec82c1-2f15-4a66-8211-330a7a5029cf-workload-certs\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx\" (UID: \"41ec82c1-2f15-4a66-8211-330a7a5029cf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" Oct 31 13:24:14.760418 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.760312 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9936f700-0761-4889-9f23-b262428f2d2f-workload-certs\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w\" (UID: \"9936f700-0761-4889-9f23-b262428f2d2f\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" Oct 31 13:24:14.760534 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.760399 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9936f700-0761-4889-9f23-b262428f2d2f-workload-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w\" (UID: \"9936f700-0761-4889-9f23-b262428f2d2f\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" Oct 31 13:24:14.760643 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.760538 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9936f700-0761-4889-9f23-b262428f2d2f-credential-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w\" (UID: \"9936f700-0761-4889-9f23-b262428f2d2f\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" Oct 31 13:24:14.760643 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.760641 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ec82c1-2f15-4a66-8211-330a7a5029cf-credential-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx\" (UID: \"41ec82c1-2f15-4a66-8211-330a7a5029cf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" Oct 31 13:24:14.761114 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.760885 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41ec82c1-2f15-4a66-8211-330a7a5029cf-istio-data\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx\" (UID: \"41ec82c1-2f15-4a66-8211-330a7a5029cf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" Oct 31 13:24:14.761292 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.761271 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9936f700-0761-4889-9f23-b262428f2d2f-istiod-ca-cert\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w\" (UID: \"9936f700-0761-4889-9f23-b262428f2d2f\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" Oct 31 13:24:14.761553 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.761534 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41ec82c1-2f15-4a66-8211-330a7a5029cf-istiod-ca-cert\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx\" (UID: \"41ec82c1-2f15-4a66-8211-330a7a5029cf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" Oct 31 13:24:14.762851 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.762826 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9936f700-0761-4889-9f23-b262428f2d2f-istio-podinfo\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w\" (UID: \"9936f700-0761-4889-9f23-b262428f2d2f\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" Oct 31 13:24:14.763925 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.763898 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41ec82c1-2f15-4a66-8211-330a7a5029cf-istio-podinfo\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx\" (UID: \"41ec82c1-2f15-4a66-8211-330a7a5029cf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" Oct 31 13:24:14.764186 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.764163 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9936f700-0761-4889-9f23-b262428f2d2f-istio-envoy\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w\" (UID: \"9936f700-0761-4889-9f23-b262428f2d2f\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" Oct 31 13:24:14.765267 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.765248 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41ec82c1-2f15-4a66-8211-330a7a5029cf-istio-envoy\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx\" (UID: \"41ec82c1-2f15-4a66-8211-330a7a5029cf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" Oct 31 13:24:14.803868 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.803849 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41ec82c1-2f15-4a66-8211-330a7a5029cf-istio-token\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx\" (UID: \"41ec82c1-2f15-4a66-8211-330a7a5029cf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" Oct 31 13:24:14.807370 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.807346 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9936f700-0761-4889-9f23-b262428f2d2f-istio-token\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w\" (UID: \"9936f700-0761-4889-9f23-b262428f2d2f\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" Oct 31 13:24:14.809124 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.809106 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdcxd\" (UniqueName: \"kubernetes.io/projected/41ec82c1-2f15-4a66-8211-330a7a5029cf-kube-api-access-mdcxd\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx\" (UID: \"41ec82c1-2f15-4a66-8211-330a7a5029cf\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" Oct 31 13:24:14.812763 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.812745 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lgdx\" (UniqueName: \"kubernetes.io/projected/9936f700-0761-4889-9f23-b262428f2d2f-kube-api-access-4lgdx\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w\" (UID: \"9936f700-0761-4889-9f23-b262428f2d2f\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" Oct 31 13:24:14.857663 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.857590 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:14.912750 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.912727 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" Oct 31 13:24:14.947194 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:14.947168 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" Oct 31 13:24:15.212334 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:24:15.212306 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9936f700_0761_4889_9f23_b262428f2d2f.slice/crio-b806eeb87a8cd7f54d9f3702c0101374d9aa1e9c4d86be9a289d15bf21dedec3 WatchSource:0}: Error finding container b806eeb87a8cd7f54d9f3702c0101374d9aa1e9c4d86be9a289d15bf21dedec3: Status 404 returned error can't find the container with id b806eeb87a8cd7f54d9f3702c0101374d9aa1e9c4d86be9a289d15bf21dedec3 Oct 31 13:24:15.212680 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:15.212656 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w"] Oct 31 13:24:15.215480 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:15.215425 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:24:15.215598 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:15.215503 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:24:15.215598 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:15.215549 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:24:15.264734 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:15.264393 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx"] Oct 31 13:24:15.278812 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:15.278029 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:24:15.278812 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:15.278121 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:24:15.278812 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:15.278161 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:24:15.652740 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:15.652651 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" event={"ID":"41ec82c1-2f15-4a66-8211-330a7a5029cf","Type":"ContainerStarted","Data":"600ec71a0cd3c632ef9326206b93a90e760fb443216f4534e6e92271b8b3b25c"} Oct 31 13:24:15.652740 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:15.652717 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" event={"ID":"41ec82c1-2f15-4a66-8211-330a7a5029cf","Type":"ContainerStarted","Data":"f46c8e23e05986d1821e196d613fa09801a4b5838b270562cd6c5b6a1e5dee07"} Oct 31 13:24:15.654810 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:15.654784 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" event={"ID":"9936f700-0761-4889-9f23-b262428f2d2f","Type":"ContainerStarted","Data":"42df17d8d39f6e87de7c8c63aac9d01da24f36654742845f597d5d08491985f1"} Oct 31 13:24:15.654936 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:15.654829 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" event={"ID":"9936f700-0761-4889-9f23-b262428f2d2f","Type":"ContainerStarted","Data":"b806eeb87a8cd7f54d9f3702c0101374d9aa1e9c4d86be9a289d15bf21dedec3"} Oct 31 13:24:15.682226 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:15.682166 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" podStartSLOduration=1.682148304 podStartE2EDuration="1.682148304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:15.67990789 +0000 UTC m=+3102.163030793" watchObservedRunningTime="2026-10-31 13:24:15.682148304 +0000 UTC m=+3102.165271208" Oct 31 13:24:15.712804 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:15.712704 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" podStartSLOduration=1.712688513 podStartE2EDuration="1.712688513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:15.707875545 +0000 UTC m=+3102.190998445" watchObservedRunningTime="2026-10-31 13:24:15.712688513 +0000 UTC m=+3102.195811413" Oct 31 13:24:15.826770 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:15.826734 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:15.912969 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:15.912874 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" Oct 31 13:24:15.947644 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:15.947612 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" Oct 31 13:24:16.720007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:16.709307 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:16.915438 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:16.915406 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Oct 31 13:24:16.915642 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:16.915459 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" podUID="9936f700-0761-4889-9f23-b262428f2d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Oct 31 13:24:16.974060 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:16.973083 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Oct 31 13:24:16.974060 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:16.973145 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" podUID="41ec82c1-2f15-4a66-8211-330a7a5029cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Oct 31 13:24:17.529008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:17.517520 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:17.913635 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:17.913211 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Oct 31 13:24:17.913635 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:17.913263 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" podUID="9936f700-0761-4889-9f23-b262428f2d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Oct 31 13:24:17.966128 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:17.966087 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Oct 31 13:24:17.966442 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:17.966409 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" podUID="41ec82c1-2f15-4a66-8211-330a7a5029cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Oct 31 13:24:18.403007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:18.394474 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:18.596107 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:18.594073 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:18.625959 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:18.625932 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d" Oct 31 13:24:18.913280 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:18.913243 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Oct 31 13:24:18.913489 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:18.913300 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" podUID="9936f700-0761-4889-9f23-b262428f2d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Oct 31 13:24:18.998882 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:18.998846 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Oct 31 13:24:18.999190 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:18.999161 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" podUID="41ec82c1-2f15-4a66-8211-330a7a5029cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Oct 31 13:24:19.258807 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:19.258765 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:19.914141 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:19.914099 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Oct 31 13:24:19.914594 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:19.914155 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" podUID="9936f700-0761-4889-9f23-b262428f2d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Oct 31 13:24:19.964005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:19.963268 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Oct 31 13:24:19.964005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:19.963327 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" podUID="41ec82c1-2f15-4a66-8211-330a7a5029cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Oct 31 13:24:20.070070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:20.054964 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:20.137247 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:24:20.137205 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:24:20.703472 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:20.703437 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7"] Oct 31 13:24:20.709398 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:20.709357 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7" Oct 31 13:24:20.716839 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:20.716814 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-5dbd39-serving-cert\"" Oct 31 13:24:20.731922 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:20.731878 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7"] Oct 31 13:24:20.804480 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:20.804441 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df0aa4b6-4450-4322-afca-7cfd6c6cf330-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7\" (UID: \"df0aa4b6-4450-4322-afca-7cfd6c6cf330\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7" Oct 31 13:24:20.804694 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:20.804518 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ttw4\" (UniqueName: \"kubernetes.io/projected/df0aa4b6-4450-4322-afca-7cfd6c6cf330-kube-api-access-5ttw4\") pod \"maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7\" (UID: \"df0aa4b6-4450-4322-afca-7cfd6c6cf330\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7" Oct 31 13:24:20.804694 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:20.804621 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df0aa4b6-4450-4322-afca-7cfd6c6cf330-metrics-certs\") pod \"maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7\" (UID: \"df0aa4b6-4450-4322-afca-7cfd6c6cf330\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7" Oct 31 13:24:20.804694 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:20.804654 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df0aa4b6-4450-4322-afca-7cfd6c6cf330-maas-api-tls\") pod \"maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7\" (UID: \"df0aa4b6-4450-4322-afca-7cfd6c6cf330\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7" Oct 31 13:24:20.932009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:20.920308 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df0aa4b6-4450-4322-afca-7cfd6c6cf330-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7\" (UID: \"df0aa4b6-4450-4322-afca-7cfd6c6cf330\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7" Oct 31 13:24:20.932009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:20.920389 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ttw4\" (UniqueName: \"kubernetes.io/projected/df0aa4b6-4450-4322-afca-7cfd6c6cf330-kube-api-access-5ttw4\") pod \"maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7\" (UID: \"df0aa4b6-4450-4322-afca-7cfd6c6cf330\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7" Oct 31 13:24:20.932009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:20.920490 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df0aa4b6-4450-4322-afca-7cfd6c6cf330-metrics-certs\") pod \"maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7\" (UID: \"df0aa4b6-4450-4322-afca-7cfd6c6cf330\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7" Oct 31 13:24:20.932009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:20.920522 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df0aa4b6-4450-4322-afca-7cfd6c6cf330-maas-api-tls\") pod \"maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7\" (UID: \"df0aa4b6-4450-4322-afca-7cfd6c6cf330\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7" Oct 31 13:24:20.932009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:20.922321 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df0aa4b6-4450-4322-afca-7cfd6c6cf330-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7\" (UID: \"df0aa4b6-4450-4322-afca-7cfd6c6cf330\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7" Oct 31 13:24:20.932009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:20.922976 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Oct 31 13:24:20.932009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:20.923121 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" podUID="9936f700-0761-4889-9f23-b262428f2d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Oct 31 13:24:20.960874 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:20.950736 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Oct 31 13:24:20.960874 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:20.950795 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" podUID="41ec82c1-2f15-4a66-8211-330a7a5029cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Oct 31 13:24:21.014771 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:21.014648 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:21.015574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:21.015546 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df0aa4b6-4450-4322-afca-7cfd6c6cf330-maas-api-tls\") pod \"maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7\" (UID: \"df0aa4b6-4450-4322-afca-7cfd6c6cf330\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7" Oct 31 13:24:21.018628 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:21.018548 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df0aa4b6-4450-4322-afca-7cfd6c6cf330-metrics-certs\") pod \"maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7\" (UID: \"df0aa4b6-4450-4322-afca-7cfd6c6cf330\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7" Oct 31 13:24:21.019771 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:21.019721 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ttw4\" (UniqueName: \"kubernetes.io/projected/df0aa4b6-4450-4322-afca-7cfd6c6cf330-kube-api-access-5ttw4\") pod \"maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7\" (UID: \"df0aa4b6-4450-4322-afca-7cfd6c6cf330\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7" Oct 31 13:24:21.103744 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:21.103701 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7" Oct 31 13:24:21.218808 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:21.218720 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:21.729251 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:21.729227 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7"] Oct 31 13:24:21.814414 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:21.814321 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7" event={"ID":"df0aa4b6-4450-4322-afca-7cfd6c6cf330","Type":"ContainerStarted","Data":"0e1289fb397a953af3a0301bf3ac2ac1972b014a459d37fda58bfc0690c5de4e"} Oct 31 13:24:21.919019 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:21.918591 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Oct 31 13:24:21.919019 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:21.918657 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" podUID="9936f700-0761-4889-9f23-b262428f2d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Oct 31 13:24:21.951572 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:21.951539 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Oct 31 13:24:21.952101 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:21.952024 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" podUID="41ec82c1-2f15-4a66-8211-330a7a5029cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Oct 31 13:24:21.952884 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:21.952854 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:22.702372 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:22.702330 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:22.824268 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:22.824223 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7" event={"ID":"df0aa4b6-4450-4322-afca-7cfd6c6cf330","Type":"ContainerStarted","Data":"e4d01cca31f6cfc709a8f957316d78dd1394b60edacbb9acea4d0d5bace8be17"} Oct 31 13:24:22.824461 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:22.824294 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7" Oct 31 13:24:22.855475 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:22.855426 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7" podStartSLOduration=2.401654453 podStartE2EDuration="2.855410161s" totalImagesPullingTime="453.755708ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:22.850375696 +0000 UTC m=+3109.333498596" watchObservedRunningTime="2026-10-31 13:24:22.855410161 +0000 UTC m=+3109.338533067" Oct 31 13:24:22.919535 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:22.919506 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" Oct 31 13:24:22.919910 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:22.919888 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" Oct 31 13:24:22.921157 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:22.921136 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" Oct 31 13:24:22.952774 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:22.952705 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" Oct 31 13:24:22.953148 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:22.952999 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" Oct 31 13:24:22.953644 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:22.953625 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" Oct 31 13:24:23.405357 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:23.404556 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:24.129111 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:24.129067 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:24.810532 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:24.810400 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:25.595249 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:25.595206 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:26.509796 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:26.509491 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:27.263534 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:27.262293 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:27.932211 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:27.932171 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:28.711690 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:28.711649 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:28.837669 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:28.837640 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7" Oct 31 13:24:29.504053 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:29.503974 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:30.234681 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:30.234637 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:31.138224 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:31.138189 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:31.934925 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:31.934893 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:32.795371 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:32.795337 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:33.612505 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:33.612463 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:34.959506 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:34.959450 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:35.821334 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:35.821302 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:36.613544 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:36.613507 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:36.814035 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:36.813991 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:37.183099 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:37.183064 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:37.436057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:37.435935 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:37.538316 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:37.538280 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7"] Oct 31 13:24:37.544726 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:37.544702 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:24:37.550900 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:37.550873 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c6c445-openshift-default-dockercfg-8mlf6\"" Oct 31 13:24:37.555467 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:37.555437 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7"] Oct 31 13:24:37.683899 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:37.683867 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:37.777006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:37.763108 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-workload-certs\") pod \"e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7\" (UID: \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\") " pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:24:37.777006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:37.763206 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-istio-podinfo\") pod \"e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7\" (UID: \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\") " pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:24:37.777006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:37.763303 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-istiod-ca-cert\") pod \"e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7\" (UID: \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\") " pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:24:37.777006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:37.763422 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-istio-token\") pod \"e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7\" (UID: \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\") " pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:24:37.777006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:37.763450 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-istio-envoy\") pod \"e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7\" (UID: \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\") " pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:24:37.777006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:37.763527 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-workload-socket\") pod \"e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7\" (UID: \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\") " pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:24:37.777006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:37.763604 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-istio-data\") pod \"e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7\" (UID: \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\") " pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:24:37.777006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:37.763682 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-credential-socket\") pod \"e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7\" (UID: \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\") " pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:24:37.777006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:37.763715 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmhmr\" (UniqueName: \"kubernetes.io/projected/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-kube-api-access-vmhmr\") pod \"e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7\" (UID: \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\") " pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:24:37.879014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:37.877134 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-istio-token\") pod \"e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7\" (UID: \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\") " pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:24:37.879014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:37.877190 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-istio-envoy\") pod \"e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7\" (UID: \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\") " pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:24:37.879014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:37.877259 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-workload-socket\") pod \"e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7\" (UID: \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\") " pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:24:37.879014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:37.877324 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-istio-data\") pod \"e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7\" (UID: \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\") " pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:24:37.879014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:37.877397 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-credential-socket\") pod \"e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7\" (UID: \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\") " pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:24:37.879014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:37.877424 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmhmr\" (UniqueName: \"kubernetes.io/projected/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-kube-api-access-vmhmr\") pod \"e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7\" (UID: \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\") " pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:24:37.879014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:37.877513 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-workload-certs\") pod \"e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7\" (UID: \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\") " pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:24:37.879014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:37.877579 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-istio-podinfo\") pod \"e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7\" (UID: \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\") " pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:24:37.879014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:37.877666 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-istiod-ca-cert\") pod \"e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7\" (UID: \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\") " pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:24:37.879014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:37.878643 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-istiod-ca-cert\") pod \"e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7\" (UID: \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\") " pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:24:37.880481 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:37.880452 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-credential-socket\") pod \"e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7\" (UID: \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\") " pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:24:37.880939 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:37.880915 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-workload-socket\") pod \"e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7\" (UID: \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\") " pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:24:37.881363 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:37.881339 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-istio-data\") pod \"e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7\" (UID: \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\") " pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:24:37.881764 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:37.881743 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-workload-certs\") pod \"e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7\" (UID: \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\") " pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:24:37.909406 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:37.909351 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:37.928394 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:37.927854 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-istio-token\") pod \"e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7\" (UID: \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\") " pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:24:37.940078 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:37.932720 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmhmr\" (UniqueName: \"kubernetes.io/projected/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-kube-api-access-vmhmr\") pod \"e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7\" (UID: \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\") " pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:24:37.940590 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:37.940562 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-istio-envoy\") pod \"e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7\" (UID: \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\") " pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:24:37.957851 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:37.944817 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-istio-podinfo\") pod \"e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7\" (UID: \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\") " pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:24:38.165328 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:38.165257 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:24:38.458730 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:24:38.458699 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d2d6ca_9ce2_421e_919c_7d5c8200ad4f.slice/crio-ecc9af2001de9e8cb8063d6e950430174edde5a65e154c28f82d9ab4897ec86a WatchSource:0}: Error finding container ecc9af2001de9e8cb8063d6e950430174edde5a65e154c28f82d9ab4897ec86a: Status 404 returned error can't find the container with id ecc9af2001de9e8cb8063d6e950430174edde5a65e154c28f82d9ab4897ec86a Oct 31 13:24:38.459122 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:38.459096 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7"] Oct 31 13:24:38.461778 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:38.461743 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:24:38.461895 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:38.461822 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:24:38.461895 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:38.461872 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:24:38.638319 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:38.638284 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:38.931139 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:38.931067 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:39.010871 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:39.010836 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" event={"ID":"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f","Type":"ContainerStarted","Data":"79b46d3cdeee3c8947262d926fcde8e152aabb749f88e4d1d46db61ea61822a5"} Oct 31 13:24:39.011124 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:39.010903 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" event={"ID":"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f","Type":"ContainerStarted","Data":"ecc9af2001de9e8cb8063d6e950430174edde5a65e154c28f82d9ab4897ec86a"} Oct 31 13:24:39.040501 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:39.040456 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" podStartSLOduration=2.040443256 podStartE2EDuration="2.040443256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:39.040122121 +0000 UTC m=+3125.523245024" watchObservedRunningTime="2026-10-31 13:24:39.040443256 +0000 UTC m=+3125.523566156" Oct 31 13:24:39.165511 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:39.165476 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:24:39.172268 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:39.172242 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:24:39.320533 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:39.320497 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:40.019738 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:40.019706 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:24:40.020752 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:40.020733 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:24:40.123233 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:40.123197 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:41.024611 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:41.024574 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:41.595605 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:41.595562 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:41.800414 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:41.800376 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:42.000878 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:42.000840 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:42.698022 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:42.697007 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:42.909376 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:42.909320 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:43.148057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:43.147037 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:43.774637 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:43.774600 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:44.547211 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:44.547123 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:45.603648 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:45.603617 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:46.510005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:46.509132 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:47.545007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:47.543058 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:48.214761 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:48.214726 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:48.627675 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:48.627589 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:48.832062 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:48.832024 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:48.990826 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:48.990790 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp"] Oct 31 13:24:48.997553 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:48.997527 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:24:49.004297 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.004252 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-384dbe-openshift-default-dockercfg-nhwzf\"" Oct 31 13:24:49.010312 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.010287 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp"] Oct 31 13:24:49.141017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.136524 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-istio-data\") pod \"e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp\" (UID: \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\") " pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:24:49.141017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.136670 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-workload-socket\") pod \"e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp\" (UID: \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\") " pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:24:49.141017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.136780 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-istio-podinfo\") pod \"e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp\" (UID: \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\") " pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:24:49.141017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.136915 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-credential-socket\") pod \"e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp\" (UID: \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\") " pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:24:49.141017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.136952 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8b88\" (UniqueName: \"kubernetes.io/projected/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-kube-api-access-s8b88\") pod \"e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp\" (UID: \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\") " pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:24:49.141017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.137081 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-istio-token\") pod \"e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp\" (UID: \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\") " pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:24:49.141017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.137142 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-workload-certs\") pod \"e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp\" (UID: \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\") " pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:24:49.141017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.137212 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-istiod-ca-cert\") pod \"e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp\" (UID: \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\") " pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:24:49.141017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.137370 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-istio-envoy\") pod \"e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp\" (UID: \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\") " pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:24:49.214007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.194695 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:49.258057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.257382 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-istio-data\") pod \"e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp\" (UID: \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\") " pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:24:49.258057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.257454 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-workload-socket\") pod \"e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp\" (UID: \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\") " pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:24:49.258057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.257513 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-istio-podinfo\") pod \"e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp\" (UID: \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\") " pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:24:49.258057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.257621 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-credential-socket\") pod \"e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp\" (UID: \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\") " pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:24:49.258057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.257651 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8b88\" (UniqueName: \"kubernetes.io/projected/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-kube-api-access-s8b88\") pod \"e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp\" (UID: \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\") " pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:24:49.258057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.257708 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-istio-token\") pod \"e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp\" (UID: \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\") " pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:24:49.258057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.257749 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-workload-certs\") pod \"e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp\" (UID: \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\") " pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:24:49.258057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.257800 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-istiod-ca-cert\") pod \"e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp\" (UID: \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\") " pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:24:49.258057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.257919 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-istio-envoy\") pod \"e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp\" (UID: \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\") " pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:24:49.283007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.268728 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-credential-socket\") pod \"e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp\" (UID: \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\") " pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:24:49.283007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.269553 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-workload-certs\") pod \"e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp\" (UID: \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\") " pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:24:49.283007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.270383 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-istio-data\") pod \"e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp\" (UID: \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\") " pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:24:49.283007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.270371 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-istiod-ca-cert\") pod \"e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp\" (UID: \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\") " pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:24:49.283007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.270614 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-workload-socket\") pod \"e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp\" (UID: \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\") " pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:24:49.411005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.410792 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8b88\" (UniqueName: \"kubernetes.io/projected/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-kube-api-access-s8b88\") pod \"e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp\" (UID: \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\") " pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:24:49.421088 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.421050 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-istio-podinfo\") pod \"e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp\" (UID: \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\") " pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:24:49.434889 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.434209 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-istio-envoy\") pod \"e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp\" (UID: \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\") " pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:24:49.436774 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.436714 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-istio-token\") pod \"e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp\" (UID: \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\") " pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:24:49.622303 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.622221 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:24:49.784176 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.784019 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:49.916262 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:24:49.916222 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dfe8ea5_8d8c_4b1e_ae0b_d7b7483b6b1a.slice/crio-402c95be22d7ca72ec239a81dc805e082c162f0c8e0f85ccbcf72a196f1e1c78 WatchSource:0}: Error finding container 402c95be22d7ca72ec239a81dc805e082c162f0c8e0f85ccbcf72a196f1e1c78: Status 404 returned error can't find the container with id 402c95be22d7ca72ec239a81dc805e082c162f0c8e0f85ccbcf72a196f1e1c78 Oct 31 13:24:49.916372 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.916343 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp"] Oct 31 13:24:49.919294 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.919260 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:24:49.919388 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.919326 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:24:49.919388 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.919356 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:24:49.984598 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:49.984568 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:50.035774 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:24:50.035742 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:24:50.421751 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:50.421711 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" event={"ID":"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a","Type":"ContainerStarted","Data":"63c97c8347b67d1012a4027d06e9e76aebdbe0a24a2dc927092c2205621636bb"} Oct 31 13:24:50.421959 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:50.421783 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" event={"ID":"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a","Type":"ContainerStarted","Data":"402c95be22d7ca72ec239a81dc805e082c162f0c8e0f85ccbcf72a196f1e1c78"} Oct 31 13:24:50.450926 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:50.450858 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" podStartSLOduration=2.4508408680000002 podStartE2EDuration="2.450840868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:50.449660977 +0000 UTC m=+3136.932783871" watchObservedRunningTime="2026-10-31 13:24:50.450840868 +0000 UTC m=+3136.933963776" Oct 31 13:24:50.626231 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:50.626093 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:24:50.755411 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:50.755373 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:51.629459 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:51.629427 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:24:51.866530 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:51.866491 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:52.438694 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:52.438657 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:24:52.439585 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:52.439567 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:24:52.777393 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:52.777362 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:53.097470 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:53.097376 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:53.343042 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:53.343004 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:54.086148 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:54.086115 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:54.371241 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:54.371155 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:54.660969 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:54.660865 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:55.488543 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:55.488506 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:56.371265 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:56.371228 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:57.408134 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:57.408020 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:58.192030 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:58.191700 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:58.910141 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:58.910103 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:24:59.774565 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:24:59.774534 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:00.479081 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:00.479037 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:01.393974 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:01.393941 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:02.232227 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:02.231977 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:03.112221 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:03.112181 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:03.838376 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:03.838333 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:04.864678 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:04.864642 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:05.731135 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:05.731098 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:06.550799 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:06.550753 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:07.537145 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:07.537106 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:07.758814 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:07.758778 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:08.765826 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:08.765791 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:09.927066 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:09.926516 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:10.789278 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:10.789240 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:11.541463 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:11.541402 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:12.391932 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:12.391345 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:13.099417 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:13.099380 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:13.988768 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:13.988728 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:15.010293 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:15.008586 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:15.703005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:15.702293 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:16.544370 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:16.544340 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:17.460823 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:17.460784 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:18.281406 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:18.281369 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:18.527494 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:18.527462 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:19.240184 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:19.240130 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:20.026030 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:20.025910 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:20.113431 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:25:20.113390 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:25:20.914375 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:20.914286 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:21.733936 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:21.733897 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:22.561702 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:22.561651 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:23.173221 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:23.173187 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:23.387161 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:23.387121 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:23.591679 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:23.591645 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:24.518846 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:24.518814 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:25.405892 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:25.405807 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:26.140040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:26.140007 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:26.935653 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:26.935617 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:29.180461 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:29.180409 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:29.397589 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:29.397553 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:30.103586 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:30.103550 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:30.812428 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:30.812394 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:31.520007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:31.519835 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:32.232290 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:32.232253 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:33.012586 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:33.012554 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:33.835004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:33.833576 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:34.819569 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:34.819479 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:35.041153 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:35.041113 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:35.565609 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:35.565571 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:35.921921 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:35.919018 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:36.826824 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:36.826794 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:37.513696 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:37.513647 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:37.720183 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:37.720133 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:38.552275 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:38.552240 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:38.604606 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:38.604568 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5"] Oct 31 13:25:38.609773 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:38.609750 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5" Oct 31 13:25:38.625723 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:38.625697 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5"] Oct 31 13:25:38.628028 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:38.628000 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4023ad\"/\"test-model-4023ad-kserve-self-signed-certs\"" Oct 31 13:25:38.628123 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:38.628033 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4023ad\"/\"default-dockercfg-xlhlt\"" Oct 31 13:25:38.651425 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:38.651377 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4023ad\"/\"openshift-service-ca.crt\"" Oct 31 13:25:38.651645 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:38.651458 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4023ad\"/\"kube-root-ca.crt\"" Oct 31 13:25:38.706061 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:38.706025 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-home\") pod \"test-model-4023ad-kserve-85694f8f59-5ggn5\" (UID: \"50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1\") " pod="ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5" Oct 31 13:25:38.706247 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:38.706175 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-model-cache\") pod \"test-model-4023ad-kserve-85694f8f59-5ggn5\" (UID: \"50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1\") " pod="ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5" Oct 31 13:25:38.706247 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:38.706240 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-tmp-dir\") pod \"test-model-4023ad-kserve-85694f8f59-5ggn5\" (UID: \"50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1\") " pod="ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5" Oct 31 13:25:38.706550 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:38.706514 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-dshm\") pod \"test-model-4023ad-kserve-85694f8f59-5ggn5\" (UID: \"50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1\") " pod="ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5" Oct 31 13:25:38.706698 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:38.706615 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-tls-certs\") pod \"test-model-4023ad-kserve-85694f8f59-5ggn5\" (UID: \"50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1\") " pod="ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5" Oct 31 13:25:38.706698 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:38.706683 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx94f\" (UniqueName: \"kubernetes.io/projected/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-kube-api-access-rx94f\") pod \"test-model-4023ad-kserve-85694f8f59-5ggn5\" (UID: \"50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1\") " pod="ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5" Oct 31 13:25:38.822078 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:38.808410 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-model-cache\") pod \"test-model-4023ad-kserve-85694f8f59-5ggn5\" (UID: \"50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1\") " pod="ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5" Oct 31 13:25:38.822434 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:38.822376 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-tmp-dir\") pod \"test-model-4023ad-kserve-85694f8f59-5ggn5\" (UID: \"50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1\") " pod="ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5" Oct 31 13:25:38.825359 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:38.825330 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-dshm\") pod \"test-model-4023ad-kserve-85694f8f59-5ggn5\" (UID: \"50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1\") " pod="ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5" Oct 31 13:25:38.825611 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:38.825594 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-tls-certs\") pod \"test-model-4023ad-kserve-85694f8f59-5ggn5\" (UID: \"50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1\") " pod="ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5" Oct 31 13:25:38.825763 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:38.825745 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx94f\" (UniqueName: \"kubernetes.io/projected/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-kube-api-access-rx94f\") pod \"test-model-4023ad-kserve-85694f8f59-5ggn5\" (UID: \"50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1\") " pod="ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5" Oct 31 13:25:38.826131 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:38.826112 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-home\") pod \"test-model-4023ad-kserve-85694f8f59-5ggn5\" (UID: \"50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1\") " pod="ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5" Oct 31 13:25:38.827336 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:38.827312 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-home\") pod \"test-model-4023ad-kserve-85694f8f59-5ggn5\" (UID: \"50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1\") " pod="ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5" Oct 31 13:25:38.827487 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:38.821168 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:38.828277 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:38.822949 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-tmp-dir\") pod \"test-model-4023ad-kserve-85694f8f59-5ggn5\" (UID: \"50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1\") " pod="ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5" Oct 31 13:25:38.828472 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:38.809281 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-model-cache\") pod \"test-model-4023ad-kserve-85694f8f59-5ggn5\" (UID: \"50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1\") " pod="ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5" Oct 31 13:25:38.847068 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:38.847036 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-tls-certs\") pod \"test-model-4023ad-kserve-85694f8f59-5ggn5\" (UID: \"50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1\") " pod="ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5" Oct 31 13:25:38.863187 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:38.863126 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-dshm\") pod \"test-model-4023ad-kserve-85694f8f59-5ggn5\" (UID: \"50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1\") " pod="ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5" Oct 31 13:25:38.882799 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:38.882766 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx94f\" (UniqueName: \"kubernetes.io/projected/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-kube-api-access-rx94f\") pod \"test-model-4023ad-kserve-85694f8f59-5ggn5\" (UID: \"50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1\") " pod="ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5" Oct 31 13:25:38.928873 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:38.928837 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5" Oct 31 13:25:39.212633 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:39.212595 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:39.312094 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:25:39.312063 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50fa0a40_a8cb_4da7_bd3a_65dacb0de7e1.slice/crio-e3155cfc211122df21e49d2722ac709ee69709f9d85cee888bd1384bce3bee20 WatchSource:0}: Error finding container e3155cfc211122df21e49d2722ac709ee69709f9d85cee888bd1384bce3bee20: Status 404 returned error can't find the container with id e3155cfc211122df21e49d2722ac709ee69709f9d85cee888bd1384bce3bee20 Oct 31 13:25:39.312241 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:39.312102 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5"] Oct 31 13:25:39.575356 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:39.575260 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:40.134816 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:40.134779 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5" event={"ID":"50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1","Type":"ContainerStarted","Data":"ed4c5c8602b88576a80a2900b35ac10725362d7be23c870d51e8c4b36317de36"} Oct 31 13:25:40.135023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:40.134827 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5" event={"ID":"50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1","Type":"ContainerStarted","Data":"e3155cfc211122df21e49d2722ac709ee69709f9d85cee888bd1384bce3bee20"} Oct 31 13:25:40.158549 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:40.158502 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5" podStartSLOduration=2.1584885209999998 podStartE2EDuration="2.158488521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:25:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:25:40.154745593 +0000 UTC m=+3186.637868495" watchObservedRunningTime="2026-10-31 13:25:40.158488521 +0000 UTC m=+3186.641611422" Oct 31 13:25:40.309031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:40.308905 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:40.930065 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:40.930033 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5" Oct 31 13:25:41.386105 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:41.386071 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:42.108454 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:42.108417 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:42.419525 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:42.419443 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:42.641119 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:42.641078 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:42.873927 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:42.873884 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:43.120666 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:43.120624 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:43.345044 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:43.345006 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:43.748580 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:43.748551 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:43.990929 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:43.990892 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:44.659006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:44.652147 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:45.215825 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:45.215792 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:46.000685 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:46.000647 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:46.666231 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:46.666196 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:47.733260 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:47.733152 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:48.628218 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:48.626647 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:48.929580 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:48.929499 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5" Oct 31 13:25:48.940975 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:48.940942 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:48.967088 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:48.966709 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5" Oct 31 13:25:49.739007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:49.734854 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:49.794768 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:49.794728 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5" Oct 31 13:25:50.036488 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:25:50.036407 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:25:50.781799 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:50.781764 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:51.601004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:51.600337 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:52.377705 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:52.377672 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:52.671187 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:52.671087 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:52.897785 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:52.897752 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:53.815611 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:53.815571 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:54.088140 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:54.080281 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:55.127968 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:55.127929 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:55.899594 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:55.899563 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:56.752829 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:56.752783 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:57.408157 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:57.408071 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j"] Oct 31 13:25:57.422275 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:57.422244 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j"] Oct 31 13:25:57.422656 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:57.422635 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j" Oct 31 13:25:57.429696 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:57.429671 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5dbd39\"/\"default-dockercfg-wdjk4\"" Oct 31 13:25:57.429845 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:57.429730 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5dbd39\"/\"test-model-5dbd39-kserve-self-signed-certs\"" Oct 31 13:25:57.430837 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:57.430819 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5dbd39\"/\"kube-root-ca.crt\"" Oct 31 13:25:57.430943 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:57.430897 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5dbd39\"/\"openshift-service-ca.crt\"" Oct 31 13:25:57.546297 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:57.546264 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:57.550158 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:57.550129 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86ce378f-511a-4301-a0c7-85ec5027f0c5-tls-certs\") pod \"test-model-5dbd39-kserve-85ddc47969-7947j\" (UID: \"86ce378f-511a-4301-a0c7-85ec5027f0c5\") " pod="ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j" Oct 31 13:25:57.550290 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:57.550273 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86ce378f-511a-4301-a0c7-85ec5027f0c5-dshm\") pod \"test-model-5dbd39-kserve-85ddc47969-7947j\" (UID: \"86ce378f-511a-4301-a0c7-85ec5027f0c5\") " pod="ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j" Oct 31 13:25:57.550336 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:57.550305 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86ce378f-511a-4301-a0c7-85ec5027f0c5-model-cache\") pod \"test-model-5dbd39-kserve-85ddc47969-7947j\" (UID: \"86ce378f-511a-4301-a0c7-85ec5027f0c5\") " pod="ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j" Oct 31 13:25:57.550509 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:57.550494 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqhwb\" (UniqueName: \"kubernetes.io/projected/86ce378f-511a-4301-a0c7-85ec5027f0c5-kube-api-access-zqhwb\") pod \"test-model-5dbd39-kserve-85ddc47969-7947j\" (UID: \"86ce378f-511a-4301-a0c7-85ec5027f0c5\") " pod="ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j" Oct 31 13:25:57.550634 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:57.550623 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86ce378f-511a-4301-a0c7-85ec5027f0c5-tmp-dir\") pod \"test-model-5dbd39-kserve-85ddc47969-7947j\" (UID: \"86ce378f-511a-4301-a0c7-85ec5027f0c5\") " pod="ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j" Oct 31 13:25:57.550671 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:57.550661 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86ce378f-511a-4301-a0c7-85ec5027f0c5-home\") pod \"test-model-5dbd39-kserve-85ddc47969-7947j\" (UID: \"86ce378f-511a-4301-a0c7-85ec5027f0c5\") " pod="ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j" Oct 31 13:25:57.658008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:57.657269 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86ce378f-511a-4301-a0c7-85ec5027f0c5-tmp-dir\") pod \"test-model-5dbd39-kserve-85ddc47969-7947j\" (UID: \"86ce378f-511a-4301-a0c7-85ec5027f0c5\") " pod="ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j" Oct 31 13:25:57.658008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:57.657344 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86ce378f-511a-4301-a0c7-85ec5027f0c5-home\") pod \"test-model-5dbd39-kserve-85ddc47969-7947j\" (UID: \"86ce378f-511a-4301-a0c7-85ec5027f0c5\") " pod="ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j" Oct 31 13:25:57.658008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:57.657413 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86ce378f-511a-4301-a0c7-85ec5027f0c5-tls-certs\") pod \"test-model-5dbd39-kserve-85ddc47969-7947j\" (UID: \"86ce378f-511a-4301-a0c7-85ec5027f0c5\") " pod="ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j" Oct 31 13:25:57.658008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:57.657500 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86ce378f-511a-4301-a0c7-85ec5027f0c5-dshm\") pod \"test-model-5dbd39-kserve-85ddc47969-7947j\" (UID: \"86ce378f-511a-4301-a0c7-85ec5027f0c5\") " pod="ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j" Oct 31 13:25:57.658008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:57.657545 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86ce378f-511a-4301-a0c7-85ec5027f0c5-model-cache\") pod \"test-model-5dbd39-kserve-85ddc47969-7947j\" (UID: \"86ce378f-511a-4301-a0c7-85ec5027f0c5\") " pod="ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j" Oct 31 13:25:57.658008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:57.657750 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqhwb\" (UniqueName: \"kubernetes.io/projected/86ce378f-511a-4301-a0c7-85ec5027f0c5-kube-api-access-zqhwb\") pod \"test-model-5dbd39-kserve-85ddc47969-7947j\" (UID: \"86ce378f-511a-4301-a0c7-85ec5027f0c5\") " pod="ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j" Oct 31 13:25:57.681052 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:57.658589 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86ce378f-511a-4301-a0c7-85ec5027f0c5-tmp-dir\") pod \"test-model-5dbd39-kserve-85ddc47969-7947j\" (UID: \"86ce378f-511a-4301-a0c7-85ec5027f0c5\") " pod="ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j" Oct 31 13:25:57.681052 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:57.658860 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86ce378f-511a-4301-a0c7-85ec5027f0c5-home\") pod \"test-model-5dbd39-kserve-85ddc47969-7947j\" (UID: \"86ce378f-511a-4301-a0c7-85ec5027f0c5\") " pod="ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j" Oct 31 13:25:57.681052 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:57.674877 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86ce378f-511a-4301-a0c7-85ec5027f0c5-model-cache\") pod \"test-model-5dbd39-kserve-85ddc47969-7947j\" (UID: \"86ce378f-511a-4301-a0c7-85ec5027f0c5\") " pod="ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j" Oct 31 13:25:57.776517 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:57.776405 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqhwb\" (UniqueName: \"kubernetes.io/projected/86ce378f-511a-4301-a0c7-85ec5027f0c5-kube-api-access-zqhwb\") pod \"test-model-5dbd39-kserve-85ddc47969-7947j\" (UID: \"86ce378f-511a-4301-a0c7-85ec5027f0c5\") " pod="ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j" Oct 31 13:25:57.776517 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:57.776376 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86ce378f-511a-4301-a0c7-85ec5027f0c5-tls-certs\") pod \"test-model-5dbd39-kserve-85ddc47969-7947j\" (UID: \"86ce378f-511a-4301-a0c7-85ec5027f0c5\") " pod="ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j" Oct 31 13:25:57.777092 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:57.776658 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:57.778032 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:57.777922 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86ce378f-511a-4301-a0c7-85ec5027f0c5-dshm\") pod \"test-model-5dbd39-kserve-85ddc47969-7947j\" (UID: \"86ce378f-511a-4301-a0c7-85ec5027f0c5\") " pod="ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j" Oct 31 13:25:58.039804 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:58.039768 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j" Oct 31 13:25:58.333345 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:25:58.333302 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86ce378f_511a_4301_a0c7_85ec5027f0c5.slice/crio-0eb3b7ba0a976276e4a87a655151fa94eb7744e64082508846e849adc275d311 WatchSource:0}: Error finding container 0eb3b7ba0a976276e4a87a655151fa94eb7744e64082508846e849adc275d311: Status 404 returned error can't find the container with id 0eb3b7ba0a976276e4a87a655151fa94eb7744e64082508846e849adc275d311 Oct 31 13:25:58.336648 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:58.336621 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j"] Oct 31 13:25:58.588517 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:58.588435 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:58.910442 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:58.910342 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j" event={"ID":"86ce378f-511a-4301-a0c7-85ec5027f0c5","Type":"ContainerStarted","Data":"fefc27ffe1f8009b18ef35dbb73dbff171c885744bbf0a763e1806f6d7bcbc7d"} Oct 31 13:25:58.910442 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:58.910409 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j" event={"ID":"86ce378f-511a-4301-a0c7-85ec5027f0c5","Type":"ContainerStarted","Data":"0eb3b7ba0a976276e4a87a655151fa94eb7744e64082508846e849adc275d311"} Oct 31 13:25:58.943247 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:58.943189 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j" podStartSLOduration=1.943170432 podStartE2EDuration="1.943170432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:25:58.933792151 +0000 UTC m=+3205.416915052" watchObservedRunningTime="2026-10-31 13:25:58.943170432 +0000 UTC m=+3205.426293336" Oct 31 13:25:59.037937 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:59.037886 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz"] Oct 31 13:25:59.045180 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:59.045148 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j" Oct 31 13:25:59.045374 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:59.045266 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz" Oct 31 13:25:59.058078 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:59.058048 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-384dbe\"/\"test-model-384dbe-kserve-self-signed-certs\"" Oct 31 13:25:59.058239 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:59.058143 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-384dbe\"/\"default-dockercfg-qdftm\"" Oct 31 13:25:59.060240 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:59.060216 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz"] Oct 31 13:25:59.060969 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:59.060948 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-384dbe\"/\"kube-root-ca.crt\"" Oct 31 13:25:59.060969 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:59.060953 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-384dbe\"/\"openshift-service-ca.crt\"" Oct 31 13:25:59.199932 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:59.192615 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:59.208541 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:59.208508 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77r59\" (UniqueName: \"kubernetes.io/projected/28f4f307-72eb-4607-8ca9-547bdecd1ed0-kube-api-access-77r59\") pod \"test-model-384dbe-kserve-7f464759f5-gnqhz\" (UID: \"28f4f307-72eb-4607-8ca9-547bdecd1ed0\") " pod="ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz" Oct 31 13:25:59.208718 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:59.208561 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28f4f307-72eb-4607-8ca9-547bdecd1ed0-model-cache\") pod \"test-model-384dbe-kserve-7f464759f5-gnqhz\" (UID: \"28f4f307-72eb-4607-8ca9-547bdecd1ed0\") " pod="ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz" Oct 31 13:25:59.208718 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:59.208671 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28f4f307-72eb-4607-8ca9-547bdecd1ed0-tls-certs\") pod \"test-model-384dbe-kserve-7f464759f5-gnqhz\" (UID: \"28f4f307-72eb-4607-8ca9-547bdecd1ed0\") " pod="ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz" Oct 31 13:25:59.208868 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:59.208845 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28f4f307-72eb-4607-8ca9-547bdecd1ed0-dshm\") pod \"test-model-384dbe-kserve-7f464759f5-gnqhz\" (UID: \"28f4f307-72eb-4607-8ca9-547bdecd1ed0\") " pod="ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz" Oct 31 13:25:59.208962 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:59.208876 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28f4f307-72eb-4607-8ca9-547bdecd1ed0-home\") pod \"test-model-384dbe-kserve-7f464759f5-gnqhz\" (UID: \"28f4f307-72eb-4607-8ca9-547bdecd1ed0\") " pod="ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz" Oct 31 13:25:59.208962 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:59.208922 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28f4f307-72eb-4607-8ca9-547bdecd1ed0-tmp-dir\") pod \"test-model-384dbe-kserve-7f464759f5-gnqhz\" (UID: \"28f4f307-72eb-4607-8ca9-547bdecd1ed0\") " pod="ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz" Oct 31 13:25:59.314278 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:59.314236 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28f4f307-72eb-4607-8ca9-547bdecd1ed0-dshm\") pod \"test-model-384dbe-kserve-7f464759f5-gnqhz\" (UID: \"28f4f307-72eb-4607-8ca9-547bdecd1ed0\") " pod="ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz" Oct 31 13:25:59.315875 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:59.315849 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28f4f307-72eb-4607-8ca9-547bdecd1ed0-home\") pod \"test-model-384dbe-kserve-7f464759f5-gnqhz\" (UID: \"28f4f307-72eb-4607-8ca9-547bdecd1ed0\") " pod="ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz" Oct 31 13:25:59.316529 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:59.316400 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28f4f307-72eb-4607-8ca9-547bdecd1ed0-home\") pod \"test-model-384dbe-kserve-7f464759f5-gnqhz\" (UID: \"28f4f307-72eb-4607-8ca9-547bdecd1ed0\") " pod="ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz" Oct 31 13:25:59.316944 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:59.316904 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28f4f307-72eb-4607-8ca9-547bdecd1ed0-tmp-dir\") pod \"test-model-384dbe-kserve-7f464759f5-gnqhz\" (UID: \"28f4f307-72eb-4607-8ca9-547bdecd1ed0\") " pod="ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz" Oct 31 13:25:59.317074 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:59.316477 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28f4f307-72eb-4607-8ca9-547bdecd1ed0-tmp-dir\") pod \"test-model-384dbe-kserve-7f464759f5-gnqhz\" (UID: \"28f4f307-72eb-4607-8ca9-547bdecd1ed0\") " pod="ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz" Oct 31 13:25:59.317755 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:59.317733 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77r59\" (UniqueName: \"kubernetes.io/projected/28f4f307-72eb-4607-8ca9-547bdecd1ed0-kube-api-access-77r59\") pod \"test-model-384dbe-kserve-7f464759f5-gnqhz\" (UID: \"28f4f307-72eb-4607-8ca9-547bdecd1ed0\") " pod="ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz" Oct 31 13:25:59.329937 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:59.329885 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28f4f307-72eb-4607-8ca9-547bdecd1ed0-model-cache\") pod \"test-model-384dbe-kserve-7f464759f5-gnqhz\" (UID: \"28f4f307-72eb-4607-8ca9-547bdecd1ed0\") " pod="ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz" Oct 31 13:25:59.330198 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:59.330177 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28f4f307-72eb-4607-8ca9-547bdecd1ed0-model-cache\") pod \"test-model-384dbe-kserve-7f464759f5-gnqhz\" (UID: \"28f4f307-72eb-4607-8ca9-547bdecd1ed0\") " pod="ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz" Oct 31 13:25:59.330408 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:59.330392 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28f4f307-72eb-4607-8ca9-547bdecd1ed0-tls-certs\") pod \"test-model-384dbe-kserve-7f464759f5-gnqhz\" (UID: \"28f4f307-72eb-4607-8ca9-547bdecd1ed0\") " pod="ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz" Oct 31 13:25:59.353997 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:59.352798 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28f4f307-72eb-4607-8ca9-547bdecd1ed0-tls-certs\") pod \"test-model-384dbe-kserve-7f464759f5-gnqhz\" (UID: \"28f4f307-72eb-4607-8ca9-547bdecd1ed0\") " pod="ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz" Oct 31 13:25:59.402162 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:59.369267 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77r59\" (UniqueName: \"kubernetes.io/projected/28f4f307-72eb-4607-8ca9-547bdecd1ed0-kube-api-access-77r59\") pod \"test-model-384dbe-kserve-7f464759f5-gnqhz\" (UID: \"28f4f307-72eb-4607-8ca9-547bdecd1ed0\") " pod="ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz" Oct 31 13:25:59.402162 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:59.380519 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28f4f307-72eb-4607-8ca9-547bdecd1ed0-dshm\") pod \"test-model-384dbe-kserve-7f464759f5-gnqhz\" (UID: \"28f4f307-72eb-4607-8ca9-547bdecd1ed0\") " pod="ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz" Oct 31 13:25:59.430045 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:59.427697 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:25:59.666161 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:59.666096 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz" Oct 31 13:25:59.921188 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:59.921113 6841 generic.go:356] "Generic (PLEG): container finished" podID="4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a" containerID="63c97c8347b67d1012a4027d06e9e76aebdbe0a24a2dc927092c2205621636bb" exitCode=0 Oct 31 13:25:59.921563 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:59.921181 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" event={"ID":"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a","Type":"ContainerDied","Data":"63c97c8347b67d1012a4027d06e9e76aebdbe0a24a2dc927092c2205621636bb"} Oct 31 13:25:59.921835 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:59.921820 6841 scope.go:122] "RemoveContainer" containerID="63c97c8347b67d1012a4027d06e9e76aebdbe0a24a2dc927092c2205621636bb" Oct 31 13:25:59.923726 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:59.923692 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:25:59.923863 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:59.923776 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:25:59.923863 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:59.923824 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:25:59.971931 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:25:59.971902 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz"] Oct 31 13:25:59.977865 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:25:59.976939 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28f4f307_72eb_4607_8ca9_547bdecd1ed0.slice/crio-f290963236661374aaeee855fab70b959a7134af90e03b775d569a3a94f52978 WatchSource:0}: Error finding container f290963236661374aaeee855fab70b959a7134af90e03b775d569a3a94f52978: Status 404 returned error can't find the container with id f290963236661374aaeee855fab70b959a7134af90e03b775d569a3a94f52978 Oct 31 13:26:00.147718 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:00.147362 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:00.931169 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:00.931134 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" event={"ID":"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a","Type":"ContainerStarted","Data":"051bfea054619709544afe81b0c0a83a4440abdd9289e0d9b9b0bdaadb675215"} Oct 31 13:26:00.932954 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:00.932926 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz" event={"ID":"28f4f307-72eb-4607-8ca9-547bdecd1ed0","Type":"ContainerStarted","Data":"d7c128aa94aff8db71f57bc1e469ef7c6fec00e0f2d60ec00cc871bc7d9c2043"} Oct 31 13:26:00.933103 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:00.932965 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz" event={"ID":"28f4f307-72eb-4607-8ca9-547bdecd1ed0","Type":"ContainerStarted","Data":"f290963236661374aaeee855fab70b959a7134af90e03b775d569a3a94f52978"} Oct 31 13:26:01.068046 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:01.050920 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:01.083335 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:01.083278 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz" podStartSLOduration=2.0832608 podStartE2EDuration="2.0832608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:25:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:26:01.01348183 +0000 UTC m=+3207.496604731" watchObservedRunningTime="2026-10-31 13:26:01.0832608 +0000 UTC m=+3207.566383702" Oct 31 13:26:01.622384 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:01.622349 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:26:01.622596 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:01.622569 6841 patch_prober.go:29] interesting pod/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 31 13:26:01.622645 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:01.622624 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" podUID="4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 31 13:26:01.670150 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:01.670116 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz" Oct 31 13:26:01.829650 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:01.829347 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:02.597541 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:02.597378 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:02.623476 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:02.623443 6841 patch_prober.go:29] interesting pod/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 31 13:26:02.623658 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:02.623493 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" podUID="4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 31 13:26:03.424729 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:03.424641 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:03.622788 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:03.622753 6841 patch_prober.go:29] interesting pod/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 31 13:26:03.623273 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:03.622812 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" podUID="4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 31 13:26:04.228996 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:04.227142 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:04.623519 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:04.623441 6841 patch_prober.go:29] interesting pod/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 31 13:26:04.623519 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:04.623490 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" podUID="4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 31 13:26:05.032756 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:05.032715 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:05.623559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:05.623525 6841 patch_prober.go:29] interesting pod/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 31 13:26:05.624018 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:05.623582 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" podUID="4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 31 13:26:05.717042 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:05.717005 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:06.545597 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:06.545542 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:06.628662 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:06.628629 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:26:06.629216 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:06.629190 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:26:06.629925 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:06.629906 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:26:07.403007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:07.390694 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:08.040043 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:08.040009 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j" Oct 31 13:26:08.053287 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:08.053264 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j" Oct 31 13:26:08.113220 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:08.113190 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j" Oct 31 13:26:08.218690 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:08.218656 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:08.948328 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:08.948285 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:09.603583 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:09.603546 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:09.666768 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:09.666738 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz" Oct 31 13:26:09.680103 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:09.680076 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz" Oct 31 13:26:10.129079 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:10.129051 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz" Oct 31 13:26:10.339767 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:10.339729 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:11.203887 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:11.203852 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:12.082084 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:12.073939 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:12.803136 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:12.802186 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:13.623802 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:13.621281 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:14.906509 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:14.906472 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:16.317912 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:16.317880 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:16.478901 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:16.478078 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" podUID="da83bd57-52af-48d3-bd76-42e36c18112f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:26:17.192045 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:17.190848 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:18.027239 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:18.027202 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:18.929779 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:18.925611 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:19.718321 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:19.718280 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:20.036559 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:26:20.036484 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:26:20.403059 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:20.400334 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:21.147012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:21.140783 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:21.985786 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:21.985755 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:22.635412 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:22.635377 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:23.349536 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:23.349500 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:24.149022 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:24.148917 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:24.802452 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:24.802409 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:25.737267 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:25.737231 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:26.062234 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:26.062141 6841 patch_prober.go:29] interesting pod/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:26:26.062234 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:26.062200 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" podUID="c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:26:26.427856 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:26.427769 6841 generic.go:356] "Generic (PLEG): container finished" podID="c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f" containerID="79b46d3cdeee3c8947262d926fcde8e152aabb749f88e4d1d46db61ea61822a5" exitCode=0 Oct 31 13:26:26.427856 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:26.427841 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" event={"ID":"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f","Type":"ContainerDied","Data":"79b46d3cdeee3c8947262d926fcde8e152aabb749f88e4d1d46db61ea61822a5"} Oct 31 13:26:26.428400 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:26.428366 6841 scope.go:122] "RemoveContainer" containerID="79b46d3cdeee3c8947262d926fcde8e152aabb749f88e4d1d46db61ea61822a5" Oct 31 13:26:26.430678 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:26.430644 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:26:26.430785 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:26.430722 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:26:26.430785 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:26.430757 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:26:26.557509 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:26.555016 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:27.214744 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:27.214709 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:27.447543 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:27.446806 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" event={"ID":"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f","Type":"ContainerStarted","Data":"ffebabf8fa00e2f3db0f57152935bac2b6c85f742bcb5638725aa7bb053c6f0b"} Oct 31 13:26:28.016044 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:28.005220 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:28.166479 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:28.166435 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:26:28.166658 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:28.166574 6841 patch_prober.go:29] interesting pod/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 31 13:26:28.166658 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:28.166632 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" podUID="c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 31 13:26:28.738014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:28.736085 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:29.166647 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:29.166551 6841 patch_prober.go:29] interesting pod/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 31 13:26:29.166647 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:29.166624 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" podUID="c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 31 13:26:29.512792 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:29.512755 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:30.167008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:30.166480 6841 patch_prober.go:29] interesting pod/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 31 13:26:30.167008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:30.166539 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" podUID="c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 31 13:26:30.378545 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:30.378511 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:31.132828 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:31.132790 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:31.167809 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:31.167773 6841 patch_prober.go:29] interesting pod/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 31 13:26:31.168432 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:31.168399 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" podUID="c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 31 13:26:32.038064 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:32.038026 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:32.166459 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:32.166417 6841 patch_prober.go:29] interesting pod/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 31 13:26:32.166671 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:32.166496 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" podUID="c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 31 13:26:32.766083 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:32.766043 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:33.165583 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:33.165505 6841 patch_prober.go:29] interesting pod/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 31 13:26:33.165583 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:33.165564 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" podUID="c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 31 13:26:33.508066 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:33.507963 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:34.171899 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:34.171867 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:26:34.172351 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:34.172332 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:26:34.172950 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:34.172933 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:26:34.416011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:34.415650 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:35.155841 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:35.155802 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:35.942203 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:35.942164 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:36.734451 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:36.729858 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:37.518551 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:37.518512 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:38.439051 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:38.439017 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:39.127504 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:39.127415 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:39.403966 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:39.403874 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:39.918399 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:39.912707 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:40.727118 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:40.727078 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:41.667009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:41.666312 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:41.882478 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:41.882440 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:42.213858 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:42.213815 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-93e2ca/e2e-usage-sim-e84206-869bc5dff9-2qgqm"] Oct 31 13:26:42.214396 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:42.214360 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-93e2ca/e2e-usage-sim-e84206-869bc5dff9-2qgqm" podUID="e2f041ef-d532-4840-936f-895135d81e19" containerName="sim" containerID="cri-o://5a5f866245fca7a38642bec2b963c458fd0b786f266ee7f9888ea3c40373bf7e" gracePeriod=30 Oct 31 13:26:42.591210 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:42.589977 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:42.691823 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:42.691793 6841 generic.go:356] "Generic (PLEG): container finished" podID="e2f041ef-d532-4840-936f-895135d81e19" containerID="5a5f866245fca7a38642bec2b963c458fd0b786f266ee7f9888ea3c40373bf7e" exitCode=0 Oct 31 13:26:42.692022 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:42.691867 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-93e2ca/e2e-usage-sim-e84206-869bc5dff9-2qgqm" event={"ID":"e2f041ef-d532-4840-936f-895135d81e19","Type":"ContainerDied","Data":"5a5f866245fca7a38642bec2b963c458fd0b786f266ee7f9888ea3c40373bf7e"} Oct 31 13:26:42.790029 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:42.790001 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-93e2ca/e2e-usage-sim-e84206-869bc5dff9-2qgqm" Oct 31 13:26:42.836590 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:42.836559 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2f041ef-d532-4840-936f-895135d81e19-kube-api-access-mp8fb\" (UniqueName: \"kubernetes.io/projected/e2f041ef-d532-4840-936f-895135d81e19-kube-api-access-mp8fb\") pod \"e2f041ef-d532-4840-936f-895135d81e19\" (UID: \"e2f041ef-d532-4840-936f-895135d81e19\") " Oct 31 13:26:42.836841 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:42.836828 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2f041ef-d532-4840-936f-895135d81e19-tls\" (UniqueName: \"kubernetes.io/secret/e2f041ef-d532-4840-936f-895135d81e19-tls\") pod \"e2f041ef-d532-4840-936f-895135d81e19\" (UID: \"e2f041ef-d532-4840-936f-895135d81e19\") " Oct 31 13:26:42.838798 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:42.838769 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2f041ef-d532-4840-936f-895135d81e19-kube-api-access-mp8fb" pod "e2f041ef-d532-4840-936f-895135d81e19" (UID: "e2f041ef-d532-4840-936f-895135d81e19"). InnerVolumeSpecName "kube-api-access-mp8fb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:26:42.838916 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:42.838835 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2f041ef-d532-4840-936f-895135d81e19-tls" pod "e2f041ef-d532-4840-936f-895135d81e19" (UID: "e2f041ef-d532-4840-936f-895135d81e19"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:26:42.938925 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:42.938894 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp8fb\" (UniqueName: \"kubernetes.io/projected/e2f041ef-d532-4840-936f-895135d81e19-kube-api-access-mp8fb\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:42.938925 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:42.938924 6841 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e2f041ef-d532-4840-936f-895135d81e19-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:43.175087 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:43.172758 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:43.706549 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:43.706511 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-93e2ca/e2e-usage-sim-e84206-869bc5dff9-2qgqm" Oct 31 13:26:43.706549 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:43.706537 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-93e2ca/e2e-usage-sim-e84206-869bc5dff9-2qgqm" event={"ID":"e2f041ef-d532-4840-936f-895135d81e19","Type":"ContainerDied","Data":"17b297fa001833a572d06790442adf03309e0915eb6bc6a8dd88f8b0b69e6e4b"} Oct 31 13:26:43.706818 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:43.706596 6841 scope.go:122] "RemoveContainer" containerID="5a5f866245fca7a38642bec2b963c458fd0b786f266ee7f9888ea3c40373bf7e" Oct 31 13:26:43.734597 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:43.734571 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-93e2ca/e2e-usage-sim-e84206-869bc5dff9-2qgqm"] Oct 31 13:26:43.738787 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:43.738764 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-93e2ca/e2e-usage-sim-e84206-869bc5dff9-2qgqm"] Oct 31 13:26:43.839998 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:43.839935 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:44.049004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:44.047431 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:44.317599 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:44.315104 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:44.531008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:44.530537 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:44.761122 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:44.761086 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:45.376704 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:45.376674 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:45.646287 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:45.646203 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2f041ef-d532-4840-936f-895135d81e19" path="/var/lib/kubelet/pods/e2f041ef-d532-4840-936f-895135d81e19/volumes" Oct 31 13:26:46.115146 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:46.115110 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:46.932335 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:46.932276 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:47.635945 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:47.635910 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:48.309776 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:48.309640 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:49.117734 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:49.117698 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:49.850920 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:49.850886 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:50.220871 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:26:50.220839 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:26:50.642364 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:50.642217 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:51.500260 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:51.499932 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:51.733416 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:51.733383 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:52.077714 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:52.077681 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:52.360696 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:52.360611 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:52.642221 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:52.632578 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:53.319051 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:53.318010 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:53.521024 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:53.520949 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:54.250142 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:54.250083 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:56.527313 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:56.527274 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:57.439367 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:57.439327 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:57.787004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:57.778493 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:58.618799 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:58.618766 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:26:59.374835 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:26:59.374798 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:00.282311 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:00.282274 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:01.486006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:01.485153 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:02.431002 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:02.427118 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:03.429645 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:03.429615 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:04.197382 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:04.197342 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:04.909246 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:04.909151 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:05.674930 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:05.674894 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:06.427501 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:06.427463 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:07.249806 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:07.249752 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:08.231519 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:08.231482 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:08.975410 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:08.975251 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:09.911895 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:09.911860 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:10.724345 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:10.724312 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:11.669545 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:11.669502 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:12.623068 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:12.623031 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:13.596909 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:13.596862 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:14.741669 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:14.741627 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:15.156546 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:15.156455 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:15.843394 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:15.843356 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:16.621854 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:16.621814 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:17.467782 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:17.467751 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:17.951118 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:17.951073 6841 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-70.us-east-2.compute.internal" podUID="578d51cfd2278b7149c7a4e11f7c55d2" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Oct 31 13:27:18.113280 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:18.113244 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:18.845852 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:18.845816 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:19.631284 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:19.631251 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:20.036160 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:27:20.036125 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:27:20.356698 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:20.356610 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:21.138603 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:21.138523 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:22.032023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:22.031829 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:22.247246 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:22.247208 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:23.066213 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:23.066166 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:24.125701 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:24.125641 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:24.883513 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:24.883457 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:25.635470 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:25.635437 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:26.528189 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:26.528155 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:27.217558 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:27.217517 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:27.957251 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:27.955337 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:28.723664 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:28.723604 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:29.413385 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:29.413350 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:30.225873 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:30.225839 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:30.933303 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:30.933261 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:31.849293 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:31.849230 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:32.621441 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:32.621403 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:33.334169 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:33.334131 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:34.114870 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:34.114833 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:34.842185 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:34.842091 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:35.640545 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:35.640501 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:36.411201 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:36.411152 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:37.238434 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:37.236591 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:37.793912 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:37.793885 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:38.051478 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:38.051383 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:38.685880 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:38.685818 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:39.454785 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:39.454749 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:39.653899 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:39.653852 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:40.324685 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:40.324648 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:41.004497 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:41.004209 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:41.834448 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:41.834412 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:42.714128 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:42.714094 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:43.459306 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:43.459260 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:44.233793 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:44.233660 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:45.106233 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:45.106196 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:45.924836 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:45.924799 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:46.743003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:46.737510 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:47.779040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:47.777347 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:48.596845 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:48.596811 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:49.532179 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:49.532131 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:50.036085 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:27:50.036050 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:27:51.090007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:51.078530 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:51.116007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:51.115069 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j" podUID="86ce378f-511a-4301-a0c7-85ec5027f0c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.212:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:27:52.243005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:52.238518 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:52.442233 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:52.442192 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:53.145412 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:53.145371 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:54.142735 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:54.142691 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:55.040737 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:55.040697 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:55.949233 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:55.949199 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:56.739797 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:56.739763 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:57.819054 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:57.818957 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:58.531535 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:58.531499 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:27:59.339935 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:27:59.339889 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:00.214430 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:00.214387 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:01.027182 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:01.027134 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:01.808005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:01.804482 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:02.606805 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:02.606771 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:03.040808 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:03.040774 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:03.560008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:03.541013 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:03.774018 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:03.773332 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:03.978883 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:03.978847 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:04.421184 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:04.421087 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:05.233796 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:05.233756 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:06.344158 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:06.344111 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:07.218799 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:07.218766 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:08.339785 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:08.339756 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:09.125800 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:09.125740 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:10.048735 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:10.048698 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:10.925162 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:10.925123 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:11.630301 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:11.630267 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:12.337150 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:12.337114 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:13.097736 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:13.097685 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:13.855317 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:13.855279 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:14.690186 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:14.690151 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:15.472073 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:15.472041 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:16.334053 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:16.331054 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:17.161331 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:17.161143 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:17.396199 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:17.396160 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2"] Oct 31 13:28:17.398135 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:17.398104 6841 cpu_manager.go:414] "containerMap: removing container" podUID="e2f041ef-d532-4840-936f-895135d81e19" containerName="sim" Oct 31 13:28:17.398313 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:17.398298 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2f041ef-d532-4840-936f-895135d81e19" containerName="sim" Oct 31 13:28:17.398632 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:17.398616 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2f041ef-d532-4840-936f-895135d81e19" containerName="sim" Oct 31 13:28:17.422496 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:17.422422 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2" Oct 31 13:28:17.471003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:17.463362 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-842ac9\"/\"payload-processing-external-model-plugins-e2e-extmod-a-842ac9\"" Oct 31 13:28:17.471003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:17.463536 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-842ac9\"/\"payload-processing-external-model-e2e--64249880-dockercfg-l2zw7\"" Oct 31 13:28:17.471003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:17.463593 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-842ac9\"/\"agc-cred-7d5d1d5a1265\"" Oct 31 13:28:17.471003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:17.463665 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-842ac9\"/\"routing-overlay\"" Oct 31 13:28:17.471003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:17.463763 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-842ac9\"/\"kube-root-ca.crt\"" Oct 31 13:28:17.471003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:17.463849 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-842ac9\"/\"openshift-service-ca.crt\"" Oct 31 13:28:17.471003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:17.465854 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:17.526001 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:17.524251 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2da1d417-1661-4b1e-a5fd-3b75cf1dc30e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2\" (UID: \"2da1d417-1661-4b1e-a5fd-3b75cf1dc30e\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2" Oct 31 13:28:17.526001 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:17.524358 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwhsv\" (UniqueName: \"kubernetes.io/projected/2da1d417-1661-4b1e-a5fd-3b75cf1dc30e-kube-api-access-wwhsv\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2\" (UID: \"2da1d417-1661-4b1e-a5fd-3b75cf1dc30e\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2" Oct 31 13:28:17.526001 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:17.524569 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2da1d417-1661-4b1e-a5fd-3b75cf1dc30e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2\" (UID: \"2da1d417-1661-4b1e-a5fd-3b75cf1dc30e\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2" Oct 31 13:28:17.526001 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:17.524647 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2da1d417-1661-4b1e-a5fd-3b75cf1dc30e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2\" (UID: \"2da1d417-1661-4b1e-a5fd-3b75cf1dc30e\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2" Oct 31 13:28:17.539213 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:17.539175 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2"] Oct 31 13:28:17.631561 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:17.628813 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2da1d417-1661-4b1e-a5fd-3b75cf1dc30e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2\" (UID: \"2da1d417-1661-4b1e-a5fd-3b75cf1dc30e\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2" Oct 31 13:28:17.631561 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:17.628924 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwhsv\" (UniqueName: \"kubernetes.io/projected/2da1d417-1661-4b1e-a5fd-3b75cf1dc30e-kube-api-access-wwhsv\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2\" (UID: \"2da1d417-1661-4b1e-a5fd-3b75cf1dc30e\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2" Oct 31 13:28:17.631561 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:17.630242 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2da1d417-1661-4b1e-a5fd-3b75cf1dc30e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2\" (UID: \"2da1d417-1661-4b1e-a5fd-3b75cf1dc30e\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2" Oct 31 13:28:17.631561 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:17.630330 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2da1d417-1661-4b1e-a5fd-3b75cf1dc30e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2\" (UID: \"2da1d417-1661-4b1e-a5fd-3b75cf1dc30e\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2" Oct 31 13:28:17.633202 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:17.633174 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2da1d417-1661-4b1e-a5fd-3b75cf1dc30e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2\" (UID: \"2da1d417-1661-4b1e-a5fd-3b75cf1dc30e\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2" Oct 31 13:28:17.634038 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:17.633961 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2da1d417-1661-4b1e-a5fd-3b75cf1dc30e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2\" (UID: \"2da1d417-1661-4b1e-a5fd-3b75cf1dc30e\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2" Oct 31 13:28:17.636973 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:17.634897 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2da1d417-1661-4b1e-a5fd-3b75cf1dc30e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2\" (UID: \"2da1d417-1661-4b1e-a5fd-3b75cf1dc30e\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2" Oct 31 13:28:17.664001 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:17.663896 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwhsv\" (UniqueName: \"kubernetes.io/projected/2da1d417-1661-4b1e-a5fd-3b75cf1dc30e-kube-api-access-wwhsv\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2\" (UID: \"2da1d417-1661-4b1e-a5fd-3b75cf1dc30e\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2" Oct 31 13:28:17.906006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:17.905342 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2" Oct 31 13:28:18.679341 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:18.679284 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:18.754635 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:18.754573 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2"] Oct 31 13:28:18.755330 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:28:18.755306 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da1d417_1661_4b1e_a5fd_3b75cf1dc30e.slice/crio-14b3c8fb618a6f4ee03b436a059d06e5bdc8abdb462733307e0850b941c00dad WatchSource:0}: Error finding container 14b3c8fb618a6f4ee03b436a059d06e5bdc8abdb462733307e0850b941c00dad: Status 404 returned error can't find the container with id 14b3c8fb618a6f4ee03b436a059d06e5bdc8abdb462733307e0850b941c00dad Oct 31 13:28:19.272818 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:19.272780 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2" event={"ID":"2da1d417-1661-4b1e-a5fd-3b75cf1dc30e","Type":"ContainerStarted","Data":"fd33e4b99a38fe620d6b26a31e261beac0d049ba6eacac5788eecb39c17f6cb1"} Oct 31 13:28:19.273048 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:19.272834 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2" event={"ID":"2da1d417-1661-4b1e-a5fd-3b75cf1dc30e","Type":"ContainerStarted","Data":"14b3c8fb618a6f4ee03b436a059d06e5bdc8abdb462733307e0850b941c00dad"} Oct 31 13:28:19.273048 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:19.272860 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2" Oct 31 13:28:19.296897 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:19.296845 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2" podStartSLOduration=2.296831107 podStartE2EDuration="2.296831107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:19.293612314 +0000 UTC m=+3345.776735214" watchObservedRunningTime="2026-10-31 13:28:19.296831107 +0000 UTC m=+3345.779954008" Oct 31 13:28:19.635095 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:19.634311 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:20.038053 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:28:20.038022 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:28:20.740719 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:20.740681 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:21.616278 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:21.616248 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:22.355594 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:22.350934 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:23.322118 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:23.322080 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:24.116685 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:24.116645 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:24.939786 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:24.939741 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:25.749067 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:25.749034 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:26.580942 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:26.580890 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:27.409086 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:27.409050 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:28.136254 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:28.136216 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:28.829802 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:28.829768 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:29.593469 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:29.592492 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:30.324695 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:30.324655 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:30.354286 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:30.354258 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2" Oct 31 13:28:31.202023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:31.201971 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:31.932448 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:31.932392 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:32.631156 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:32.631119 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:33.494020 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:33.493953 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:34.336731 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:34.336701 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:34.640371 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:34.640288 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:35.085125 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:35.085088 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:35.828281 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:35.828241 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:36.670309 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:36.670275 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:37.635081 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:37.634969 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:38.372864 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:38.372830 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:39.262288 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:39.262241 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:39.522558 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:39.522444 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:39.709088 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:39.709052 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2"] Oct 31 13:28:39.871457 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:39.871368 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:40.269326 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:40.269289 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:40.536055 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:40.532558 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:40.747018 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:40.746966 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:41.411042 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:41.410964 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:42.165006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:42.164517 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:43.141602 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:43.141562 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:43.935237 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:43.935201 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:44.709425 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:44.709361 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2" podUID="2da1d417-1661-4b1e-a5fd-3b75cf1dc30e" containerName="payload-processing" containerID="cri-o://fd33e4b99a38fe620d6b26a31e261beac0d049ba6eacac5788eecb39c17f6cb1" gracePeriod=25 Oct 31 13:28:44.797425 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:44.797369 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:45.016107 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:45.016083 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2" Oct 31 13:28:45.155954 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:45.155918 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2da1d417-1661-4b1e-a5fd-3b75cf1dc30e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2da1d417-1661-4b1e-a5fd-3b75cf1dc30e-routing-overlay-volume\") pod \"2da1d417-1661-4b1e-a5fd-3b75cf1dc30e\" (UID: \"2da1d417-1661-4b1e-a5fd-3b75cf1dc30e\") " Oct 31 13:28:45.156204 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:45.156149 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2da1d417-1661-4b1e-a5fd-3b75cf1dc30e-kube-api-access-wwhsv\" (UniqueName: \"kubernetes.io/projected/2da1d417-1661-4b1e-a5fd-3b75cf1dc30e-kube-api-access-wwhsv\") pod \"2da1d417-1661-4b1e-a5fd-3b75cf1dc30e\" (UID: \"2da1d417-1661-4b1e-a5fd-3b75cf1dc30e\") " Oct 31 13:28:45.156364 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:45.156348 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2da1d417-1661-4b1e-a5fd-3b75cf1dc30e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2da1d417-1661-4b1e-a5fd-3b75cf1dc30e-provider-credentials\") pod \"2da1d417-1661-4b1e-a5fd-3b75cf1dc30e\" (UID: \"2da1d417-1661-4b1e-a5fd-3b75cf1dc30e\") " Oct 31 13:28:45.156439 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:45.156383 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2da1d417-1661-4b1e-a5fd-3b75cf1dc30e-routing-overlay-volume" pod "2da1d417-1661-4b1e-a5fd-3b75cf1dc30e" (UID: "2da1d417-1661-4b1e-a5fd-3b75cf1dc30e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:28:45.156539 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:45.156522 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2da1d417-1661-4b1e-a5fd-3b75cf1dc30e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2da1d417-1661-4b1e-a5fd-3b75cf1dc30e-plugins-config-volume\") pod \"2da1d417-1661-4b1e-a5fd-3b75cf1dc30e\" (UID: \"2da1d417-1661-4b1e-a5fd-3b75cf1dc30e\") " Oct 31 13:28:45.156900 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:45.156860 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2da1d417-1661-4b1e-a5fd-3b75cf1dc30e-plugins-config-volume" pod "2da1d417-1661-4b1e-a5fd-3b75cf1dc30e" (UID: "2da1d417-1661-4b1e-a5fd-3b75cf1dc30e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:28:45.157311 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:45.157296 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2da1d417-1661-4b1e-a5fd-3b75cf1dc30e-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:45.157377 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:45.157313 6841 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2da1d417-1661-4b1e-a5fd-3b75cf1dc30e-routing-overlay-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:45.158525 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:45.158502 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da1d417-1661-4b1e-a5fd-3b75cf1dc30e-kube-api-access-wwhsv" pod "2da1d417-1661-4b1e-a5fd-3b75cf1dc30e" (UID: "2da1d417-1661-4b1e-a5fd-3b75cf1dc30e"). InnerVolumeSpecName "kube-api-access-wwhsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:28:45.158585 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:45.158542 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da1d417-1661-4b1e-a5fd-3b75cf1dc30e-provider-credentials" pod "2da1d417-1661-4b1e-a5fd-3b75cf1dc30e" (UID: "2da1d417-1661-4b1e-a5fd-3b75cf1dc30e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:28:45.258710 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:45.258621 6841 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2da1d417-1661-4b1e-a5fd-3b75cf1dc30e-provider-credentials\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:45.258710 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:45.258653 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwhsv\" (UniqueName: \"kubernetes.io/projected/2da1d417-1661-4b1e-a5fd-3b75cf1dc30e-kube-api-access-wwhsv\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:45.530527 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:45.530437 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:45.619148 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:45.619111 6841 generic.go:356] "Generic (PLEG): container finished" podID="2da1d417-1661-4b1e-a5fd-3b75cf1dc30e" containerID="fd33e4b99a38fe620d6b26a31e261beac0d049ba6eacac5788eecb39c17f6cb1" exitCode=0 Oct 31 13:28:45.619350 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:45.619169 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2" Oct 31 13:28:45.619350 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:45.619198 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2" event={"ID":"2da1d417-1661-4b1e-a5fd-3b75cf1dc30e","Type":"ContainerDied","Data":"fd33e4b99a38fe620d6b26a31e261beac0d049ba6eacac5788eecb39c17f6cb1"} Oct 31 13:28:45.619350 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:45.619261 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2" event={"ID":"2da1d417-1661-4b1e-a5fd-3b75cf1dc30e","Type":"ContainerDied","Data":"14b3c8fb618a6f4ee03b436a059d06e5bdc8abdb462733307e0850b941c00dad"} Oct 31 13:28:45.619350 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:45.619292 6841 scope.go:122] "RemoveContainer" containerID="fd33e4b99a38fe620d6b26a31e261beac0d049ba6eacac5788eecb39c17f6cb1" Oct 31 13:28:45.632208 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:45.632182 6841 scope.go:122] "RemoveContainer" containerID="fd33e4b99a38fe620d6b26a31e261beac0d049ba6eacac5788eecb39c17f6cb1" Oct 31 13:28:45.632527 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:28:45.632504 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd33e4b99a38fe620d6b26a31e261beac0d049ba6eacac5788eecb39c17f6cb1\": container with ID starting with fd33e4b99a38fe620d6b26a31e261beac0d049ba6eacac5788eecb39c17f6cb1 not found: ID does not exist" containerID="fd33e4b99a38fe620d6b26a31e261beac0d049ba6eacac5788eecb39c17f6cb1" Oct 31 13:28:45.632613 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:45.632536 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd33e4b99a38fe620d6b26a31e261beac0d049ba6eacac5788eecb39c17f6cb1"} err="failed to get container status \"fd33e4b99a38fe620d6b26a31e261beac0d049ba6eacac5788eecb39c17f6cb1\": rpc error: code = NotFound desc = could not find container \"fd33e4b99a38fe620d6b26a31e261beac0d049ba6eacac5788eecb39c17f6cb1\": container with ID starting with fd33e4b99a38fe620d6b26a31e261beac0d049ba6eacac5788eecb39c17f6cb1 not found: ID does not exist" Oct 31 13:28:45.648296 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:45.648274 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2"] Oct 31 13:28:45.652561 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:45.652535 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bgvdx2"] Oct 31 13:28:46.227670 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:46.227637 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:47.018646 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:47.018613 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:47.646963 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:47.646925 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2da1d417-1661-4b1e-a5fd-3b75cf1dc30e" path="/var/lib/kubelet/pods/2da1d417-1661-4b1e-a5fd-3b75cf1dc30e/volumes" Oct 31 13:28:47.835003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:47.834954 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:48.041040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:48.041010 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:48.587744 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:48.587704 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:48.815850 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:48.815814 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:49.074004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:49.072520 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:49.359723 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:49.359525 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:49.973306 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:49.973262 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:50.039400 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:28:50.039368 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:28:50.664817 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:50.664721 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:50.891694 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:50.891659 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:51.219804 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:51.219770 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:51.878936 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:51.878900 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a13836-bb4868df9-9qxgz"] Oct 31 13:28:51.880241 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:51.880220 6841 cpu_manager.go:414] "containerMap: removing container" podUID="2da1d417-1661-4b1e-a5fd-3b75cf1dc30e" containerName="payload-processing" Oct 31 13:28:51.880323 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:51.880245 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2da1d417-1661-4b1e-a5fd-3b75cf1dc30e" containerName="payload-processing" Oct 31 13:28:51.880582 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:51.880562 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="2da1d417-1661-4b1e-a5fd-3b75cf1dc30e" containerName="payload-processing" Oct 31 13:28:51.887049 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:51.886962 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:51.888862 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:51.888836 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a13836-bb4868df9-9qxgz" Oct 31 13:28:51.893875 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:51.893850 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a13836-bb4868df9-9qxgz"] Oct 31 13:28:51.979184 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:51.973419 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-a13836-dockercfg-4j9gf\"" Oct 31 13:28:51.986808 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:51.979705 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-a13836\"" Oct 31 13:28:52.018447 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:52.018404 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61c9f172-a17d-4531-8626-6bd1433a9a39-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a13836-bb4868df9-9qxgz\" (UID: \"61c9f172-a17d-4531-8626-6bd1433a9a39\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a13836-bb4868df9-9qxgz" Oct 31 13:28:52.018865 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:52.018840 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d58h\" (UniqueName: \"kubernetes.io/projected/61c9f172-a17d-4531-8626-6bd1433a9a39-kube-api-access-9d58h\") pod \"payload-processing-e2e-inf-b-a13836-bb4868df9-9qxgz\" (UID: \"61c9f172-a17d-4531-8626-6bd1433a9a39\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a13836-bb4868df9-9qxgz" Oct 31 13:28:52.107096 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:52.097566 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:52.189216 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:52.127759 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61c9f172-a17d-4531-8626-6bd1433a9a39-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a13836-bb4868df9-9qxgz\" (UID: \"61c9f172-a17d-4531-8626-6bd1433a9a39\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a13836-bb4868df9-9qxgz" Oct 31 13:28:52.189678 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:52.189654 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d58h\" (UniqueName: \"kubernetes.io/projected/61c9f172-a17d-4531-8626-6bd1433a9a39-kube-api-access-9d58h\") pod \"payload-processing-e2e-inf-b-a13836-bb4868df9-9qxgz\" (UID: \"61c9f172-a17d-4531-8626-6bd1433a9a39\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a13836-bb4868df9-9qxgz" Oct 31 13:28:52.192056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:52.180906 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61c9f172-a17d-4531-8626-6bd1433a9a39-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a13836-bb4868df9-9qxgz\" (UID: \"61c9f172-a17d-4531-8626-6bd1433a9a39\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a13836-bb4868df9-9qxgz" Oct 31 13:28:52.283808 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:52.283738 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d58h\" (UniqueName: \"kubernetes.io/projected/61c9f172-a17d-4531-8626-6bd1433a9a39-kube-api-access-9d58h\") pod \"payload-processing-e2e-inf-b-a13836-bb4868df9-9qxgz\" (UID: \"61c9f172-a17d-4531-8626-6bd1433a9a39\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a13836-bb4868df9-9qxgz" Oct 31 13:28:52.507871 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:52.507839 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a13836-bb4868df9-9qxgz" Oct 31 13:28:52.813023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:52.812948 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a13836-bb4868df9-9qxgz"] Oct 31 13:28:52.814839 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:28:52.814018 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61c9f172_a17d_4531_8626_6bd1433a9a39.slice/crio-a2f6c3e7d3b75eb2914f807cbe3b6c6a752e1c961f33f194465a80c17e4fc554 WatchSource:0}: Error finding container a2f6c3e7d3b75eb2914f807cbe3b6c6a752e1c961f33f194465a80c17e4fc554: Status 404 returned error can't find the container with id a2f6c3e7d3b75eb2914f807cbe3b6c6a752e1c961f33f194465a80c17e4fc554 Oct 31 13:28:52.895336 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:52.895303 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:53.669865 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:53.669829 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:53.745046 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:53.744976 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a13836-bb4868df9-9qxgz" event={"ID":"61c9f172-a17d-4531-8626-6bd1433a9a39","Type":"ContainerStarted","Data":"1652a9dab04fee18b97355cbd9745471cfcdba9d1e880d91fe93575c3b5d8d4c"} Oct 31 13:28:53.745236 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:53.745074 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a13836-bb4868df9-9qxgz" event={"ID":"61c9f172-a17d-4531-8626-6bd1433a9a39","Type":"ContainerStarted","Data":"a2f6c3e7d3b75eb2914f807cbe3b6c6a752e1c961f33f194465a80c17e4fc554"} Oct 31 13:28:53.745236 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:53.745105 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a13836-bb4868df9-9qxgz" Oct 31 13:28:53.768723 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:53.768677 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-a13836-bb4868df9-9qxgz" podStartSLOduration=2.768663251 podStartE2EDuration="2.768663251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:53.764416448 +0000 UTC m=+3380.247539351" watchObservedRunningTime="2026-10-31 13:28:53.768663251 +0000 UTC m=+3380.251786158" Oct 31 13:28:54.536306 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:54.536241 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:55.261841 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:55.259272 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:56.011049 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:56.010966 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:56.843079 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:56.843036 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:57.611072 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:57.611037 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:58.430144 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:58.430093 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:28:59.403391 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:28:59.403357 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:00.266861 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:00.266822 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:01.671177 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:01.671140 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:02.847576 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:02.847535 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:03.724454 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:03.724418 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:04.735386 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:04.735347 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:04.757798 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:04.757763 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a13836-bb4868df9-9qxgz" Oct 31 13:29:05.415797 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:05.415749 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:05.665776 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:05.665741 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:05.878433 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:05.878401 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:06.734842 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:06.734755 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:06.955008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:06.950254 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:07.916361 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:07.916328 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:08.827918 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:08.827880 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:09.734042 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:09.734005 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:10.574994 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:10.574944 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:11.358555 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:11.358511 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:12.147347 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:12.147309 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:12.919782 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:12.919743 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:13.368344 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:13.368304 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x"] Oct 31 13:29:13.368848 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:13.368660 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" podUID="5bd82005-b57c-4b49-9bf5-b9a77c624c69" containerName="istio-proxy" containerID="cri-o://695d4845ce63682aac4c250910b1d13a2c7e6a6bc2d0c6bd326517895539c1a2" gracePeriod=30 Oct 31 13:29:13.814493 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:13.814457 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:14.482565 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:14.482504 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:15.434821 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:15.434776 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:16.212451 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:16.212412 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:16.906329 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:16.906289 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:17.733043 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:17.731076 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:17.980997 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:17.980945 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:18.237945 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.237912 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:18.771023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.770976 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" Oct 31 13:29:18.851267 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.850092 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:18.864702 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.864669 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h"] Oct 31 13:29:18.869926 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.869889 6841 cpu_manager.go:414] "containerMap: removing container" podUID="5bd82005-b57c-4b49-9bf5-b9a77c624c69" containerName="istio-proxy" Oct 31 13:29:18.870145 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.870129 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bd82005-b57c-4b49-9bf5-b9a77c624c69" containerName="istio-proxy" Oct 31 13:29:18.870403 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.870392 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bd82005-b57c-4b49-9bf5-b9a77c624c69" containerName="istio-proxy" Oct 31 13:29:18.874527 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.874484 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5bd82005-b57c-4b49-9bf5-b9a77c624c69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd82005-b57c-4b49-9bf5-b9a77c624c69-istio-podinfo\") pod \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\" (UID: \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\") " Oct 31 13:29:18.874707 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.874637 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bd82005-b57c-4b49-9bf5-b9a77c624c69-istio-token\" (UniqueName: \"kubernetes.io/projected/5bd82005-b57c-4b49-9bf5-b9a77c624c69-istio-token\") pod \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\" (UID: \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\") " Oct 31 13:29:18.874800 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.874770 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bd82005-b57c-4b49-9bf5-b9a77c624c69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd82005-b57c-4b49-9bf5-b9a77c624c69-istiod-ca-cert\") pod \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\" (UID: \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\") " Oct 31 13:29:18.874882 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.874854 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bd82005-b57c-4b49-9bf5-b9a77c624c69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd82005-b57c-4b49-9bf5-b9a77c624c69-workload-socket\") pod \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\" (UID: \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\") " Oct 31 13:29:18.875020 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.874966 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bd82005-b57c-4b49-9bf5-b9a77c624c69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd82005-b57c-4b49-9bf5-b9a77c624c69-workload-certs\") pod \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\" (UID: \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\") " Oct 31 13:29:18.875155 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.875137 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bd82005-b57c-4b49-9bf5-b9a77c624c69-kube-api-access-ztb2c\" (UniqueName: \"kubernetes.io/projected/5bd82005-b57c-4b49-9bf5-b9a77c624c69-kube-api-access-ztb2c\") pod \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\" (UID: \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\") " Oct 31 13:29:18.875313 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.875296 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bd82005-b57c-4b49-9bf5-b9a77c624c69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd82005-b57c-4b49-9bf5-b9a77c624c69-credential-socket\") pod \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\" (UID: \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\") " Oct 31 13:29:18.875464 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.875450 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bd82005-b57c-4b49-9bf5-b9a77c624c69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd82005-b57c-4b49-9bf5-b9a77c624c69-istio-envoy\") pod \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\" (UID: \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\") " Oct 31 13:29:18.875717 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.875698 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bd82005-b57c-4b49-9bf5-b9a77c624c69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd82005-b57c-4b49-9bf5-b9a77c624c69-istio-data\") pod \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\" (UID: \"5bd82005-b57c-4b49-9bf5-b9a77c624c69\") " Oct 31 13:29:18.876144 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.876121 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd82005-b57c-4b49-9bf5-b9a77c624c69-workload-socket" pod "5bd82005-b57c-4b49-9bf5-b9a77c624c69" (UID: "5bd82005-b57c-4b49-9bf5-b9a77c624c69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:18.876376 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.876351 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd82005-b57c-4b49-9bf5-b9a77c624c69-istio-data" pod "5bd82005-b57c-4b49-9bf5-b9a77c624c69" (UID: "5bd82005-b57c-4b49-9bf5-b9a77c624c69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:18.876590 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.876570 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd82005-b57c-4b49-9bf5-b9a77c624c69-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:18.876673 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.876594 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd82005-b57c-4b49-9bf5-b9a77c624c69-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:18.876801 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.876774 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h"] Oct 31 13:29:18.876935 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.876876 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" Oct 31 13:29:18.878666 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.878643 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd82005-b57c-4b49-9bf5-b9a77c624c69-credential-socket" pod "5bd82005-b57c-4b49-9bf5-b9a77c624c69" (UID: "5bd82005-b57c-4b49-9bf5-b9a77c624c69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:18.878960 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.878932 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bd82005-b57c-4b49-9bf5-b9a77c624c69-istio-token" pod "5bd82005-b57c-4b49-9bf5-b9a77c624c69" (UID: "5bd82005-b57c-4b49-9bf5-b9a77c624c69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:29:18.879253 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.879146 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bd82005-b57c-4b49-9bf5-b9a77c624c69-istio-podinfo" pod "5bd82005-b57c-4b49-9bf5-b9a77c624c69" (UID: "5bd82005-b57c-4b49-9bf5-b9a77c624c69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:29:18.879395 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.879373 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd82005-b57c-4b49-9bf5-b9a77c624c69-workload-certs" pod "5bd82005-b57c-4b49-9bf5-b9a77c624c69" (UID: "5bd82005-b57c-4b49-9bf5-b9a77c624c69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:18.879457 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.879390 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bd82005-b57c-4b49-9bf5-b9a77c624c69-istiod-ca-cert" pod "5bd82005-b57c-4b49-9bf5-b9a77c624c69" (UID: "5bd82005-b57c-4b49-9bf5-b9a77c624c69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:29:18.880295 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.880270 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd82005-b57c-4b49-9bf5-b9a77c624c69-istio-envoy" pod "5bd82005-b57c-4b49-9bf5-b9a77c624c69" (UID: "5bd82005-b57c-4b49-9bf5-b9a77c624c69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:18.881291 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.881275 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bd82005-b57c-4b49-9bf5-b9a77c624c69-kube-api-access-ztb2c" pod "5bd82005-b57c-4b49-9bf5-b9a77c624c69" (UID: "5bd82005-b57c-4b49-9bf5-b9a77c624c69"). InnerVolumeSpecName "kube-api-access-ztb2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:29:18.890607 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.890583 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5dbd39-openshift-default-dockercfg-l8k75\"" Oct 31 13:29:18.978018 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.977964 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31d9443c-5638-44ee-af89-e78960decfab-istiod-ca-cert\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h\" (UID: \"31d9443c-5638-44ee-af89-e78960decfab\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" Oct 31 13:29:18.978208 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.978107 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31d9443c-5638-44ee-af89-e78960decfab-istio-data\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h\" (UID: \"31d9443c-5638-44ee-af89-e78960decfab\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" Oct 31 13:29:18.978208 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.978153 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d9443c-5638-44ee-af89-e78960decfab-credential-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h\" (UID: \"31d9443c-5638-44ee-af89-e78960decfab\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" Oct 31 13:29:18.978339 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.978318 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31d9443c-5638-44ee-af89-e78960decfab-istio-envoy\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h\" (UID: \"31d9443c-5638-44ee-af89-e78960decfab\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" Oct 31 13:29:18.978396 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.978381 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d9443c-5638-44ee-af89-e78960decfab-workload-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h\" (UID: \"31d9443c-5638-44ee-af89-e78960decfab\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" Oct 31 13:29:18.978463 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.978446 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4lw2\" (UniqueName: \"kubernetes.io/projected/31d9443c-5638-44ee-af89-e78960decfab-kube-api-access-x4lw2\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h\" (UID: \"31d9443c-5638-44ee-af89-e78960decfab\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" Oct 31 13:29:18.978507 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.978480 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31d9443c-5638-44ee-af89-e78960decfab-istio-token\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h\" (UID: \"31d9443c-5638-44ee-af89-e78960decfab\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" Oct 31 13:29:18.978507 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.978499 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31d9443c-5638-44ee-af89-e78960decfab-workload-certs\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h\" (UID: \"31d9443c-5638-44ee-af89-e78960decfab\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" Oct 31 13:29:18.978684 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.978658 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31d9443c-5638-44ee-af89-e78960decfab-istio-podinfo\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h\" (UID: \"31d9443c-5638-44ee-af89-e78960decfab\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" Oct 31 13:29:18.978819 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.978805 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd82005-b57c-4b49-9bf5-b9a77c624c69-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:18.978856 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.978825 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd82005-b57c-4b49-9bf5-b9a77c624c69-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:18.978856 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.978843 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztb2c\" (UniqueName: \"kubernetes.io/projected/5bd82005-b57c-4b49-9bf5-b9a77c624c69-kube-api-access-ztb2c\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:18.978925 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.978858 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd82005-b57c-4b49-9bf5-b9a77c624c69-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:18.978925 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.978872 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd82005-b57c-4b49-9bf5-b9a77c624c69-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:18.978925 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.978888 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd82005-b57c-4b49-9bf5-b9a77c624c69-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:18.978925 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:18.978901 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd82005-b57c-4b49-9bf5-b9a77c624c69-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:19.078368 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:19.078335 6841 generic.go:356] "Generic (PLEG): container finished" podID="5bd82005-b57c-4b49-9bf5-b9a77c624c69" containerID="695d4845ce63682aac4c250910b1d13a2c7e6a6bc2d0c6bd326517895539c1a2" exitCode=0 Oct 31 13:29:19.078587 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:19.078409 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" Oct 31 13:29:19.078587 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:19.078425 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" event={"ID":"5bd82005-b57c-4b49-9bf5-b9a77c624c69","Type":"ContainerDied","Data":"695d4845ce63682aac4c250910b1d13a2c7e6a6bc2d0c6bd326517895539c1a2"} Oct 31 13:29:19.078587 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:19.078482 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x" event={"ID":"5bd82005-b57c-4b49-9bf5-b9a77c624c69","Type":"ContainerDied","Data":"ceab437bc96ff71f521af6678dc9590fc78e24951e42282eff5afd7f5fe0ecac"} Oct 31 13:29:19.078587 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:19.078505 6841 scope.go:122] "RemoveContainer" containerID="695d4845ce63682aac4c250910b1d13a2c7e6a6bc2d0c6bd326517895539c1a2" Oct 31 13:29:19.079868 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:19.079850 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31d9443c-5638-44ee-af89-e78960decfab-istio-data\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h\" (UID: \"31d9443c-5638-44ee-af89-e78960decfab\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" Oct 31 13:29:19.079967 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:19.079893 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d9443c-5638-44ee-af89-e78960decfab-credential-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h\" (UID: \"31d9443c-5638-44ee-af89-e78960decfab\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" Oct 31 13:29:19.080155 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:19.080137 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31d9443c-5638-44ee-af89-e78960decfab-istio-envoy\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h\" (UID: \"31d9443c-5638-44ee-af89-e78960decfab\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" Oct 31 13:29:19.080236 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:19.080212 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d9443c-5638-44ee-af89-e78960decfab-workload-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h\" (UID: \"31d9443c-5638-44ee-af89-e78960decfab\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" Oct 31 13:29:19.080346 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:19.080324 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31d9443c-5638-44ee-af89-e78960decfab-istio-data\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h\" (UID: \"31d9443c-5638-44ee-af89-e78960decfab\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" Oct 31 13:29:19.080346 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:19.080337 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d9443c-5638-44ee-af89-e78960decfab-credential-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h\" (UID: \"31d9443c-5638-44ee-af89-e78960decfab\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" Oct 31 13:29:19.080468 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:19.080378 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4lw2\" (UniqueName: \"kubernetes.io/projected/31d9443c-5638-44ee-af89-e78960decfab-kube-api-access-x4lw2\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h\" (UID: \"31d9443c-5638-44ee-af89-e78960decfab\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" Oct 31 13:29:19.080468 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:19.080409 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31d9443c-5638-44ee-af89-e78960decfab-istio-token\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h\" (UID: \"31d9443c-5638-44ee-af89-e78960decfab\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" Oct 31 13:29:19.080581 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:19.080466 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31d9443c-5638-44ee-af89-e78960decfab-workload-certs\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h\" (UID: \"31d9443c-5638-44ee-af89-e78960decfab\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" Oct 31 13:29:19.080581 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:19.080524 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31d9443c-5638-44ee-af89-e78960decfab-istio-podinfo\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h\" (UID: \"31d9443c-5638-44ee-af89-e78960decfab\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" Oct 31 13:29:19.080699 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:19.080578 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31d9443c-5638-44ee-af89-e78960decfab-istiod-ca-cert\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h\" (UID: \"31d9443c-5638-44ee-af89-e78960decfab\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" Oct 31 13:29:19.080699 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:19.080671 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d9443c-5638-44ee-af89-e78960decfab-workload-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h\" (UID: \"31d9443c-5638-44ee-af89-e78960decfab\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" Oct 31 13:29:19.080920 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:19.080895 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31d9443c-5638-44ee-af89-e78960decfab-workload-certs\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h\" (UID: \"31d9443c-5638-44ee-af89-e78960decfab\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" Oct 31 13:29:19.081383 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:19.081358 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31d9443c-5638-44ee-af89-e78960decfab-istiod-ca-cert\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h\" (UID: \"31d9443c-5638-44ee-af89-e78960decfab\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" Oct 31 13:29:19.082910 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:19.082883 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31d9443c-5638-44ee-af89-e78960decfab-istio-envoy\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h\" (UID: \"31d9443c-5638-44ee-af89-e78960decfab\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" Oct 31 13:29:19.083680 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:19.083658 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31d9443c-5638-44ee-af89-e78960decfab-istio-podinfo\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h\" (UID: \"31d9443c-5638-44ee-af89-e78960decfab\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" Oct 31 13:29:19.091823 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:19.091798 6841 scope.go:122] "RemoveContainer" containerID="695d4845ce63682aac4c250910b1d13a2c7e6a6bc2d0c6bd326517895539c1a2" Oct 31 13:29:19.092185 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:29:19.092163 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"695d4845ce63682aac4c250910b1d13a2c7e6a6bc2d0c6bd326517895539c1a2\": container with ID starting with 695d4845ce63682aac4c250910b1d13a2c7e6a6bc2d0c6bd326517895539c1a2 not found: ID does not exist" containerID="695d4845ce63682aac4c250910b1d13a2c7e6a6bc2d0c6bd326517895539c1a2" Oct 31 13:29:19.092289 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:19.092193 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"695d4845ce63682aac4c250910b1d13a2c7e6a6bc2d0c6bd326517895539c1a2"} err="failed to get container status \"695d4845ce63682aac4c250910b1d13a2c7e6a6bc2d0c6bd326517895539c1a2\": rpc error: code = NotFound desc = could not find container \"695d4845ce63682aac4c250910b1d13a2c7e6a6bc2d0c6bd326517895539c1a2\": container with ID starting with 695d4845ce63682aac4c250910b1d13a2c7e6a6bc2d0c6bd326517895539c1a2 not found: ID does not exist" Oct 31 13:29:19.092960 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:19.092940 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31d9443c-5638-44ee-af89-e78960decfab-istio-token\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h\" (UID: \"31d9443c-5638-44ee-af89-e78960decfab\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" Oct 31 13:29:19.093090 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:19.093077 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4lw2\" (UniqueName: \"kubernetes.io/projected/31d9443c-5638-44ee-af89-e78960decfab-kube-api-access-x4lw2\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h\" (UID: \"31d9443c-5638-44ee-af89-e78960decfab\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" Oct 31 13:29:19.107143 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:19.107071 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x"] Oct 31 13:29:19.122346 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:19.122319 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-52h6x"] Oct 31 13:29:19.256428 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:19.256395 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" Oct 31 13:29:19.554020 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:29:19.553975 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d9443c_5638_44ee_af89_e78960decfab.slice/crio-6db4b41ff4e7979f1c90d3afa6d1a0605821b7d6c5607fd3dd389b541aabe057 WatchSource:0}: Error finding container 6db4b41ff4e7979f1c90d3afa6d1a0605821b7d6c5607fd3dd389b541aabe057: Status 404 returned error can't find the container with id 6db4b41ff4e7979f1c90d3afa6d1a0605821b7d6c5607fd3dd389b541aabe057 Oct 31 13:29:19.554272 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:19.554250 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h"] Oct 31 13:29:19.557709 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:19.557674 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:29:19.557843 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:19.557762 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:29:19.557843 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:19.557793 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:29:19.647005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:19.646966 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bd82005-b57c-4b49-9bf5-b9a77c624c69" path="/var/lib/kubelet/pods/5bd82005-b57c-4b49-9bf5-b9a77c624c69/volumes" Oct 31 13:29:19.837726 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:19.837634 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:20.041648 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:29:20.041617 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:29:20.095581 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:20.095481 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" event={"ID":"31d9443c-5638-44ee-af89-e78960decfab","Type":"ContainerStarted","Data":"09ee9e5baf6944e6c7757bacf382ce9b36b9f45d6a43f4df7deca581cd3473ae"} Oct 31 13:29:20.095581 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:20.095552 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" event={"ID":"31d9443c-5638-44ee-af89-e78960decfab","Type":"ContainerStarted","Data":"6db4b41ff4e7979f1c90d3afa6d1a0605821b7d6c5607fd3dd389b541aabe057"} Oct 31 13:29:20.125327 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:20.125263 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" podStartSLOduration=87.125249501 podStartE2EDuration="1m27.125249501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:27:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:20.123324379 +0000 UTC m=+3406.606447281" watchObservedRunningTime="2026-10-31 13:29:20.125249501 +0000 UTC m=+3406.608372401" Oct 31 13:29:20.271025 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:20.257675 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" Oct 31 13:29:20.272432 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:20.272398 6841 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 31 13:29:20.272587 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:20.272480 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" podUID="31d9443c-5638-44ee-af89-e78960decfab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 31 13:29:20.731096 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:20.730930 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:21.268008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:21.267069 6841 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 31 13:29:21.268008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:21.267132 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" podUID="31d9443c-5638-44ee-af89-e78960decfab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 31 13:29:21.455008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:21.441739 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:21.835439 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:21.835409 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97"] Oct 31 13:29:21.835644 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:21.835618 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" podUID="cb204311-3184-4407-bea7-a759f629ec3e" containerName="istio-proxy" containerID="cri-o://51bbbf9626a1223e286fd1e7afe46de522c4eee1f293ac840987e5d0b64a75e4" gracePeriod=30 Oct 31 13:29:22.265006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:22.264400 6841 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 31 13:29:22.265006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:22.264464 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" podUID="31d9443c-5638-44ee-af89-e78960decfab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 31 13:29:22.333811 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:22.333768 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:23.245868 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:23.244716 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:23.265006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:23.263357 6841 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 31 13:29:23.265006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:23.263420 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" podUID="31d9443c-5638-44ee-af89-e78960decfab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 31 13:29:24.129006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:24.128514 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:24.261007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:24.258397 6841 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 31 13:29:24.261007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:24.258462 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" podUID="31d9443c-5638-44ee-af89-e78960decfab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 31 13:29:24.831052 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:24.829630 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:25.260094 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:25.260045 6841 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 31 13:29:25.260724 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:25.260688 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" podUID="31d9443c-5638-44ee-af89-e78960decfab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 31 13:29:25.611260 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:25.611178 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:26.262145 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:26.262111 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" Oct 31 13:29:26.262641 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:26.262620 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" Oct 31 13:29:26.263288 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:26.263268 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" Oct 31 13:29:26.417209 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:26.417150 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:27.170245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.170078 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:27.200516 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.200488 6841 generic.go:356] "Generic (PLEG): container finished" podID="cb204311-3184-4407-bea7-a759f629ec3e" containerID="51bbbf9626a1223e286fd1e7afe46de522c4eee1f293ac840987e5d0b64a75e4" exitCode=0 Oct 31 13:29:27.200653 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.200570 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" event={"ID":"cb204311-3184-4407-bea7-a759f629ec3e","Type":"ContainerDied","Data":"51bbbf9626a1223e286fd1e7afe46de522c4eee1f293ac840987e5d0b64a75e4"} Oct 31 13:29:27.200720 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.200655 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" event={"ID":"cb204311-3184-4407-bea7-a759f629ec3e","Type":"ContainerDied","Data":"4d1487d4dd623c13010d08afd4850035d93f87eee5237cf482c606964801318c"} Oct 31 13:29:27.200720 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.200675 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d1487d4dd623c13010d08afd4850035d93f87eee5237cf482c606964801318c" Oct 31 13:29:27.202924 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.202908 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" Oct 31 13:29:27.242597 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.242560 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w"] Oct 31 13:29:27.243427 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.243406 6841 cpu_manager.go:414] "containerMap: removing container" podUID="cb204311-3184-4407-bea7-a759f629ec3e" containerName="istio-proxy" Oct 31 13:29:27.243427 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.243424 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb204311-3184-4407-bea7-a759f629ec3e" containerName="istio-proxy" Oct 31 13:29:27.243627 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.243582 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb204311-3184-4407-bea7-a759f629ec3e" containerName="istio-proxy" Oct 31 13:29:27.247583 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.247561 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" Oct 31 13:29:27.267439 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.267410 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w"] Oct 31 13:29:27.306247 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.306216 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb204311-3184-4407-bea7-a759f629ec3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb204311-3184-4407-bea7-a759f629ec3e-credential-socket\") pod \"cb204311-3184-4407-bea7-a759f629ec3e\" (UID: \"cb204311-3184-4407-bea7-a759f629ec3e\") " Oct 31 13:29:27.306459 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.306268 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb204311-3184-4407-bea7-a759f629ec3e-kube-api-access-6rd8f\" (UniqueName: \"kubernetes.io/projected/cb204311-3184-4407-bea7-a759f629ec3e-kube-api-access-6rd8f\") pod \"cb204311-3184-4407-bea7-a759f629ec3e\" (UID: \"cb204311-3184-4407-bea7-a759f629ec3e\") " Oct 31 13:29:27.306459 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.306399 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb204311-3184-4407-bea7-a759f629ec3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb204311-3184-4407-bea7-a759f629ec3e-istiod-ca-cert\") pod \"cb204311-3184-4407-bea7-a759f629ec3e\" (UID: \"cb204311-3184-4407-bea7-a759f629ec3e\") " Oct 31 13:29:27.306587 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.306514 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb204311-3184-4407-bea7-a759f629ec3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb204311-3184-4407-bea7-a759f629ec3e-workload-certs\") pod \"cb204311-3184-4407-bea7-a759f629ec3e\" (UID: \"cb204311-3184-4407-bea7-a759f629ec3e\") " Oct 31 13:29:27.306646 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.306510 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb204311-3184-4407-bea7-a759f629ec3e-credential-socket" pod "cb204311-3184-4407-bea7-a759f629ec3e" (UID: "cb204311-3184-4407-bea7-a759f629ec3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:27.306775 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.306677 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb204311-3184-4407-bea7-a759f629ec3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb204311-3184-4407-bea7-a759f629ec3e-istio-envoy\") pod \"cb204311-3184-4407-bea7-a759f629ec3e\" (UID: \"cb204311-3184-4407-bea7-a759f629ec3e\") " Oct 31 13:29:27.306775 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.306708 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb204311-3184-4407-bea7-a759f629ec3e-workload-certs" pod "cb204311-3184-4407-bea7-a759f629ec3e" (UID: "cb204311-3184-4407-bea7-a759f629ec3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:27.306898 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.306797 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb204311-3184-4407-bea7-a759f629ec3e-istiod-ca-cert" pod "cb204311-3184-4407-bea7-a759f629ec3e" (UID: "cb204311-3184-4407-bea7-a759f629ec3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:29:27.307190 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.307162 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb204311-3184-4407-bea7-a759f629ec3e-istio-token\" (UniqueName: \"kubernetes.io/projected/cb204311-3184-4407-bea7-a759f629ec3e-istio-token\") pod \"cb204311-3184-4407-bea7-a759f629ec3e\" (UID: \"cb204311-3184-4407-bea7-a759f629ec3e\") " Oct 31 13:29:27.307363 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.307267 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb204311-3184-4407-bea7-a759f629ec3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb204311-3184-4407-bea7-a759f629ec3e-workload-socket\") pod \"cb204311-3184-4407-bea7-a759f629ec3e\" (UID: \"cb204311-3184-4407-bea7-a759f629ec3e\") " Oct 31 13:29:27.307363 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.307316 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb204311-3184-4407-bea7-a759f629ec3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb204311-3184-4407-bea7-a759f629ec3e-istio-data\") pod \"cb204311-3184-4407-bea7-a759f629ec3e\" (UID: \"cb204311-3184-4407-bea7-a759f629ec3e\") " Oct 31 13:29:27.307591 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.307379 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb204311-3184-4407-bea7-a759f629ec3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb204311-3184-4407-bea7-a759f629ec3e-istio-podinfo\") pod \"cb204311-3184-4407-bea7-a759f629ec3e\" (UID: \"cb204311-3184-4407-bea7-a759f629ec3e\") " Oct 31 13:29:27.307881 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.307809 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/777dacd7-0cef-42aa-b932-f25c78e7fb40-istio-envoy\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w\" (UID: \"777dacd7-0cef-42aa-b932-f25c78e7fb40\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" Oct 31 13:29:27.307969 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.307906 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/777dacd7-0cef-42aa-b932-f25c78e7fb40-workload-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w\" (UID: \"777dacd7-0cef-42aa-b932-f25c78e7fb40\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" Oct 31 13:29:27.307969 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.307961 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/777dacd7-0cef-42aa-b932-f25c78e7fb40-istiod-ca-cert\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w\" (UID: \"777dacd7-0cef-42aa-b932-f25c78e7fb40\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" Oct 31 13:29:27.308120 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.308028 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb204311-3184-4407-bea7-a759f629ec3e-istio-data" pod "cb204311-3184-4407-bea7-a759f629ec3e" (UID: "cb204311-3184-4407-bea7-a759f629ec3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:27.308258 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.308233 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb204311-3184-4407-bea7-a759f629ec3e-workload-socket" pod "cb204311-3184-4407-bea7-a759f629ec3e" (UID: "cb204311-3184-4407-bea7-a759f629ec3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:27.308359 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.308244 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/777dacd7-0cef-42aa-b932-f25c78e7fb40-istio-token\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w\" (UID: \"777dacd7-0cef-42aa-b932-f25c78e7fb40\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" Oct 31 13:29:27.308633 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.308611 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzflc\" (UniqueName: \"kubernetes.io/projected/777dacd7-0cef-42aa-b932-f25c78e7fb40-kube-api-access-fzflc\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w\" (UID: \"777dacd7-0cef-42aa-b932-f25c78e7fb40\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" Oct 31 13:29:27.308778 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.308761 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/777dacd7-0cef-42aa-b932-f25c78e7fb40-credential-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w\" (UID: \"777dacd7-0cef-42aa-b932-f25c78e7fb40\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" Oct 31 13:29:27.308932 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.308912 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/777dacd7-0cef-42aa-b932-f25c78e7fb40-istio-data\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w\" (UID: \"777dacd7-0cef-42aa-b932-f25c78e7fb40\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" Oct 31 13:29:27.310042 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.309592 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb204311-3184-4407-bea7-a759f629ec3e-kube-api-access-6rd8f" pod "cb204311-3184-4407-bea7-a759f629ec3e" (UID: "cb204311-3184-4407-bea7-a759f629ec3e"). InnerVolumeSpecName "kube-api-access-6rd8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:29:27.310133 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.309969 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb204311-3184-4407-bea7-a759f629ec3e-istio-envoy" pod "cb204311-3184-4407-bea7-a759f629ec3e" (UID: "cb204311-3184-4407-bea7-a759f629ec3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:27.310219 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.310034 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/777dacd7-0cef-42aa-b932-f25c78e7fb40-istio-podinfo\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w\" (UID: \"777dacd7-0cef-42aa-b932-f25c78e7fb40\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" Oct 31 13:29:27.310402 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.310370 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/777dacd7-0cef-42aa-b932-f25c78e7fb40-workload-certs\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w\" (UID: \"777dacd7-0cef-42aa-b932-f25c78e7fb40\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" Oct 31 13:29:27.310402 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.310368 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb204311-3184-4407-bea7-a759f629ec3e-istio-podinfo" pod "cb204311-3184-4407-bea7-a759f629ec3e" (UID: "cb204311-3184-4407-bea7-a759f629ec3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:29:27.310531 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.310479 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb204311-3184-4407-bea7-a759f629ec3e-istio-token" pod "cb204311-3184-4407-bea7-a759f629ec3e" (UID: "cb204311-3184-4407-bea7-a759f629ec3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:29:27.310665 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.310645 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb204311-3184-4407-bea7-a759f629ec3e-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:27.310704 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.310672 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb204311-3184-4407-bea7-a759f629ec3e-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:27.310704 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.310686 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb204311-3184-4407-bea7-a759f629ec3e-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:27.310773 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.310701 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb204311-3184-4407-bea7-a759f629ec3e-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:27.310773 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.310715 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb204311-3184-4407-bea7-a759f629ec3e-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:27.310773 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.310728 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rd8f\" (UniqueName: \"kubernetes.io/projected/cb204311-3184-4407-bea7-a759f629ec3e-kube-api-access-6rd8f\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:27.310773 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.310742 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb204311-3184-4407-bea7-a759f629ec3e-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:27.310773 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.310756 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb204311-3184-4407-bea7-a759f629ec3e-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:27.310773 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.310770 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb204311-3184-4407-bea7-a759f629ec3e-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:27.412550 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.412516 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/777dacd7-0cef-42aa-b932-f25c78e7fb40-istio-token\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w\" (UID: \"777dacd7-0cef-42aa-b932-f25c78e7fb40\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" Oct 31 13:29:27.412753 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.412598 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzflc\" (UniqueName: \"kubernetes.io/projected/777dacd7-0cef-42aa-b932-f25c78e7fb40-kube-api-access-fzflc\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w\" (UID: \"777dacd7-0cef-42aa-b932-f25c78e7fb40\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" Oct 31 13:29:27.412753 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.412627 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/777dacd7-0cef-42aa-b932-f25c78e7fb40-credential-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w\" (UID: \"777dacd7-0cef-42aa-b932-f25c78e7fb40\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" Oct 31 13:29:27.412753 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.412654 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/777dacd7-0cef-42aa-b932-f25c78e7fb40-istio-data\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w\" (UID: \"777dacd7-0cef-42aa-b932-f25c78e7fb40\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" Oct 31 13:29:27.412753 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.412685 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/777dacd7-0cef-42aa-b932-f25c78e7fb40-istio-podinfo\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w\" (UID: \"777dacd7-0cef-42aa-b932-f25c78e7fb40\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" Oct 31 13:29:27.412753 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.412732 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/777dacd7-0cef-42aa-b932-f25c78e7fb40-workload-certs\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w\" (UID: \"777dacd7-0cef-42aa-b932-f25c78e7fb40\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" Oct 31 13:29:27.413233 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.412882 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/777dacd7-0cef-42aa-b932-f25c78e7fb40-istio-envoy\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w\" (UID: \"777dacd7-0cef-42aa-b932-f25c78e7fb40\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" Oct 31 13:29:27.413233 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.413199 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/777dacd7-0cef-42aa-b932-f25c78e7fb40-istio-data\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w\" (UID: \"777dacd7-0cef-42aa-b932-f25c78e7fb40\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" Oct 31 13:29:27.413360 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.413259 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/777dacd7-0cef-42aa-b932-f25c78e7fb40-workload-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w\" (UID: \"777dacd7-0cef-42aa-b932-f25c78e7fb40\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" Oct 31 13:29:27.413360 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.413309 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/777dacd7-0cef-42aa-b932-f25c78e7fb40-istiod-ca-cert\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w\" (UID: \"777dacd7-0cef-42aa-b932-f25c78e7fb40\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" Oct 31 13:29:27.413360 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.413328 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/777dacd7-0cef-42aa-b932-f25c78e7fb40-workload-certs\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w\" (UID: \"777dacd7-0cef-42aa-b932-f25c78e7fb40\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" Oct 31 13:29:27.413525 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.413497 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/777dacd7-0cef-42aa-b932-f25c78e7fb40-credential-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w\" (UID: \"777dacd7-0cef-42aa-b932-f25c78e7fb40\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" Oct 31 13:29:27.413694 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.413674 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/777dacd7-0cef-42aa-b932-f25c78e7fb40-workload-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w\" (UID: \"777dacd7-0cef-42aa-b932-f25c78e7fb40\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" Oct 31 13:29:27.413954 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.413936 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/777dacd7-0cef-42aa-b932-f25c78e7fb40-istiod-ca-cert\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w\" (UID: \"777dacd7-0cef-42aa-b932-f25c78e7fb40\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" Oct 31 13:29:27.415708 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.415690 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/777dacd7-0cef-42aa-b932-f25c78e7fb40-istio-envoy\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w\" (UID: \"777dacd7-0cef-42aa-b932-f25c78e7fb40\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" Oct 31 13:29:27.415929 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.415909 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/777dacd7-0cef-42aa-b932-f25c78e7fb40-istio-podinfo\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w\" (UID: \"777dacd7-0cef-42aa-b932-f25c78e7fb40\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" Oct 31 13:29:27.424765 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.424740 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/777dacd7-0cef-42aa-b932-f25c78e7fb40-istio-token\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w\" (UID: \"777dacd7-0cef-42aa-b932-f25c78e7fb40\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" Oct 31 13:29:27.424846 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.424808 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzflc\" (UniqueName: \"kubernetes.io/projected/777dacd7-0cef-42aa-b932-f25c78e7fb40-kube-api-access-fzflc\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w\" (UID: \"777dacd7-0cef-42aa-b932-f25c78e7fb40\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" Oct 31 13:29:27.568045 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.567970 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" Oct 31 13:29:27.866486 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:29:27.866440 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod777dacd7_0cef_42aa_b932_f25c78e7fb40.slice/crio-4b17f3c5db0a87fe7345b62ef177f946cff70902604cb7ea2d7a9e9439274b81 WatchSource:0}: Error finding container 4b17f3c5db0a87fe7345b62ef177f946cff70902604cb7ea2d7a9e9439274b81: Status 404 returned error can't find the container with id 4b17f3c5db0a87fe7345b62ef177f946cff70902604cb7ea2d7a9e9439274b81 Oct 31 13:29:27.867445 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.867410 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w"] Oct 31 13:29:27.870783 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.870743 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:29:27.870893 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.870835 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:29:27.870893 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.870882 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:29:27.931995 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:27.931945 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:28.212265 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:28.212231 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" event={"ID":"777dacd7-0cef-42aa-b932-f25c78e7fb40","Type":"ContainerStarted","Data":"d818ed145097da59e8f3e72625ae399fddfaf70ef6f9b2c6a3aedef7011e61b6"} Oct 31 13:29:28.212449 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:28.212298 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" event={"ID":"777dacd7-0cef-42aa-b932-f25c78e7fb40","Type":"ContainerStarted","Data":"4b17f3c5db0a87fe7345b62ef177f946cff70902604cb7ea2d7a9e9439274b81"} Oct 31 13:29:28.212449 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:28.212345 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97" Oct 31 13:29:28.240159 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:28.240104 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" podStartSLOduration=95.240086329 podStartE2EDuration="1m35.240086329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:27:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:28.237857211 +0000 UTC m=+3414.720980117" watchObservedRunningTime="2026-10-31 13:29:28.240086329 +0000 UTC m=+3414.723209231" Oct 31 13:29:28.259273 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:28.259235 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97"] Oct 31 13:29:28.266552 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:28.266523 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8xq97"] Oct 31 13:29:28.568498 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:28.568408 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" Oct 31 13:29:28.729656 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:28.729614 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:29.465007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:29.459876 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:29.570312 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:29.570278 6841 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Oct 31 13:29:29.570748 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:29.570335 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" podUID="777dacd7-0cef-42aa-b932-f25c78e7fb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Oct 31 13:29:29.670893 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:29.670852 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb204311-3184-4407-bea7-a759f629ec3e" path="/var/lib/kubelet/pods/cb204311-3184-4407-bea7-a759f629ec3e/volumes" Oct 31 13:29:30.426943 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:30.426741 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:30.568516 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:30.568469 6841 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Oct 31 13:29:30.568723 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:30.568530 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" podUID="777dacd7-0cef-42aa-b932-f25c78e7fb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Oct 31 13:29:31.168006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:31.166622 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:31.568494 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:31.568452 6841 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Oct 31 13:29:31.568692 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:31.568520 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" podUID="777dacd7-0cef-42aa-b932-f25c78e7fb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Oct 31 13:29:31.903224 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:31.903077 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:32.108054 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:32.108017 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:32.323022 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:32.321453 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:32.569008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:32.568908 6841 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Oct 31 13:29:32.569233 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:32.569023 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" podUID="777dacd7-0cef-42aa-b932-f25c78e7fb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Oct 31 13:29:32.832721 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:32.832629 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:33.523057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:33.515971 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:33.568870 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:33.568763 6841 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Oct 31 13:29:33.568870 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:33.568820 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" podUID="777dacd7-0cef-42aa-b932-f25c78e7fb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Oct 31 13:29:34.275015 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:34.270697 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:34.568711 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:34.568627 6841 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Oct 31 13:29:34.568711 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:34.568680 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" podUID="777dacd7-0cef-42aa-b932-f25c78e7fb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Oct 31 13:29:35.125548 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:35.122608 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:35.574252 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:35.574222 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" Oct 31 13:29:35.574735 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:35.574724 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" Oct 31 13:29:35.575392 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:35.575370 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" Oct 31 13:29:35.822627 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:35.822586 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:36.726513 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:36.726469 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:37.422583 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:37.422547 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:38.212122 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:38.212092 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:39.086079 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:39.086036 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:39.594333 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:39.594300 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:39.906101 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:39.905959 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:40.544540 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:40.544506 6841 scope.go:122] "RemoveContainer" containerID="51bbbf9626a1223e286fd1e7afe46de522c4eee1f293ac840987e5d0b64a75e4" Oct 31 13:29:40.805494 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:40.805418 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:41.818005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:41.817671 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:42.817611 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:42.817574 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:43.863381 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:43.863348 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:44.611499 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:44.611465 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:45.379663 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:45.379622 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:46.379698 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:46.379615 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:46.786357 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:46.786328 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:47.380587 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:47.380552 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:48.160960 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:48.160859 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:48.381571 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:48.381535 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:48.957710 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:48.957676 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:49.704197 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:49.704165 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd"] Oct 31 13:29:49.716487 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:49.715357 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd" Oct 31 13:29:49.720707 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:49.720677 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd"] Oct 31 13:29:49.724578 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:49.724547 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a39960\"/\"kube-root-ca.crt\"" Oct 31 13:29:49.725310 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:49.725290 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a39960\"/\"default-dockercfg-7gssq\"" Oct 31 13:29:49.729301 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:49.729277 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a39960\"/\"test-model-a39960-kserve-self-signed-certs\"" Oct 31 13:29:49.741509 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:49.741479 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:49.764394 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:49.764364 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-tmp-dir\") pod \"test-model-a39960-kserve-8674d5c7c5-r22xd\" (UID: \"3ff4f1a8-99b6-4d81-8772-7ea00af5bbad\") " pod="ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd" Oct 31 13:29:49.764634 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:49.764609 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-model-cache\") pod \"test-model-a39960-kserve-8674d5c7c5-r22xd\" (UID: \"3ff4f1a8-99b6-4d81-8772-7ea00af5bbad\") " pod="ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd" Oct 31 13:29:49.764752 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:49.764657 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-dshm\") pod \"test-model-a39960-kserve-8674d5c7c5-r22xd\" (UID: \"3ff4f1a8-99b6-4d81-8772-7ea00af5bbad\") " pod="ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd" Oct 31 13:29:49.764892 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:49.764837 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-home\") pod \"test-model-a39960-kserve-8674d5c7c5-r22xd\" (UID: \"3ff4f1a8-99b6-4d81-8772-7ea00af5bbad\") " pod="ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd" Oct 31 13:29:49.765345 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:49.765150 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qqlz\" (UniqueName: \"kubernetes.io/projected/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-kube-api-access-4qqlz\") pod \"test-model-a39960-kserve-8674d5c7c5-r22xd\" (UID: \"3ff4f1a8-99b6-4d81-8772-7ea00af5bbad\") " pod="ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd" Oct 31 13:29:49.765345 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:49.765277 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-tls-certs\") pod \"test-model-a39960-kserve-8674d5c7c5-r22xd\" (UID: \"3ff4f1a8-99b6-4d81-8772-7ea00af5bbad\") " pod="ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd" Oct 31 13:29:49.820849 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:49.820823 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a39960\"/\"openshift-service-ca.crt\"" Oct 31 13:29:49.867383 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:49.867352 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qqlz\" (UniqueName: \"kubernetes.io/projected/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-kube-api-access-4qqlz\") pod \"test-model-a39960-kserve-8674d5c7c5-r22xd\" (UID: \"3ff4f1a8-99b6-4d81-8772-7ea00af5bbad\") " pod="ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd" Oct 31 13:29:49.867531 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:49.867446 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-tls-certs\") pod \"test-model-a39960-kserve-8674d5c7c5-r22xd\" (UID: \"3ff4f1a8-99b6-4d81-8772-7ea00af5bbad\") " pod="ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd" Oct 31 13:29:49.867531 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:49.867489 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-tmp-dir\") pod \"test-model-a39960-kserve-8674d5c7c5-r22xd\" (UID: \"3ff4f1a8-99b6-4d81-8772-7ea00af5bbad\") " pod="ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd" Oct 31 13:29:49.867664 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:49.867610 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-model-cache\") pod \"test-model-a39960-kserve-8674d5c7c5-r22xd\" (UID: \"3ff4f1a8-99b6-4d81-8772-7ea00af5bbad\") " pod="ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd" Oct 31 13:29:49.867664 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:49.867643 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-dshm\") pod \"test-model-a39960-kserve-8674d5c7c5-r22xd\" (UID: \"3ff4f1a8-99b6-4d81-8772-7ea00af5bbad\") " pod="ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd" Oct 31 13:29:49.867887 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:49.867704 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-home\") pod \"test-model-a39960-kserve-8674d5c7c5-r22xd\" (UID: \"3ff4f1a8-99b6-4d81-8772-7ea00af5bbad\") " pod="ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd" Oct 31 13:29:49.868087 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:49.868066 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-model-cache\") pod \"test-model-a39960-kserve-8674d5c7c5-r22xd\" (UID: \"3ff4f1a8-99b6-4d81-8772-7ea00af5bbad\") " pod="ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd" Oct 31 13:29:49.868282 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:49.868258 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-home\") pod \"test-model-a39960-kserve-8674d5c7c5-r22xd\" (UID: \"3ff4f1a8-99b6-4d81-8772-7ea00af5bbad\") " pod="ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd" Oct 31 13:29:49.868400 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:49.868328 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-tmp-dir\") pod \"test-model-a39960-kserve-8674d5c7c5-r22xd\" (UID: \"3ff4f1a8-99b6-4d81-8772-7ea00af5bbad\") " pod="ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd" Oct 31 13:29:49.870296 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:49.870273 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-dshm\") pod \"test-model-a39960-kserve-8674d5c7c5-r22xd\" (UID: \"3ff4f1a8-99b6-4d81-8772-7ea00af5bbad\") " pod="ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd" Oct 31 13:29:49.871283 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:49.871259 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-tls-certs\") pod \"test-model-a39960-kserve-8674d5c7c5-r22xd\" (UID: \"3ff4f1a8-99b6-4d81-8772-7ea00af5bbad\") " pod="ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd" Oct 31 13:29:49.890876 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:49.890853 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qqlz\" (UniqueName: \"kubernetes.io/projected/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-kube-api-access-4qqlz\") pod \"test-model-a39960-kserve-8674d5c7c5-r22xd\" (UID: \"3ff4f1a8-99b6-4d81-8772-7ea00af5bbad\") " pod="ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd" Oct 31 13:29:49.985512 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:49.985421 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:50.039299 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:50.039026 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd" Oct 31 13:29:50.164124 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:29:50.164091 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:29:50.435261 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:50.435221 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd"] Oct 31 13:29:50.450845 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:50.450816 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd" event={"ID":"3ff4f1a8-99b6-4d81-8772-7ea00af5bbad","Type":"ContainerStarted","Data":"19732f129087bf36f864a08aab9a885ad46edf8a7b94d08ab10f1591a3a0a083"} Oct 31 13:29:50.729224 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:50.729154 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:51.461756 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:51.461715 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd" event={"ID":"3ff4f1a8-99b6-4d81-8772-7ea00af5bbad","Type":"ContainerStarted","Data":"03fab0677cb7639c413a34c1bc771d08ff9f19e02dad31a9f4699d8820d84368"} Oct 31 13:29:51.487430 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:51.487381 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd" podStartSLOduration=2.487365111 podStartE2EDuration="2.487365111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:51.483417045 +0000 UTC m=+3437.966539948" watchObservedRunningTime="2026-10-31 13:29:51.487365111 +0000 UTC m=+3437.970488011" Oct 31 13:29:51.529318 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:51.529286 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:52.055259 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:52.055229 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd" Oct 31 13:29:52.316353 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:52.316245 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:53.226447 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:53.226408 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:54.011520 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:54.011487 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:54.817608 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:54.817574 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:55.746163 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:55.746128 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:56.484918 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:56.484875 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:57.266859 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:57.266821 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:58.108118 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:58.108079 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:58.800628 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:58.800584 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:29:59.864467 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:29:59.864432 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:00.054476 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:00.054439 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd" Oct 31 13:30:00.067706 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:00.067677 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd" Oct 31 13:30:00.558133 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:00.558104 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd" Oct 31 13:30:01.240598 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:01.240557 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:02.193474 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:02.192925 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:03.015671 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:03.015637 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:03.259036 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:03.253774 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:04.018598 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:04.018552 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:04.969469 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:04.969431 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:05.814767 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:05.814730 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:06.749536 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:06.749508 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:07.517639 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:07.517602 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:08.425605 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:08.425560 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:09.233037 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:09.232968 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:09.444005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:09.442976 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:09.753899 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:09.753862 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:09.986516 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:09.986019 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:10.222445 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:10.222416 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:11.308777 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:11.308738 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:12.310895 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:12.310860 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:13.153584 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:13.153547 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:13.351099 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:13.351066 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:13.981662 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:13.981613 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:15.063684 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:15.063613 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:16.064038 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:16.063933 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:17.174457 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:17.174420 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:17.555103 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:17.555070 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:17.813938 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:17.813855 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:18.182232 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:18.182142 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:18.456004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:18.445615 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:18.890718 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:18.890631 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:19.839225 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:19.839188 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:20.037494 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:30:20.037457 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:30:20.885539 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:20.885491 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:21.917123 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:21.917081 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:22.983606 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:22.983574 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:23.909184 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:23.909146 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:24.747581 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:24.747531 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:25.822676 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:25.822640 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:26.614319 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:26.614282 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:27.466338 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:27.466267 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:28.447942 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:28.447904 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:29.216273 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:29.216228 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:30.158405 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:30.158363 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:31.190845 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:31.190803 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:32.355793 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:32.355757 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:33.446777 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:33.446740 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:34.651504 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:34.651470 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:35.708538 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:35.708506 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:36.496040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:36.496007 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:37.427729 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:37.427692 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:37.631690 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:37.631660 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:38.368380 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:38.368348 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:38.976967 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:38.976906 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:39.261506 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:39.261403 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:39.554951 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:39.554837 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:39.841370 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:39.841271 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:40.204309 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:40.204279 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:40.574065 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:40.573966 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:40.926761 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:40.926670 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:41.753765 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:41.753730 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:42.377597 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:42.377559 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:42.645055 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:42.637785 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:43.766298 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:43.759618 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:43.998618 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:43.998571 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:44.865036 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:44.864997 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:45.359193 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:45.359161 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:45.709882 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:45.709830 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:46.281634 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:46.281597 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:47.270036 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:47.270002 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:48.438710 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:48.438650 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:49.236123 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:49.236086 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:50.070263 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:30:50.070186 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:30:50.116439 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:50.116400 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:50.994653 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:50.994616 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:51.937767 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:51.937725 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:52.683505 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:52.683467 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:53.472114 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:53.472073 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:54.204929 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:54.204881 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:55.074911 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:55.074821 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:55.847770 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:55.847729 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:56.658486 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:56.658451 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:57.487735 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:57.487701 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:58.332152 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:58.332118 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:59.087932 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:59.087894 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:30:59.858337 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:30:59.858303 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:00.818910 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:00.818870 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:01.627691 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:01.627644 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:02.532486 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:02.532449 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:03.271659 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:03.271596 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:04.166143 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:04.166103 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:04.879892 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:04.879855 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:05.717914 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:05.717857 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:06.645876 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:06.645838 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:07.462676 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:07.462638 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:08.372860 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:08.372825 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:09.379656 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:09.379625 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:10.158668 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:10.158637 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:11.051859 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:11.051817 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:11.266842 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:11.266807 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:12.148523 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:12.148466 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:12.940131 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:12.940097 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:13.904732 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:13.904685 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:14.139157 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:14.139116 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:14.850869 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:14.850828 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:15.407374 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:15.407338 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w"] Oct 31 13:31:15.407784 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:15.407548 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" podUID="9936f700-0761-4889-9f23-b262428f2d2f" containerName="istio-proxy" containerID="cri-o://42df17d8d39f6e87de7c8c63aac9d01da24f36654742845f597d5d08491985f1" gracePeriod=30 Oct 31 13:31:15.768745 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:15.768652 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:16.727393 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:16.727355 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:17.508240 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:17.508190 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:18.448209 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:18.448171 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:19.354655 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:19.354621 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:20.036831 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:31:20.036797 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:31:20.065828 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.065794 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr"] Oct 31 13:31:20.066061 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.066039 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" podUID="24ea07e9-65f1-4208-b5db-1d283caf05e1" containerName="istio-proxy" containerID="cri-o://e2d1183359ee3db9a2e86c3413bffd2ee936d17d29108ce4a7ebe33d93977f37" gracePeriod=30 Oct 31 13:31:20.172668 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.172587 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:20.558773 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.558743 6841 generic.go:356] "Generic (PLEG): container finished" podID="9936f700-0761-4889-9f23-b262428f2d2f" containerID="42df17d8d39f6e87de7c8c63aac9d01da24f36654742845f597d5d08491985f1" exitCode=0 Oct 31 13:31:20.558963 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.558817 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" event={"ID":"9936f700-0761-4889-9f23-b262428f2d2f","Type":"ContainerDied","Data":"42df17d8d39f6e87de7c8c63aac9d01da24f36654742845f597d5d08491985f1"} Oct 31 13:31:20.769090 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.769066 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" Oct 31 13:31:20.817296 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.817257 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r"] Oct 31 13:31:20.819641 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.819612 6841 cpu_manager.go:414] "containerMap: removing container" podUID="9936f700-0761-4889-9f23-b262428f2d2f" containerName="istio-proxy" Oct 31 13:31:20.819818 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.819802 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9936f700-0761-4889-9f23-b262428f2d2f" containerName="istio-proxy" Oct 31 13:31:20.820307 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.820285 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="9936f700-0761-4889-9f23-b262428f2d2f" containerName="istio-proxy" Oct 31 13:31:20.827212 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.827188 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r" Oct 31 13:31:20.832342 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.832316 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r"] Oct 31 13:31:20.835289 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.835263 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-3cddb3-serving-cert\"" Oct 31 13:31:20.886096 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.886063 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9936f700-0761-4889-9f23-b262428f2d2f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9936f700-0761-4889-9f23-b262428f2d2f-credential-socket\") pod \"9936f700-0761-4889-9f23-b262428f2d2f\" (UID: \"9936f700-0761-4889-9f23-b262428f2d2f\") " Oct 31 13:31:20.886269 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.886208 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9936f700-0761-4889-9f23-b262428f2d2f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9936f700-0761-4889-9f23-b262428f2d2f-istio-data\") pod \"9936f700-0761-4889-9f23-b262428f2d2f\" (UID: \"9936f700-0761-4889-9f23-b262428f2d2f\") " Oct 31 13:31:20.886385 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.886368 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9936f700-0761-4889-9f23-b262428f2d2f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9936f700-0761-4889-9f23-b262428f2d2f-istio-podinfo\") pod \"9936f700-0761-4889-9f23-b262428f2d2f\" (UID: \"9936f700-0761-4889-9f23-b262428f2d2f\") " Oct 31 13:31:20.886475 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.886415 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9936f700-0761-4889-9f23-b262428f2d2f-istio-token\" (UniqueName: \"kubernetes.io/projected/9936f700-0761-4889-9f23-b262428f2d2f-istio-token\") pod \"9936f700-0761-4889-9f23-b262428f2d2f\" (UID: \"9936f700-0761-4889-9f23-b262428f2d2f\") " Oct 31 13:31:20.886544 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.886482 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9936f700-0761-4889-9f23-b262428f2d2f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9936f700-0761-4889-9f23-b262428f2d2f-workload-certs\") pod \"9936f700-0761-4889-9f23-b262428f2d2f\" (UID: \"9936f700-0761-4889-9f23-b262428f2d2f\") " Oct 31 13:31:20.886544 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.886480 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9936f700-0761-4889-9f23-b262428f2d2f-credential-socket" pod "9936f700-0761-4889-9f23-b262428f2d2f" (UID: "9936f700-0761-4889-9f23-b262428f2d2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:20.886667 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.886554 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9936f700-0761-4889-9f23-b262428f2d2f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9936f700-0761-4889-9f23-b262428f2d2f-workload-socket\") pod \"9936f700-0761-4889-9f23-b262428f2d2f\" (UID: \"9936f700-0761-4889-9f23-b262428f2d2f\") " Oct 31 13:31:20.886667 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.886580 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9936f700-0761-4889-9f23-b262428f2d2f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9936f700-0761-4889-9f23-b262428f2d2f-istio-envoy\") pod \"9936f700-0761-4889-9f23-b262428f2d2f\" (UID: \"9936f700-0761-4889-9f23-b262428f2d2f\") " Oct 31 13:31:20.886786 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.886693 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9936f700-0761-4889-9f23-b262428f2d2f-kube-api-access-4lgdx\" (UniqueName: \"kubernetes.io/projected/9936f700-0761-4889-9f23-b262428f2d2f-kube-api-access-4lgdx\") pod \"9936f700-0761-4889-9f23-b262428f2d2f\" (UID: \"9936f700-0761-4889-9f23-b262428f2d2f\") " Oct 31 13:31:20.886786 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.886770 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9936f700-0761-4889-9f23-b262428f2d2f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9936f700-0761-4889-9f23-b262428f2d2f-istiod-ca-cert\") pod \"9936f700-0761-4889-9f23-b262428f2d2f\" (UID: \"9936f700-0761-4889-9f23-b262428f2d2f\") " Oct 31 13:31:20.887149 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.887122 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/283a00d5-0c1d-4e1e-9996-58f08babdb21-maas-api-tls\") pod \"maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r\" (UID: \"283a00d5-0c1d-4e1e-9996-58f08babdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r" Oct 31 13:31:20.887149 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.887140 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9936f700-0761-4889-9f23-b262428f2d2f-workload-socket" pod "9936f700-0761-4889-9f23-b262428f2d2f" (UID: "9936f700-0761-4889-9f23-b262428f2d2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:20.887343 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.887258 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9936f700-0761-4889-9f23-b262428f2d2f-istio-data" pod "9936f700-0761-4889-9f23-b262428f2d2f" (UID: "9936f700-0761-4889-9f23-b262428f2d2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:20.887419 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.887345 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2kz7\" (UniqueName: \"kubernetes.io/projected/283a00d5-0c1d-4e1e-9996-58f08babdb21-kube-api-access-q2kz7\") pod \"maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r\" (UID: \"283a00d5-0c1d-4e1e-9996-58f08babdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r" Oct 31 13:31:20.887822 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.887478 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/283a00d5-0c1d-4e1e-9996-58f08babdb21-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r\" (UID: \"283a00d5-0c1d-4e1e-9996-58f08babdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r" Oct 31 13:31:20.887822 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.887672 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/283a00d5-0c1d-4e1e-9996-58f08babdb21-metrics-certs\") pod \"maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r\" (UID: \"283a00d5-0c1d-4e1e-9996-58f08babdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r" Oct 31 13:31:20.888080 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.888054 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9936f700-0761-4889-9f23-b262428f2d2f-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:20.888171 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.888085 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9936f700-0761-4889-9f23-b262428f2d2f-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:20.888171 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.888101 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9936f700-0761-4889-9f23-b262428f2d2f-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:20.888599 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.888576 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9936f700-0761-4889-9f23-b262428f2d2f-istiod-ca-cert" pod "9936f700-0761-4889-9f23-b262428f2d2f" (UID: "9936f700-0761-4889-9f23-b262428f2d2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:31:20.889040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.889014 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9936f700-0761-4889-9f23-b262428f2d2f-workload-certs" pod "9936f700-0761-4889-9f23-b262428f2d2f" (UID: "9936f700-0761-4889-9f23-b262428f2d2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:20.891029 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.890968 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9936f700-0761-4889-9f23-b262428f2d2f-istio-token" pod "9936f700-0761-4889-9f23-b262428f2d2f" (UID: "9936f700-0761-4889-9f23-b262428f2d2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:20.891403 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.891383 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9936f700-0761-4889-9f23-b262428f2d2f-istio-envoy" pod "9936f700-0761-4889-9f23-b262428f2d2f" (UID: "9936f700-0761-4889-9f23-b262428f2d2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:20.891506 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.891400 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9936f700-0761-4889-9f23-b262428f2d2f-istio-podinfo" pod "9936f700-0761-4889-9f23-b262428f2d2f" (UID: "9936f700-0761-4889-9f23-b262428f2d2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:31:20.891784 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.891767 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9936f700-0761-4889-9f23-b262428f2d2f-kube-api-access-4lgdx" pod "9936f700-0761-4889-9f23-b262428f2d2f" (UID: "9936f700-0761-4889-9f23-b262428f2d2f"). InnerVolumeSpecName "kube-api-access-4lgdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:20.990116 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.990025 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/283a00d5-0c1d-4e1e-9996-58f08babdb21-maas-api-tls\") pod \"maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r\" (UID: \"283a00d5-0c1d-4e1e-9996-58f08babdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r" Oct 31 13:31:20.990296 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.990189 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2kz7\" (UniqueName: \"kubernetes.io/projected/283a00d5-0c1d-4e1e-9996-58f08babdb21-kube-api-access-q2kz7\") pod \"maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r\" (UID: \"283a00d5-0c1d-4e1e-9996-58f08babdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r" Oct 31 13:31:20.990296 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.990264 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/283a00d5-0c1d-4e1e-9996-58f08babdb21-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r\" (UID: \"283a00d5-0c1d-4e1e-9996-58f08babdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r" Oct 31 13:31:20.990428 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.990380 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/283a00d5-0c1d-4e1e-9996-58f08babdb21-metrics-certs\") pod \"maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r\" (UID: \"283a00d5-0c1d-4e1e-9996-58f08babdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r" Oct 31 13:31:20.990585 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.990560 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9936f700-0761-4889-9f23-b262428f2d2f-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:20.990670 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.990589 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9936f700-0761-4889-9f23-b262428f2d2f-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:20.990670 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.990605 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lgdx\" (UniqueName: \"kubernetes.io/projected/9936f700-0761-4889-9f23-b262428f2d2f-kube-api-access-4lgdx\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:20.990670 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.990619 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9936f700-0761-4889-9f23-b262428f2d2f-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:20.990670 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.990635 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9936f700-0761-4889-9f23-b262428f2d2f-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:20.990670 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.990648 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9936f700-0761-4889-9f23-b262428f2d2f-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:20.991264 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.991239 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/283a00d5-0c1d-4e1e-9996-58f08babdb21-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r\" (UID: \"283a00d5-0c1d-4e1e-9996-58f08babdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r" Oct 31 13:31:20.994075 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.993576 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/283a00d5-0c1d-4e1e-9996-58f08babdb21-maas-api-tls\") pod \"maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r\" (UID: \"283a00d5-0c1d-4e1e-9996-58f08babdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r" Oct 31 13:31:20.994075 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:20.994032 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/283a00d5-0c1d-4e1e-9996-58f08babdb21-metrics-certs\") pod \"maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r\" (UID: \"283a00d5-0c1d-4e1e-9996-58f08babdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r" Oct 31 13:31:21.009780 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:21.009741 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2kz7\" (UniqueName: \"kubernetes.io/projected/283a00d5-0c1d-4e1e-9996-58f08babdb21-kube-api-access-q2kz7\") pod \"maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r\" (UID: \"283a00d5-0c1d-4e1e-9996-58f08babdb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r" Oct 31 13:31:21.157132 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:21.157095 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r" Oct 31 13:31:21.501551 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:21.497239 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r"] Oct 31 13:31:21.503541 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:21.501952 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:21.513960 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:21.513903 6841 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 13:31:21.557040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:21.556977 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j"] Oct 31 13:31:21.574213 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:21.574182 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" event={"ID":"9936f700-0761-4889-9f23-b262428f2d2f","Type":"ContainerDied","Data":"b806eeb87a8cd7f54d9f3702c0101374d9aa1e9c4d86be9a289d15bf21dedec3"} Oct 31 13:31:21.574351 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:21.574198 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w" Oct 31 13:31:21.574351 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:21.574257 6841 scope.go:122] "RemoveContainer" containerID="42df17d8d39f6e87de7c8c63aac9d01da24f36654742845f597d5d08491985f1" Oct 31 13:31:21.575883 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:21.575855 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r" event={"ID":"283a00d5-0c1d-4e1e-9996-58f08babdb21","Type":"ContainerStarted","Data":"1f4da1bbd14f5a6fc519520fc5f5a13460e81f40f2656134c6f50669ef83e091"} Oct 31 13:31:21.837092 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:21.834933 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:21.856613 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:21.856585 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w"] Oct 31 13:31:21.865026 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:21.864992 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-5sz2w"] Oct 31 13:31:22.587937 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:22.587828 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r" event={"ID":"283a00d5-0c1d-4e1e-9996-58f08babdb21","Type":"ContainerStarted","Data":"273973c94298446c550bb39231b2e1567cc5035bc56ef1d37af21bb9d8b16015"} Oct 31 13:31:22.587937 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:22.587926 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r" Oct 31 13:31:22.623755 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:22.623702 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r" podStartSLOduration=9.066393184 podStartE2EDuration="9.623689355s" totalImagesPullingTime="557.296171ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:31:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:31:22.619111013 +0000 UTC m=+3529.102233914" watchObservedRunningTime="2026-10-31 13:31:22.623689355 +0000 UTC m=+3529.106812298" Oct 31 13:31:22.822873 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:22.822835 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:23.647973 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:23.647932 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9936f700-0761-4889-9f23-b262428f2d2f" path="/var/lib/kubelet/pods/9936f700-0761-4889-9f23-b262428f2d2f/volumes" Oct 31 13:31:23.823515 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:23.823474 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:24.685358 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:24.685170 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:25.310470 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.310445 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" Oct 31 13:31:25.401091 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.401003 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:25.413813 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.413784 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24ea07e9-65f1-4208-b5db-1d283caf05e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24ea07e9-65f1-4208-b5db-1d283caf05e1-istio-data\") pod \"24ea07e9-65f1-4208-b5db-1d283caf05e1\" (UID: \"24ea07e9-65f1-4208-b5db-1d283caf05e1\") " Oct 31 13:31:25.414029 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.413858 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24ea07e9-65f1-4208-b5db-1d283caf05e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24ea07e9-65f1-4208-b5db-1d283caf05e1-credential-socket\") pod \"24ea07e9-65f1-4208-b5db-1d283caf05e1\" (UID: \"24ea07e9-65f1-4208-b5db-1d283caf05e1\") " Oct 31 13:31:25.414029 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.413961 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24ea07e9-65f1-4208-b5db-1d283caf05e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24ea07e9-65f1-4208-b5db-1d283caf05e1-istio-podinfo\") pod \"24ea07e9-65f1-4208-b5db-1d283caf05e1\" (UID: \"24ea07e9-65f1-4208-b5db-1d283caf05e1\") " Oct 31 13:31:25.414029 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.414005 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24ea07e9-65f1-4208-b5db-1d283caf05e1-istio-token\" (UniqueName: \"kubernetes.io/projected/24ea07e9-65f1-4208-b5db-1d283caf05e1-istio-token\") pod \"24ea07e9-65f1-4208-b5db-1d283caf05e1\" (UID: \"24ea07e9-65f1-4208-b5db-1d283caf05e1\") " Oct 31 13:31:25.414233 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.414041 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24ea07e9-65f1-4208-b5db-1d283caf05e1-kube-api-access-95vh4\" (UniqueName: \"kubernetes.io/projected/24ea07e9-65f1-4208-b5db-1d283caf05e1-kube-api-access-95vh4\") pod \"24ea07e9-65f1-4208-b5db-1d283caf05e1\" (UID: \"24ea07e9-65f1-4208-b5db-1d283caf05e1\") " Oct 31 13:31:25.414233 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.414140 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24ea07e9-65f1-4208-b5db-1d283caf05e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24ea07e9-65f1-4208-b5db-1d283caf05e1-workload-socket\") pod \"24ea07e9-65f1-4208-b5db-1d283caf05e1\" (UID: \"24ea07e9-65f1-4208-b5db-1d283caf05e1\") " Oct 31 13:31:25.414233 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.414211 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24ea07e9-65f1-4208-b5db-1d283caf05e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24ea07e9-65f1-4208-b5db-1d283caf05e1-workload-certs\") pod \"24ea07e9-65f1-4208-b5db-1d283caf05e1\" (UID: \"24ea07e9-65f1-4208-b5db-1d283caf05e1\") " Oct 31 13:31:25.414394 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.414247 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24ea07e9-65f1-4208-b5db-1d283caf05e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24ea07e9-65f1-4208-b5db-1d283caf05e1-istiod-ca-cert\") pod \"24ea07e9-65f1-4208-b5db-1d283caf05e1\" (UID: \"24ea07e9-65f1-4208-b5db-1d283caf05e1\") " Oct 31 13:31:25.414394 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.414248 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ea07e9-65f1-4208-b5db-1d283caf05e1-istio-data" pod "24ea07e9-65f1-4208-b5db-1d283caf05e1" (UID: "24ea07e9-65f1-4208-b5db-1d283caf05e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:25.415260 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.414455 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24ea07e9-65f1-4208-b5db-1d283caf05e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24ea07e9-65f1-4208-b5db-1d283caf05e1-istio-envoy\") pod \"24ea07e9-65f1-4208-b5db-1d283caf05e1\" (UID: \"24ea07e9-65f1-4208-b5db-1d283caf05e1\") " Oct 31 13:31:25.415260 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.415093 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ea07e9-65f1-4208-b5db-1d283caf05e1-credential-socket" pod "24ea07e9-65f1-4208-b5db-1d283caf05e1" (UID: "24ea07e9-65f1-4208-b5db-1d283caf05e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:25.415260 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.415167 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ea07e9-65f1-4208-b5db-1d283caf05e1-workload-certs" pod "24ea07e9-65f1-4208-b5db-1d283caf05e1" (UID: "24ea07e9-65f1-4208-b5db-1d283caf05e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:25.415525 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.415419 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ea07e9-65f1-4208-b5db-1d283caf05e1-workload-socket" pod "24ea07e9-65f1-4208-b5db-1d283caf05e1" (UID: "24ea07e9-65f1-4208-b5db-1d283caf05e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:25.415525 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.415497 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24ea07e9-65f1-4208-b5db-1d283caf05e1-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:25.415525 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.415519 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24ea07e9-65f1-4208-b5db-1d283caf05e1-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:25.415698 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.415536 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24ea07e9-65f1-4208-b5db-1d283caf05e1-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:25.415755 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.415718 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24ea07e9-65f1-4208-b5db-1d283caf05e1-istiod-ca-cert" pod "24ea07e9-65f1-4208-b5db-1d283caf05e1" (UID: "24ea07e9-65f1-4208-b5db-1d283caf05e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:31:25.417795 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.417769 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ea07e9-65f1-4208-b5db-1d283caf05e1-istio-token" pod "24ea07e9-65f1-4208-b5db-1d283caf05e1" (UID: "24ea07e9-65f1-4208-b5db-1d283caf05e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:25.418410 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.418338 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ea07e9-65f1-4208-b5db-1d283caf05e1-istio-envoy" pod "24ea07e9-65f1-4208-b5db-1d283caf05e1" (UID: "24ea07e9-65f1-4208-b5db-1d283caf05e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:25.418572 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.418553 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24ea07e9-65f1-4208-b5db-1d283caf05e1-istio-podinfo" pod "24ea07e9-65f1-4208-b5db-1d283caf05e1" (UID: "24ea07e9-65f1-4208-b5db-1d283caf05e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:31:25.420442 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.420421 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ea07e9-65f1-4208-b5db-1d283caf05e1-kube-api-access-95vh4" pod "24ea07e9-65f1-4208-b5db-1d283caf05e1" (UID: "24ea07e9-65f1-4208-b5db-1d283caf05e1"). InnerVolumeSpecName "kube-api-access-95vh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:25.517559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.517523 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24ea07e9-65f1-4208-b5db-1d283caf05e1-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:25.517559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.517557 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24ea07e9-65f1-4208-b5db-1d283caf05e1-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:25.517791 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.517568 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95vh4\" (UniqueName: \"kubernetes.io/projected/24ea07e9-65f1-4208-b5db-1d283caf05e1-kube-api-access-95vh4\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:25.517791 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.517578 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24ea07e9-65f1-4208-b5db-1d283caf05e1-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:25.517791 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.517587 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24ea07e9-65f1-4208-b5db-1d283caf05e1-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:25.517791 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.517596 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24ea07e9-65f1-4208-b5db-1d283caf05e1-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:25.647683 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.647643 6841 generic.go:356] "Generic (PLEG): container finished" podID="24ea07e9-65f1-4208-b5db-1d283caf05e1" containerID="e2d1183359ee3db9a2e86c3413bffd2ee936d17d29108ce4a7ebe33d93977f37" exitCode=0 Oct 31 13:31:25.647683 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.647682 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" Oct 31 13:31:25.647918 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.647728 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" event={"ID":"24ea07e9-65f1-4208-b5db-1d283caf05e1","Type":"ContainerDied","Data":"e2d1183359ee3db9a2e86c3413bffd2ee936d17d29108ce4a7ebe33d93977f37"} Oct 31 13:31:25.647918 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.647792 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr" event={"ID":"24ea07e9-65f1-4208-b5db-1d283caf05e1","Type":"ContainerDied","Data":"2658dd7c48f816b5e2b636c3afeddc70150fd1406f2b51f6a21397364af2fede"} Oct 31 13:31:25.647918 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.647824 6841 scope.go:122] "RemoveContainer" containerID="e2d1183359ee3db9a2e86c3413bffd2ee936d17d29108ce4a7ebe33d93977f37" Oct 31 13:31:25.662523 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.662505 6841 scope.go:122] "RemoveContainer" containerID="e2d1183359ee3db9a2e86c3413bffd2ee936d17d29108ce4a7ebe33d93977f37" Oct 31 13:31:25.662759 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:31:25.662741 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2d1183359ee3db9a2e86c3413bffd2ee936d17d29108ce4a7ebe33d93977f37\": container with ID starting with e2d1183359ee3db9a2e86c3413bffd2ee936d17d29108ce4a7ebe33d93977f37 not found: ID does not exist" containerID="e2d1183359ee3db9a2e86c3413bffd2ee936d17d29108ce4a7ebe33d93977f37" Oct 31 13:31:25.662808 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.662768 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2d1183359ee3db9a2e86c3413bffd2ee936d17d29108ce4a7ebe33d93977f37"} err="failed to get container status \"e2d1183359ee3db9a2e86c3413bffd2ee936d17d29108ce4a7ebe33d93977f37\": rpc error: code = NotFound desc = could not find container \"e2d1183359ee3db9a2e86c3413bffd2ee936d17d29108ce4a7ebe33d93977f37\": container with ID starting with e2d1183359ee3db9a2e86c3413bffd2ee936d17d29108ce4a7ebe33d93977f37 not found: ID does not exist" Oct 31 13:31:25.695859 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.694939 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr"] Oct 31 13:31:25.697031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:25.696996 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-mh6nr"] Oct 31 13:31:26.272460 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:26.272342 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:27.156298 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:27.156263 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:27.646998 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:27.646958 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24ea07e9-65f1-4208-b5db-1d283caf05e1" path="/var/lib/kubelet/pods/24ea07e9-65f1-4208-b5db-1d283caf05e1/volumes" Oct 31 13:31:28.117081 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:28.117032 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:28.601262 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:28.601231 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r" Oct 31 13:31:28.875128 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:28.873351 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:29.745507 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:29.745472 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:30.591651 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:30.591618 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:30.863824 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:30.863740 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5"] Oct 31 13:31:31.008063 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:30.978768 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:31.621768 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:31.621319 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:32.431837 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:32.431736 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:33.313554 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:33.313516 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:34.211083 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:34.211050 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:35.155502 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:35.155468 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:35.938584 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:35.938551 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:36.608743 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:36.608672 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j" podUID="86ce378f-511a-4301-a0c7-85ec5027f0c5" containerName="main" containerID="cri-o://fefc27ffe1f8009b18ef35dbb73dbff171c885744bbf0a763e1806f6d7bcbc7d" gracePeriod=45 Oct 31 13:31:36.737082 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:36.737051 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:36.788218 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:36.788180 6841 generic.go:356] "Generic (PLEG): container finished" podID="86ce378f-511a-4301-a0c7-85ec5027f0c5" containerID="fefc27ffe1f8009b18ef35dbb73dbff171c885744bbf0a763e1806f6d7bcbc7d" exitCode=0 Oct 31 13:31:36.788390 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:36.788240 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j" event={"ID":"86ce378f-511a-4301-a0c7-85ec5027f0c5","Type":"ContainerDied","Data":"fefc27ffe1f8009b18ef35dbb73dbff171c885744bbf0a763e1806f6d7bcbc7d"} Oct 31 13:31:36.934185 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:36.934160 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j" Oct 31 13:31:36.995975 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:36.995942 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ce378f-511a-4301-a0c7-85ec5027f0c5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/86ce378f-511a-4301-a0c7-85ec5027f0c5-dshm\") pod \"86ce378f-511a-4301-a0c7-85ec5027f0c5\" (UID: \"86ce378f-511a-4301-a0c7-85ec5027f0c5\") " Oct 31 13:31:36.996414 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:36.996031 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86ce378f-511a-4301-a0c7-85ec5027f0c5-kube-api-access-zqhwb\" (UniqueName: \"kubernetes.io/projected/86ce378f-511a-4301-a0c7-85ec5027f0c5-kube-api-access-zqhwb\") pod \"86ce378f-511a-4301-a0c7-85ec5027f0c5\" (UID: \"86ce378f-511a-4301-a0c7-85ec5027f0c5\") " Oct 31 13:31:36.996414 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:36.996110 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ce378f-511a-4301-a0c7-85ec5027f0c5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86ce378f-511a-4301-a0c7-85ec5027f0c5-model-cache\") pod \"86ce378f-511a-4301-a0c7-85ec5027f0c5\" (UID: \"86ce378f-511a-4301-a0c7-85ec5027f0c5\") " Oct 31 13:31:36.996414 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:36.996207 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ce378f-511a-4301-a0c7-85ec5027f0c5-home\" (UniqueName: \"kubernetes.io/empty-dir/86ce378f-511a-4301-a0c7-85ec5027f0c5-home\") pod \"86ce378f-511a-4301-a0c7-85ec5027f0c5\" (UID: \"86ce378f-511a-4301-a0c7-85ec5027f0c5\") " Oct 31 13:31:36.996589 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:36.996413 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ce378f-511a-4301-a0c7-85ec5027f0c5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86ce378f-511a-4301-a0c7-85ec5027f0c5-tmp-dir\") pod \"86ce378f-511a-4301-a0c7-85ec5027f0c5\" (UID: \"86ce378f-511a-4301-a0c7-85ec5027f0c5\") " Oct 31 13:31:36.996589 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:36.996457 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ce378f-511a-4301-a0c7-85ec5027f0c5-model-cache" pod "86ce378f-511a-4301-a0c7-85ec5027f0c5" (UID: "86ce378f-511a-4301-a0c7-85ec5027f0c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:36.996589 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:36.996482 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86ce378f-511a-4301-a0c7-85ec5027f0c5-tls-certs\" (UniqueName: \"kubernetes.io/secret/86ce378f-511a-4301-a0c7-85ec5027f0c5-tls-certs\") pod \"86ce378f-511a-4301-a0c7-85ec5027f0c5\" (UID: \"86ce378f-511a-4301-a0c7-85ec5027f0c5\") " Oct 31 13:31:36.996589 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:36.996493 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ce378f-511a-4301-a0c7-85ec5027f0c5-home" pod "86ce378f-511a-4301-a0c7-85ec5027f0c5" (UID: "86ce378f-511a-4301-a0c7-85ec5027f0c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:36.997019 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:36.996670 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ce378f-511a-4301-a0c7-85ec5027f0c5-tmp-dir" pod "86ce378f-511a-4301-a0c7-85ec5027f0c5" (UID: "86ce378f-511a-4301-a0c7-85ec5027f0c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:36.997311 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:36.997288 6841 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86ce378f-511a-4301-a0c7-85ec5027f0c5-model-cache\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:36.997311 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:36.997311 6841 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86ce378f-511a-4301-a0c7-85ec5027f0c5-home\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:36.997470 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:36.997320 6841 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86ce378f-511a-4301-a0c7-85ec5027f0c5-tmp-dir\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:36.998763 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:36.998736 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ce378f-511a-4301-a0c7-85ec5027f0c5-dshm" pod "86ce378f-511a-4301-a0c7-85ec5027f0c5" (UID: "86ce378f-511a-4301-a0c7-85ec5027f0c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:36.998928 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:36.998898 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86ce378f-511a-4301-a0c7-85ec5027f0c5-tls-certs" pod "86ce378f-511a-4301-a0c7-85ec5027f0c5" (UID: "86ce378f-511a-4301-a0c7-85ec5027f0c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:31:36.999079 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:36.998934 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ce378f-511a-4301-a0c7-85ec5027f0c5-kube-api-access-zqhwb" pod "86ce378f-511a-4301-a0c7-85ec5027f0c5" (UID: "86ce378f-511a-4301-a0c7-85ec5027f0c5"). InnerVolumeSpecName "kube-api-access-zqhwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:37.099125 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:37.099090 6841 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86ce378f-511a-4301-a0c7-85ec5027f0c5-dshm\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:37.099125 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:37.099121 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqhwb\" (UniqueName: \"kubernetes.io/projected/86ce378f-511a-4301-a0c7-85ec5027f0c5-kube-api-access-zqhwb\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:37.099331 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:37.099134 6841 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86ce378f-511a-4301-a0c7-85ec5027f0c5-tls-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:37.604247 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:37.604197 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:37.797376 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:37.797352 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j" Oct 31 13:31:37.797553 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:37.797394 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j" event={"ID":"86ce378f-511a-4301-a0c7-85ec5027f0c5","Type":"ContainerDied","Data":"0eb3b7ba0a976276e4a87a655151fa94eb7744e64082508846e849adc275d311"} Oct 31 13:31:37.797553 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:37.797452 6841 scope.go:122] "RemoveContainer" containerID="fefc27ffe1f8009b18ef35dbb73dbff171c885744bbf0a763e1806f6d7bcbc7d" Oct 31 13:31:37.827469 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:37.827432 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j"] Oct 31 13:31:37.831868 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:37.831842 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-5dbd39/test-model-5dbd39-kserve-85ddc47969-7947j"] Oct 31 13:31:38.121469 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:38.121432 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz"] Oct 31 13:31:38.346911 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:38.346408 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:38.566477 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:38.566442 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:38.838170 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:38.838091 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:39.447466 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:39.447426 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:39.644816 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:39.644783 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86ce378f-511a-4301-a0c7-85ec5027f0c5" path="/var/lib/kubelet/pods/86ce378f-511a-4301-a0c7-85ec5027f0c5/volumes" Oct 31 13:31:39.856441 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:39.856366 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:40.516772 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:40.516737 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:41.336103 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:41.334904 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:42.068149 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:42.068111 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:42.767463 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:42.767428 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:43.939172 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:43.939140 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:44.150880 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:44.150846 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:45.043592 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:45.043554 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:45.264995 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:45.264956 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:45.536007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:45.535442 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:45.885934 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:45.885851 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5" podUID="50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1" containerName="main" containerID="cri-o://ed4c5c8602b88576a80a2900b35ac10725362d7be23c870d51e8c4b36317de36" gracePeriod=45 Oct 31 13:31:46.012017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:46.008501 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:46.207910 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:46.207873 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:46.221549 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:46.221523 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5" Oct 31 13:31:46.342916 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:46.342880 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-model-cache\") pod \"50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1\" (UID: \"50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1\") " Oct 31 13:31:46.343171 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:46.343113 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-dshm\") pod \"50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1\" (UID: \"50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1\") " Oct 31 13:31:46.343171 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:46.343151 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-model-cache" pod "50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1" (UID: "50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:46.343308 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:46.343197 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-tmp-dir\") pod \"50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1\" (UID: \"50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1\") " Oct 31 13:31:46.343308 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:46.343247 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-tls-certs\" (UniqueName: \"kubernetes.io/secret/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-tls-certs\") pod \"50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1\" (UID: \"50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1\") " Oct 31 13:31:46.343421 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:46.343341 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-kube-api-access-rx94f\" (UniqueName: \"kubernetes.io/projected/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-kube-api-access-rx94f\") pod \"50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1\" (UID: \"50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1\") " Oct 31 13:31:46.343421 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:46.343414 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-home\" (UniqueName: \"kubernetes.io/empty-dir/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-home\") pod \"50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1\" (UID: \"50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1\") " Oct 31 13:31:46.343578 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:46.343541 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-tmp-dir" pod "50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1" (UID: "50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:46.344048 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:46.344007 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-home" pod "50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1" (UID: "50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:46.344421 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:46.344371 6841 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-model-cache\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:46.344421 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:46.344414 6841 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-tmp-dir\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:46.344598 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:46.344429 6841 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-home\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:46.346049 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:46.346018 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-dshm" pod "50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1" (UID: "50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:46.346049 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:46.346316 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-kube-api-access-rx94f" pod "50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1" (UID: "50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1"). InnerVolumeSpecName "kube-api-access-rx94f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:46.346049 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:46.346341 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-tls-certs" pod "50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1" (UID: "50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:31:46.445834 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:46.445810 6841 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-dshm\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:46.445834 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:46.445835 6841 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-tls-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:46.446042 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:46.445846 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx94f\" (UniqueName: \"kubernetes.io/projected/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1-kube-api-access-rx94f\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:46.891853 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:46.891779 6841 generic.go:356] "Generic (PLEG): container finished" podID="50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1" containerID="ed4c5c8602b88576a80a2900b35ac10725362d7be23c870d51e8c4b36317de36" exitCode=0 Oct 31 13:31:46.891853 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:46.891827 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5" Oct 31 13:31:46.891853 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:46.891845 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5" event={"ID":"50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1","Type":"ContainerDied","Data":"ed4c5c8602b88576a80a2900b35ac10725362d7be23c870d51e8c4b36317de36"} Oct 31 13:31:46.892150 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:46.891892 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5" event={"ID":"50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1","Type":"ContainerDied","Data":"e3155cfc211122df21e49d2722ac709ee69709f9d85cee888bd1384bce3bee20"} Oct 31 13:31:46.892150 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:46.891915 6841 scope.go:122] "RemoveContainer" containerID="ed4c5c8602b88576a80a2900b35ac10725362d7be23c870d51e8c4b36317de36" Oct 31 13:31:46.907410 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:46.907282 6841 scope.go:122] "RemoveContainer" containerID="ed4c5c8602b88576a80a2900b35ac10725362d7be23c870d51e8c4b36317de36" Oct 31 13:31:46.914456 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:31:46.908692 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed4c5c8602b88576a80a2900b35ac10725362d7be23c870d51e8c4b36317de36\": container with ID starting with ed4c5c8602b88576a80a2900b35ac10725362d7be23c870d51e8c4b36317de36 not found: ID does not exist" containerID="ed4c5c8602b88576a80a2900b35ac10725362d7be23c870d51e8c4b36317de36" Oct 31 13:31:46.914456 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:46.908736 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed4c5c8602b88576a80a2900b35ac10725362d7be23c870d51e8c4b36317de36"} err="failed to get container status \"ed4c5c8602b88576a80a2900b35ac10725362d7be23c870d51e8c4b36317de36\": rpc error: code = NotFound desc = could not find container \"ed4c5c8602b88576a80a2900b35ac10725362d7be23c870d51e8c4b36317de36\": container with ID starting with ed4c5c8602b88576a80a2900b35ac10725362d7be23c870d51e8c4b36317de36 not found: ID does not exist" Oct 31 13:31:46.950216 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:46.950129 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:46.950216 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:46.950170 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5"] Oct 31 13:31:46.955855 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:46.955738 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-4023ad/test-model-4023ad-kserve-85694f8f59-5ggn5"] Oct 31 13:31:47.213067 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.213034 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:47.524388 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.524275 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:47.627004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.624026 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7"] Oct 31 13:31:47.627004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.624258 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7" podUID="df0aa4b6-4450-4322-afca-7cfd6c6cf330" containerName="maas-api" containerID="cri-o://e4d01cca31f6cfc709a8f957316d78dd1394b60edacbb9acea4d0d5bace8be17" gracePeriod=30 Oct 31 13:31:47.657114 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.657081 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1" path="/var/lib/kubelet/pods/50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1/volumes" Oct 31 13:31:47.762130 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.762095 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q"] Oct 31 13:31:47.764222 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.763476 6841 cpu_manager.go:414] "containerMap: removing container" podUID="24ea07e9-65f1-4208-b5db-1d283caf05e1" containerName="istio-proxy" Oct 31 13:31:47.764222 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.763500 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24ea07e9-65f1-4208-b5db-1d283caf05e1" containerName="istio-proxy" Oct 31 13:31:47.764222 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.763574 6841 cpu_manager.go:414] "containerMap: removing container" podUID="86ce378f-511a-4301-a0c7-85ec5027f0c5" containerName="main" Oct 31 13:31:47.764222 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.763583 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86ce378f-511a-4301-a0c7-85ec5027f0c5" containerName="main" Oct 31 13:31:47.764222 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.763599 6841 cpu_manager.go:414] "containerMap: removing container" podUID="50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1" containerName="main" Oct 31 13:31:47.764222 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.763607 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1" containerName="main" Oct 31 13:31:47.764222 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.763791 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="24ea07e9-65f1-4208-b5db-1d283caf05e1" containerName="istio-proxy" Oct 31 13:31:47.764222 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.763813 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="86ce378f-511a-4301-a0c7-85ec5027f0c5" containerName="main" Oct 31 13:31:47.764222 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.763825 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="50fa0a40-a8cb-4da7-bd3a-65dacb0de7e1" containerName="main" Oct 31 13:31:47.768966 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.768940 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:31:47.781390 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.781364 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-50b4aa-openshift-default-dockercfg-5fzkn\"" Oct 31 13:31:47.795573 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.795541 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q"] Oct 31 13:31:47.805602 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.805570 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:47.807339 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.807310 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a4d6d4-06ba-4385-94cc-66f96e07232a-istio-data\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q\" (UID: \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:31:47.807510 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.807485 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7a4d6d4-06ba-4385-94cc-66f96e07232a-istio-token\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q\" (UID: \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:31:47.807616 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.807569 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2tj4\" (UniqueName: \"kubernetes.io/projected/c7a4d6d4-06ba-4385-94cc-66f96e07232a-kube-api-access-b2tj4\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q\" (UID: \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:31:47.807748 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.807645 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a4d6d4-06ba-4385-94cc-66f96e07232a-istiod-ca-cert\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q\" (UID: \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:31:47.807748 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.807684 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a4d6d4-06ba-4385-94cc-66f96e07232a-workload-socket\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q\" (UID: \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:31:47.808003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.807960 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a4d6d4-06ba-4385-94cc-66f96e07232a-istio-envoy\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q\" (UID: \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:31:47.808108 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.808086 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a4d6d4-06ba-4385-94cc-66f96e07232a-credential-socket\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q\" (UID: \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:31:47.808180 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.808130 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a4d6d4-06ba-4385-94cc-66f96e07232a-istio-podinfo\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q\" (UID: \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:31:47.808532 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.808509 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a4d6d4-06ba-4385-94cc-66f96e07232a-workload-certs\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q\" (UID: \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:31:47.933155 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.928562 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a4d6d4-06ba-4385-94cc-66f96e07232a-istio-envoy\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q\" (UID: \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:31:47.933155 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.928660 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a4d6d4-06ba-4385-94cc-66f96e07232a-credential-socket\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q\" (UID: \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:31:47.933155 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.928694 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a4d6d4-06ba-4385-94cc-66f96e07232a-istio-podinfo\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q\" (UID: \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:31:47.933155 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.928921 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a4d6d4-06ba-4385-94cc-66f96e07232a-workload-certs\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q\" (UID: \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:31:47.933155 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.928954 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a4d6d4-06ba-4385-94cc-66f96e07232a-istio-data\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q\" (UID: \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:31:47.933155 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.929086 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7a4d6d4-06ba-4385-94cc-66f96e07232a-istio-token\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q\" (UID: \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:31:47.933155 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.929116 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2tj4\" (UniqueName: \"kubernetes.io/projected/c7a4d6d4-06ba-4385-94cc-66f96e07232a-kube-api-access-b2tj4\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q\" (UID: \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:31:47.933155 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.929177 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a4d6d4-06ba-4385-94cc-66f96e07232a-istiod-ca-cert\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q\" (UID: \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:31:47.933155 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.929213 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a4d6d4-06ba-4385-94cc-66f96e07232a-workload-socket\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q\" (UID: \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:31:47.933155 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.929833 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a4d6d4-06ba-4385-94cc-66f96e07232a-workload-socket\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q\" (UID: \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:31:48.148059 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:47.996483 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a4d6d4-06ba-4385-94cc-66f96e07232a-credential-socket\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q\" (UID: \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:31:48.148059 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:48.027807 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a4d6d4-06ba-4385-94cc-66f96e07232a-workload-certs\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q\" (UID: \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:31:48.148059 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:48.083782 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a4d6d4-06ba-4385-94cc-66f96e07232a-istio-data\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q\" (UID: \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:31:48.148059 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:48.100046 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a4d6d4-06ba-4385-94cc-66f96e07232a-istiod-ca-cert\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q\" (UID: \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:31:48.148059 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:48.100552 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a4d6d4-06ba-4385-94cc-66f96e07232a-istio-envoy\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q\" (UID: \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:31:48.148059 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:48.109689 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:48.324100 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:48.322532 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7a4d6d4-06ba-4385-94cc-66f96e07232a-istio-token\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q\" (UID: \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:31:48.324100 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:48.323287 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a4d6d4-06ba-4385-94cc-66f96e07232a-istio-podinfo\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q\" (UID: \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:31:48.327174 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:48.327084 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2tj4\" (UniqueName: \"kubernetes.io/projected/c7a4d6d4-06ba-4385-94cc-66f96e07232a-kube-api-access-b2tj4\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q\" (UID: \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:31:48.328816 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:48.328509 6841 generic.go:356] "Generic (PLEG): container finished" podID="df0aa4b6-4450-4322-afca-7cfd6c6cf330" containerID="e4d01cca31f6cfc709a8f957316d78dd1394b60edacbb9acea4d0d5bace8be17" exitCode=0 Oct 31 13:31:48.328816 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:48.328585 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7" event={"ID":"df0aa4b6-4450-4322-afca-7cfd6c6cf330","Type":"ContainerDied","Data":"e4d01cca31f6cfc709a8f957316d78dd1394b60edacbb9acea4d0d5bace8be17"} Oct 31 13:31:48.329039 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:48.328917 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:48.356972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:48.355213 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7" Oct 31 13:31:48.389570 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:48.389537 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:31:48.394284 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:48.394250 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df0aa4b6-4450-4322-afca-7cfd6c6cf330-kube-api-access-5ttw4\" (UniqueName: \"kubernetes.io/projected/df0aa4b6-4450-4322-afca-7cfd6c6cf330-kube-api-access-5ttw4\") pod \"df0aa4b6-4450-4322-afca-7cfd6c6cf330\" (UID: \"df0aa4b6-4450-4322-afca-7cfd6c6cf330\") " Oct 31 13:31:48.394554 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:48.394532 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df0aa4b6-4450-4322-afca-7cfd6c6cf330-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df0aa4b6-4450-4322-afca-7cfd6c6cf330-maas-api-tls\") pod \"df0aa4b6-4450-4322-afca-7cfd6c6cf330\" (UID: \"df0aa4b6-4450-4322-afca-7cfd6c6cf330\") " Oct 31 13:31:48.394635 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:48.394587 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df0aa4b6-4450-4322-afca-7cfd6c6cf330-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df0aa4b6-4450-4322-afca-7cfd6c6cf330-openshift-service-ca\") pod \"df0aa4b6-4450-4322-afca-7cfd6c6cf330\" (UID: \"df0aa4b6-4450-4322-afca-7cfd6c6cf330\") " Oct 31 13:31:48.394695 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:48.394642 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df0aa4b6-4450-4322-afca-7cfd6c6cf330-metrics-certs\" (UniqueName: \"kubernetes.io/secret/df0aa4b6-4450-4322-afca-7cfd6c6cf330-metrics-certs\") pod \"df0aa4b6-4450-4322-afca-7cfd6c6cf330\" (UID: \"df0aa4b6-4450-4322-afca-7cfd6c6cf330\") " Oct 31 13:31:48.395872 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:48.395841 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df0aa4b6-4450-4322-afca-7cfd6c6cf330-openshift-service-ca" pod "df0aa4b6-4450-4322-afca-7cfd6c6cf330" (UID: "df0aa4b6-4450-4322-afca-7cfd6c6cf330"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:31:48.400449 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:48.400381 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df0aa4b6-4450-4322-afca-7cfd6c6cf330-maas-api-tls" pod "df0aa4b6-4450-4322-afca-7cfd6c6cf330" (UID: "df0aa4b6-4450-4322-afca-7cfd6c6cf330"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:31:48.400449 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:48.400420 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df0aa4b6-4450-4322-afca-7cfd6c6cf330-kube-api-access-5ttw4" pod "df0aa4b6-4450-4322-afca-7cfd6c6cf330" (UID: "df0aa4b6-4450-4322-afca-7cfd6c6cf330"). InnerVolumeSpecName "kube-api-access-5ttw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:48.401934 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:48.401906 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df0aa4b6-4450-4322-afca-7cfd6c6cf330-metrics-certs" pod "df0aa4b6-4450-4322-afca-7cfd6c6cf330" (UID: "df0aa4b6-4450-4322-afca-7cfd6c6cf330"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:31:48.499323 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:48.498823 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df0aa4b6-4450-4322-afca-7cfd6c6cf330-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:48.499323 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:48.498863 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df0aa4b6-4450-4322-afca-7cfd6c6cf330-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:48.499323 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:48.498877 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df0aa4b6-4450-4322-afca-7cfd6c6cf330-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:48.499323 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:48.498891 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ttw4\" (UniqueName: \"kubernetes.io/projected/df0aa4b6-4450-4322-afca-7cfd6c6cf330-kube-api-access-5ttw4\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:48.752118 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:48.750614 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q"] Oct 31 13:31:48.765121 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:31:48.765087 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7a4d6d4_06ba_4385_94cc_66f96e07232a.slice/crio-7d018b28b7ca5189fc61b3640f309dd128568d465fed1474727787925c2a6e4c WatchSource:0}: Error finding container 7d018b28b7ca5189fc61b3640f309dd128568d465fed1474727787925c2a6e4c: Status 404 returned error can't find the container with id 7d018b28b7ca5189fc61b3640f309dd128568d465fed1474727787925c2a6e4c Oct 31 13:31:48.771840 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:48.771798 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:31:48.771949 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:48.771896 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:31:48.771949 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:48.771935 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:31:49.001037 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:49.000964 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:49.344707 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:49.344630 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7" Oct 31 13:31:49.344707 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:49.344660 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7" event={"ID":"df0aa4b6-4450-4322-afca-7cfd6c6cf330","Type":"ContainerDied","Data":"0e1289fb397a953af3a0301bf3ac2ac1972b014a459d37fda58bfc0690c5de4e"} Oct 31 13:31:49.345161 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:49.344725 6841 scope.go:122] "RemoveContainer" containerID="e4d01cca31f6cfc709a8f957316d78dd1394b60edacbb9acea4d0d5bace8be17" Oct 31 13:31:49.346814 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:49.346745 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:49.347300 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:49.347272 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" event={"ID":"c7a4d6d4-06ba-4385-94cc-66f96e07232a","Type":"ContainerStarted","Data":"116ceac509139ce93124e720a1f41afb0822806e9fb44e48bb37f30562e8232a"} Oct 31 13:31:49.347405 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:49.347338 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" event={"ID":"c7a4d6d4-06ba-4385-94cc-66f96e07232a","Type":"ContainerStarted","Data":"7d018b28b7ca5189fc61b3640f309dd128568d465fed1474727787925c2a6e4c"} Oct 31 13:31:49.384198 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:49.384156 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" podStartSLOduration=2.38414232 podStartE2EDuration="2.38414232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:31:49.377662049 +0000 UTC m=+3555.860784974" watchObservedRunningTime="2026-10-31 13:31:49.38414232 +0000 UTC m=+3555.867265221" Oct 31 13:31:49.389894 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:49.389871 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:31:49.396169 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:49.396145 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:31:49.398122 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:49.398097 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7"] Oct 31 13:31:49.404153 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:49.404129 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5dbd39-6c6fc5c9f5-v6cq7"] Oct 31 13:31:49.646825 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:49.646733 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df0aa4b6-4450-4322-afca-7cfd6c6cf330" path="/var/lib/kubelet/pods/df0aa4b6-4450-4322-afca-7cfd6c6cf330/volumes" Oct 31 13:31:49.936874 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:49.936735 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:50.036834 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:31:50.036802 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:31:50.408852 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:50.408823 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:31:50.409771 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:50.409751 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:31:50.431828 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:50.431796 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:51.499898 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:51.499856 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:51.983053 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:51.981972 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:52.284061 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:52.283935 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:53.093005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:53.088847 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:53.167408 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:53.167367 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz" podUID="28f4f307-72eb-4607-8ca9-547bdecd1ed0" containerName="main" containerID="cri-o://d7c128aa94aff8db71f57bc1e469ef7c6fec00e0f2d60ec00cc871bc7d9c2043" gracePeriod=45 Oct 31 13:31:53.353639 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:53.352736 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz" podUID="28f4f307-72eb-4607-8ca9-547bdecd1ed0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.213:8000/ready\": EOF" Oct 31 13:31:53.484470 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:53.484435 6841 generic.go:356] "Generic (PLEG): container finished" podID="28f4f307-72eb-4607-8ca9-547bdecd1ed0" containerID="d7c128aa94aff8db71f57bc1e469ef7c6fec00e0f2d60ec00cc871bc7d9c2043" exitCode=0 Oct 31 13:31:53.484662 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:53.484516 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz" event={"ID":"28f4f307-72eb-4607-8ca9-547bdecd1ed0","Type":"ContainerDied","Data":"d7c128aa94aff8db71f57bc1e469ef7c6fec00e0f2d60ec00cc871bc7d9c2043"} Oct 31 13:31:53.693150 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:53.693117 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:53.943087 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:53.939562 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:54.124004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:54.119259 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz" podUID="28f4f307-72eb-4607-8ca9-547bdecd1ed0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.213:8000/ready\": dial tcp 10.134.0.213:8000: connect: connection refused" Oct 31 13:31:54.124004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:54.119396 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz" Oct 31 13:31:54.377353 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:54.377324 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz" Oct 31 13:31:54.483337 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:54.483298 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28f4f307-72eb-4607-8ca9-547bdecd1ed0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28f4f307-72eb-4607-8ca9-547bdecd1ed0-tmp-dir\") pod \"28f4f307-72eb-4607-8ca9-547bdecd1ed0\" (UID: \"28f4f307-72eb-4607-8ca9-547bdecd1ed0\") " Oct 31 13:31:54.483581 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:54.483494 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28f4f307-72eb-4607-8ca9-547bdecd1ed0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28f4f307-72eb-4607-8ca9-547bdecd1ed0-model-cache\") pod \"28f4f307-72eb-4607-8ca9-547bdecd1ed0\" (UID: \"28f4f307-72eb-4607-8ca9-547bdecd1ed0\") " Oct 31 13:31:54.483581 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:54.483552 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28f4f307-72eb-4607-8ca9-547bdecd1ed0-kube-api-access-77r59\" (UniqueName: \"kubernetes.io/projected/28f4f307-72eb-4607-8ca9-547bdecd1ed0-kube-api-access-77r59\") pod \"28f4f307-72eb-4607-8ca9-547bdecd1ed0\" (UID: \"28f4f307-72eb-4607-8ca9-547bdecd1ed0\") " Oct 31 13:31:54.483751 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:54.483638 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28f4f307-72eb-4607-8ca9-547bdecd1ed0-home\" (UniqueName: \"kubernetes.io/empty-dir/28f4f307-72eb-4607-8ca9-547bdecd1ed0-home\") pod \"28f4f307-72eb-4607-8ca9-547bdecd1ed0\" (UID: \"28f4f307-72eb-4607-8ca9-547bdecd1ed0\") " Oct 31 13:31:54.483751 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:54.483705 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28f4f307-72eb-4607-8ca9-547bdecd1ed0-tls-certs\" (UniqueName: \"kubernetes.io/secret/28f4f307-72eb-4607-8ca9-547bdecd1ed0-tls-certs\") pod \"28f4f307-72eb-4607-8ca9-547bdecd1ed0\" (UID: \"28f4f307-72eb-4607-8ca9-547bdecd1ed0\") " Oct 31 13:31:54.483751 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:54.483736 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28f4f307-72eb-4607-8ca9-547bdecd1ed0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/28f4f307-72eb-4607-8ca9-547bdecd1ed0-dshm\") pod \"28f4f307-72eb-4607-8ca9-547bdecd1ed0\" (UID: \"28f4f307-72eb-4607-8ca9-547bdecd1ed0\") " Oct 31 13:31:54.502975 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:54.487193 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28f4f307-72eb-4607-8ca9-547bdecd1ed0-home" pod "28f4f307-72eb-4607-8ca9-547bdecd1ed0" (UID: "28f4f307-72eb-4607-8ca9-547bdecd1ed0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:54.502975 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:54.495298 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28f4f307-72eb-4607-8ca9-547bdecd1ed0-tmp-dir" pod "28f4f307-72eb-4607-8ca9-547bdecd1ed0" (UID: "28f4f307-72eb-4607-8ca9-547bdecd1ed0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:54.502975 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:54.495487 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28f4f307-72eb-4607-8ca9-547bdecd1ed0-model-cache" pod "28f4f307-72eb-4607-8ca9-547bdecd1ed0" (UID: "28f4f307-72eb-4607-8ca9-547bdecd1ed0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:54.502975 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:54.498186 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28f4f307-72eb-4607-8ca9-547bdecd1ed0-dshm" pod "28f4f307-72eb-4607-8ca9-547bdecd1ed0" (UID: "28f4f307-72eb-4607-8ca9-547bdecd1ed0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:54.511339 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:54.505762 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28f4f307-72eb-4607-8ca9-547bdecd1ed0-tls-certs" pod "28f4f307-72eb-4607-8ca9-547bdecd1ed0" (UID: "28f4f307-72eb-4607-8ca9-547bdecd1ed0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:31:54.511339 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:54.506259 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz" event={"ID":"28f4f307-72eb-4607-8ca9-547bdecd1ed0","Type":"ContainerDied","Data":"f290963236661374aaeee855fab70b959a7134af90e03b775d569a3a94f52978"} Oct 31 13:31:54.511339 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:54.506314 6841 scope.go:122] "RemoveContainer" containerID="d7c128aa94aff8db71f57bc1e469ef7c6fec00e0f2d60ec00cc871bc7d9c2043" Oct 31 13:31:54.511339 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:54.506482 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz" Oct 31 13:31:54.515750 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:54.515708 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28f4f307-72eb-4607-8ca9-547bdecd1ed0-kube-api-access-77r59" pod "28f4f307-72eb-4607-8ca9-547bdecd1ed0" (UID: "28f4f307-72eb-4607-8ca9-547bdecd1ed0"). InnerVolumeSpecName "kube-api-access-77r59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:54.543973 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:54.543939 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:54.589040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:54.588552 6841 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28f4f307-72eb-4607-8ca9-547bdecd1ed0-model-cache\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:54.589040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:54.588614 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77r59\" (UniqueName: \"kubernetes.io/projected/28f4f307-72eb-4607-8ca9-547bdecd1ed0-kube-api-access-77r59\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:54.589040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:54.588632 6841 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28f4f307-72eb-4607-8ca9-547bdecd1ed0-home\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:54.589040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:54.588646 6841 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28f4f307-72eb-4607-8ca9-547bdecd1ed0-tls-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:54.589040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:54.588659 6841 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28f4f307-72eb-4607-8ca9-547bdecd1ed0-dshm\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:54.589040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:54.588670 6841 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28f4f307-72eb-4607-8ca9-547bdecd1ed0-tmp-dir\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:54.805157 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:54.805120 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:54.916006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:54.907074 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz"] Oct 31 13:31:54.954006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:54.951349 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-384dbe/test-model-384dbe-kserve-7f464759f5-gnqhz"] Oct 31 13:31:55.550850 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:55.550788 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:55.645512 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:55.645478 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28f4f307-72eb-4607-8ca9-547bdecd1ed0" path="/var/lib/kubelet/pods/28f4f307-72eb-4607-8ca9-547bdecd1ed0/volumes" Oct 31 13:31:55.802320 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:55.802231 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w"] Oct 31 13:31:55.802524 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:55.802498 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" podUID="777dacd7-0cef-42aa-b932-f25c78e7fb40" containerName="istio-proxy" containerID="cri-o://d818ed145097da59e8f3e72625ae399fddfaf70ef6f9b2c6a3aedef7011e61b6" gracePeriod=30 Oct 31 13:31:55.806922 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:55.806900 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h"] Oct 31 13:31:55.807111 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:55.807093 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" podUID="31d9443c-5638-44ee-af89-e78960decfab" containerName="istio-proxy" containerID="cri-o://09ee9e5baf6944e6c7757bacf382ce9b36b9f45d6a43f4df7deca581cd3473ae" gracePeriod=30 Oct 31 13:31:55.933037 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:55.932951 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:56.244914 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:56.244874 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:56.265539 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:56.265506 6841 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:31:56.265697 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:56.265559 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" podUID="31d9443c-5638-44ee-af89-e78960decfab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:31:57.609597 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:57.609560 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:58.261113 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:58.261083 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:31:59.234918 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:31:59.234883 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:00.118132 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:00.118091 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:00.410732 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:00.410636 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g"] Oct 31 13:32:00.411577 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:00.411554 6841 cpu_manager.go:414] "containerMap: removing container" podUID="28f4f307-72eb-4607-8ca9-547bdecd1ed0" containerName="main" Oct 31 13:32:00.411577 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:00.411575 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28f4f307-72eb-4607-8ca9-547bdecd1ed0" containerName="main" Oct 31 13:32:00.411765 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:00.411598 6841 cpu_manager.go:414] "containerMap: removing container" podUID="df0aa4b6-4450-4322-afca-7cfd6c6cf330" containerName="maas-api" Oct 31 13:32:00.411765 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:00.411603 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df0aa4b6-4450-4322-afca-7cfd6c6cf330" containerName="maas-api" Oct 31 13:32:00.411890 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:00.411775 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="28f4f307-72eb-4607-8ca9-547bdecd1ed0" containerName="main" Oct 31 13:32:00.411890 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:00.411787 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="df0aa4b6-4450-4322-afca-7cfd6c6cf330" containerName="maas-api" Oct 31 13:32:00.417672 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:00.417649 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g" Oct 31 13:32:00.427792 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:00.427746 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-691f1e\"/\"default-dockercfg-jllb4\"" Oct 31 13:32:00.428062 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:00.427851 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-691f1e\"/\"kube-root-ca.crt\"" Oct 31 13:32:00.429486 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:00.429464 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-691f1e\"/\"openshift-service-ca.crt\"" Oct 31 13:32:00.430068 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:00.430040 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-691f1e\"/\"test-model-691f1e-kserve-self-signed-certs\"" Oct 31 13:32:00.430068 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:00.430059 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g"] Oct 31 13:32:00.547210 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:00.547170 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/109fb6e5-38ef-4030-9209-cda454c140ec-tmp-dir\") pod \"test-model-691f1e-kserve-f8575cb55-dcg2g\" (UID: \"109fb6e5-38ef-4030-9209-cda454c140ec\") " pod="ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g" Oct 31 13:32:00.547381 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:00.547298 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/109fb6e5-38ef-4030-9209-cda454c140ec-home\") pod \"test-model-691f1e-kserve-f8575cb55-dcg2g\" (UID: \"109fb6e5-38ef-4030-9209-cda454c140ec\") " pod="ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g" Oct 31 13:32:00.547432 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:00.547387 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qbpp\" (UniqueName: \"kubernetes.io/projected/109fb6e5-38ef-4030-9209-cda454c140ec-kube-api-access-5qbpp\") pod \"test-model-691f1e-kserve-f8575cb55-dcg2g\" (UID: \"109fb6e5-38ef-4030-9209-cda454c140ec\") " pod="ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g" Oct 31 13:32:00.547629 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:00.547606 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/109fb6e5-38ef-4030-9209-cda454c140ec-dshm\") pod \"test-model-691f1e-kserve-f8575cb55-dcg2g\" (UID: \"109fb6e5-38ef-4030-9209-cda454c140ec\") " pod="ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g" Oct 31 13:32:00.547696 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:00.547681 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/109fb6e5-38ef-4030-9209-cda454c140ec-tls-certs\") pod \"test-model-691f1e-kserve-f8575cb55-dcg2g\" (UID: \"109fb6e5-38ef-4030-9209-cda454c140ec\") " pod="ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g" Oct 31 13:32:00.547819 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:00.547804 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/109fb6e5-38ef-4030-9209-cda454c140ec-model-cache\") pod \"test-model-691f1e-kserve-f8575cb55-dcg2g\" (UID: \"109fb6e5-38ef-4030-9209-cda454c140ec\") " pod="ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g" Oct 31 13:32:00.580305 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:00.580271 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:00.666052 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:00.662557 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/109fb6e5-38ef-4030-9209-cda454c140ec-dshm\") pod \"test-model-691f1e-kserve-f8575cb55-dcg2g\" (UID: \"109fb6e5-38ef-4030-9209-cda454c140ec\") " pod="ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g" Oct 31 13:32:00.666052 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:00.662657 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/109fb6e5-38ef-4030-9209-cda454c140ec-tls-certs\") pod \"test-model-691f1e-kserve-f8575cb55-dcg2g\" (UID: \"109fb6e5-38ef-4030-9209-cda454c140ec\") " pod="ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g" Oct 31 13:32:00.666052 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:00.662772 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/109fb6e5-38ef-4030-9209-cda454c140ec-model-cache\") pod \"test-model-691f1e-kserve-f8575cb55-dcg2g\" (UID: \"109fb6e5-38ef-4030-9209-cda454c140ec\") " pod="ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g" Oct 31 13:32:00.666052 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:00.662850 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/109fb6e5-38ef-4030-9209-cda454c140ec-tmp-dir\") pod \"test-model-691f1e-kserve-f8575cb55-dcg2g\" (UID: \"109fb6e5-38ef-4030-9209-cda454c140ec\") " pod="ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g" Oct 31 13:32:00.666052 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:00.662919 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/109fb6e5-38ef-4030-9209-cda454c140ec-home\") pod \"test-model-691f1e-kserve-f8575cb55-dcg2g\" (UID: \"109fb6e5-38ef-4030-9209-cda454c140ec\") " pod="ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g" Oct 31 13:32:00.666052 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:00.663032 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qbpp\" (UniqueName: \"kubernetes.io/projected/109fb6e5-38ef-4030-9209-cda454c140ec-kube-api-access-5qbpp\") pod \"test-model-691f1e-kserve-f8575cb55-dcg2g\" (UID: \"109fb6e5-38ef-4030-9209-cda454c140ec\") " pod="ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g" Oct 31 13:32:00.666052 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:00.664484 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/109fb6e5-38ef-4030-9209-cda454c140ec-model-cache\") pod \"test-model-691f1e-kserve-f8575cb55-dcg2g\" (UID: \"109fb6e5-38ef-4030-9209-cda454c140ec\") " pod="ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g" Oct 31 13:32:00.668359 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:00.668327 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/109fb6e5-38ef-4030-9209-cda454c140ec-tmp-dir\") pod \"test-model-691f1e-kserve-f8575cb55-dcg2g\" (UID: \"109fb6e5-38ef-4030-9209-cda454c140ec\") " pod="ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g" Oct 31 13:32:00.681657 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:00.668594 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/109fb6e5-38ef-4030-9209-cda454c140ec-home\") pod \"test-model-691f1e-kserve-f8575cb55-dcg2g\" (UID: \"109fb6e5-38ef-4030-9209-cda454c140ec\") " pod="ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g" Oct 31 13:32:00.692019 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:00.690632 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/109fb6e5-38ef-4030-9209-cda454c140ec-tls-certs\") pod \"test-model-691f1e-kserve-f8575cb55-dcg2g\" (UID: \"109fb6e5-38ef-4030-9209-cda454c140ec\") " pod="ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g" Oct 31 13:32:00.696313 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:00.693883 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qbpp\" (UniqueName: \"kubernetes.io/projected/109fb6e5-38ef-4030-9209-cda454c140ec-kube-api-access-5qbpp\") pod \"test-model-691f1e-kserve-f8575cb55-dcg2g\" (UID: \"109fb6e5-38ef-4030-9209-cda454c140ec\") " pod="ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g" Oct 31 13:32:00.698203 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:00.697691 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/109fb6e5-38ef-4030-9209-cda454c140ec-dshm\") pod \"test-model-691f1e-kserve-f8575cb55-dcg2g\" (UID: \"109fb6e5-38ef-4030-9209-cda454c140ec\") " pod="ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g" Oct 31 13:32:00.733817 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:00.733775 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g" Oct 31 13:32:01.137764 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.137719 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g"] Oct 31 13:32:01.171932 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.171873 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" Oct 31 13:32:01.208939 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.208912 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" Oct 31 13:32:01.282685 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.282643 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31d9443c-5638-44ee-af89-e78960decfab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31d9443c-5638-44ee-af89-e78960decfab-istio-data\") pod \"31d9443c-5638-44ee-af89-e78960decfab\" (UID: \"31d9443c-5638-44ee-af89-e78960decfab\") " Oct 31 13:32:01.287204 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.283452 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31d9443c-5638-44ee-af89-e78960decfab-istio-data" pod "31d9443c-5638-44ee-af89-e78960decfab" (UID: "31d9443c-5638-44ee-af89-e78960decfab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.287204 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.286819 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:01.291042 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.290900 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/777dacd7-0cef-42aa-b932-f25c78e7fb40-istio-podinfo" pod "777dacd7-0cef-42aa-b932-f25c78e7fb40" (UID: "777dacd7-0cef-42aa-b932-f25c78e7fb40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:32:01.291181 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.286977 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/777dacd7-0cef-42aa-b932-f25c78e7fb40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/777dacd7-0cef-42aa-b932-f25c78e7fb40-istio-podinfo\") pod \"777dacd7-0cef-42aa-b932-f25c78e7fb40\" (UID: \"777dacd7-0cef-42aa-b932-f25c78e7fb40\") " Oct 31 13:32:01.291375 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.291359 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/777dacd7-0cef-42aa-b932-f25c78e7fb40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/777dacd7-0cef-42aa-b932-f25c78e7fb40-credential-socket\") pod \"777dacd7-0cef-42aa-b932-f25c78e7fb40\" (UID: \"777dacd7-0cef-42aa-b932-f25c78e7fb40\") " Oct 31 13:32:01.291879 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.291848 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/777dacd7-0cef-42aa-b932-f25c78e7fb40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/777dacd7-0cef-42aa-b932-f25c78e7fb40-workload-socket\") pod \"777dacd7-0cef-42aa-b932-f25c78e7fb40\" (UID: \"777dacd7-0cef-42aa-b932-f25c78e7fb40\") " Oct 31 13:32:01.292374 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.292355 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/777dacd7-0cef-42aa-b932-f25c78e7fb40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/777dacd7-0cef-42aa-b932-f25c78e7fb40-istio-envoy\") pod \"777dacd7-0cef-42aa-b932-f25c78e7fb40\" (UID: \"777dacd7-0cef-42aa-b932-f25c78e7fb40\") " Oct 31 13:32:01.292547 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.292531 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31d9443c-5638-44ee-af89-e78960decfab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31d9443c-5638-44ee-af89-e78960decfab-istio-envoy\") pod \"31d9443c-5638-44ee-af89-e78960decfab\" (UID: \"31d9443c-5638-44ee-af89-e78960decfab\") " Oct 31 13:32:01.292664 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.292651 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31d9443c-5638-44ee-af89-e78960decfab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31d9443c-5638-44ee-af89-e78960decfab-istiod-ca-cert\") pod \"31d9443c-5638-44ee-af89-e78960decfab\" (UID: \"31d9443c-5638-44ee-af89-e78960decfab\") " Oct 31 13:32:01.292851 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.292836 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/777dacd7-0cef-42aa-b932-f25c78e7fb40-istio-token\" (UniqueName: \"kubernetes.io/projected/777dacd7-0cef-42aa-b932-f25c78e7fb40-istio-token\") pod \"777dacd7-0cef-42aa-b932-f25c78e7fb40\" (UID: \"777dacd7-0cef-42aa-b932-f25c78e7fb40\") " Oct 31 13:32:01.293245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.293217 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31d9443c-5638-44ee-af89-e78960decfab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31d9443c-5638-44ee-af89-e78960decfab-workload-certs\") pod \"31d9443c-5638-44ee-af89-e78960decfab\" (UID: \"31d9443c-5638-44ee-af89-e78960decfab\") " Oct 31 13:32:01.293443 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.291767 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/777dacd7-0cef-42aa-b932-f25c78e7fb40-credential-socket" pod "777dacd7-0cef-42aa-b932-f25c78e7fb40" (UID: "777dacd7-0cef-42aa-b932-f25c78e7fb40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.293535 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.292281 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/777dacd7-0cef-42aa-b932-f25c78e7fb40-workload-socket" pod "777dacd7-0cef-42aa-b932-f25c78e7fb40" (UID: "777dacd7-0cef-42aa-b932-f25c78e7fb40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.293611 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.293420 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31d9443c-5638-44ee-af89-e78960decfab-workload-certs" pod "31d9443c-5638-44ee-af89-e78960decfab" (UID: "31d9443c-5638-44ee-af89-e78960decfab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.293722 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.293698 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31d9443c-5638-44ee-af89-e78960decfab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d9443c-5638-44ee-af89-e78960decfab-workload-socket\") pod \"31d9443c-5638-44ee-af89-e78960decfab\" (UID: \"31d9443c-5638-44ee-af89-e78960decfab\") " Oct 31 13:32:01.293792 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.293739 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/777dacd7-0cef-42aa-b932-f25c78e7fb40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/777dacd7-0cef-42aa-b932-f25c78e7fb40-istio-data\") pod \"777dacd7-0cef-42aa-b932-f25c78e7fb40\" (UID: \"777dacd7-0cef-42aa-b932-f25c78e7fb40\") " Oct 31 13:32:01.293792 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.293771 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31d9443c-5638-44ee-af89-e78960decfab-istio-token\" (UniqueName: \"kubernetes.io/projected/31d9443c-5638-44ee-af89-e78960decfab-istio-token\") pod \"31d9443c-5638-44ee-af89-e78960decfab\" (UID: \"31d9443c-5638-44ee-af89-e78960decfab\") " Oct 31 13:32:01.293909 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.293803 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31d9443c-5638-44ee-af89-e78960decfab-kube-api-access-x4lw2\" (UniqueName: \"kubernetes.io/projected/31d9443c-5638-44ee-af89-e78960decfab-kube-api-access-x4lw2\") pod \"31d9443c-5638-44ee-af89-e78960decfab\" (UID: \"31d9443c-5638-44ee-af89-e78960decfab\") " Oct 31 13:32:01.293909 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.293832 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31d9443c-5638-44ee-af89-e78960decfab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31d9443c-5638-44ee-af89-e78960decfab-istio-podinfo\") pod \"31d9443c-5638-44ee-af89-e78960decfab\" (UID: \"31d9443c-5638-44ee-af89-e78960decfab\") " Oct 31 13:32:01.293909 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.293865 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31d9443c-5638-44ee-af89-e78960decfab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d9443c-5638-44ee-af89-e78960decfab-credential-socket\") pod \"31d9443c-5638-44ee-af89-e78960decfab\" (UID: \"31d9443c-5638-44ee-af89-e78960decfab\") " Oct 31 13:32:01.293909 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.293890 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/777dacd7-0cef-42aa-b932-f25c78e7fb40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/777dacd7-0cef-42aa-b932-f25c78e7fb40-istiod-ca-cert\") pod \"777dacd7-0cef-42aa-b932-f25c78e7fb40\" (UID: \"777dacd7-0cef-42aa-b932-f25c78e7fb40\") " Oct 31 13:32:01.294169 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.293937 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/777dacd7-0cef-42aa-b932-f25c78e7fb40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/777dacd7-0cef-42aa-b932-f25c78e7fb40-workload-certs\") pod \"777dacd7-0cef-42aa-b932-f25c78e7fb40\" (UID: \"777dacd7-0cef-42aa-b932-f25c78e7fb40\") " Oct 31 13:32:01.294169 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.294160 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/777dacd7-0cef-42aa-b932-f25c78e7fb40-kube-api-access-fzflc\" (UniqueName: \"kubernetes.io/projected/777dacd7-0cef-42aa-b932-f25c78e7fb40-kube-api-access-fzflc\") pod \"777dacd7-0cef-42aa-b932-f25c78e7fb40\" (UID: \"777dacd7-0cef-42aa-b932-f25c78e7fb40\") " Oct 31 13:32:01.294533 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.294506 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/777dacd7-0cef-42aa-b932-f25c78e7fb40-istio-data" pod "777dacd7-0cef-42aa-b932-f25c78e7fb40" (UID: "777dacd7-0cef-42aa-b932-f25c78e7fb40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.295957 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.294861 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31d9443c-5638-44ee-af89-e78960decfab-workload-socket" pod "31d9443c-5638-44ee-af89-e78960decfab" (UID: "31d9443c-5638-44ee-af89-e78960decfab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.295957 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.295223 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31d9443c-5638-44ee-af89-e78960decfab-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.295957 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.295249 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/777dacd7-0cef-42aa-b932-f25c78e7fb40-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.295957 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.295265 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/777dacd7-0cef-42aa-b932-f25c78e7fb40-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.295957 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.295279 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/777dacd7-0cef-42aa-b932-f25c78e7fb40-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.295957 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.295293 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31d9443c-5638-44ee-af89-e78960decfab-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.295957 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.295308 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d9443c-5638-44ee-af89-e78960decfab-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.295957 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.295320 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/777dacd7-0cef-42aa-b932-f25c78e7fb40-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.295957 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.295694 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/777dacd7-0cef-42aa-b932-f25c78e7fb40-istiod-ca-cert" pod "777dacd7-0cef-42aa-b932-f25c78e7fb40" (UID: "777dacd7-0cef-42aa-b932-f25c78e7fb40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:32:01.296537 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.295998 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31d9443c-5638-44ee-af89-e78960decfab-credential-socket" pod "31d9443c-5638-44ee-af89-e78960decfab" (UID: "31d9443c-5638-44ee-af89-e78960decfab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.296537 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.296186 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/777dacd7-0cef-42aa-b932-f25c78e7fb40-workload-certs" pod "777dacd7-0cef-42aa-b932-f25c78e7fb40" (UID: "777dacd7-0cef-42aa-b932-f25c78e7fb40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.297424 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.297403 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31d9443c-5638-44ee-af89-e78960decfab-istiod-ca-cert" pod "31d9443c-5638-44ee-af89-e78960decfab" (UID: "31d9443c-5638-44ee-af89-e78960decfab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:32:01.298193 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.298145 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31d9443c-5638-44ee-af89-e78960decfab-kube-api-access-x4lw2" pod "31d9443c-5638-44ee-af89-e78960decfab" (UID: "31d9443c-5638-44ee-af89-e78960decfab"). InnerVolumeSpecName "kube-api-access-x4lw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:01.299969 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.299944 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/777dacd7-0cef-42aa-b932-f25c78e7fb40-istio-envoy" pod "777dacd7-0cef-42aa-b932-f25c78e7fb40" (UID: "777dacd7-0cef-42aa-b932-f25c78e7fb40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.299969 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.299956 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/777dacd7-0cef-42aa-b932-f25c78e7fb40-istio-token" pod "777dacd7-0cef-42aa-b932-f25c78e7fb40" (UID: "777dacd7-0cef-42aa-b932-f25c78e7fb40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:01.300244 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.300207 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31d9443c-5638-44ee-af89-e78960decfab-istio-envoy" pod "31d9443c-5638-44ee-af89-e78960decfab" (UID: "31d9443c-5638-44ee-af89-e78960decfab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.300349 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.300334 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31d9443c-5638-44ee-af89-e78960decfab-istio-token" pod "31d9443c-5638-44ee-af89-e78960decfab" (UID: "31d9443c-5638-44ee-af89-e78960decfab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:01.300822 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.300787 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31d9443c-5638-44ee-af89-e78960decfab-istio-podinfo" pod "31d9443c-5638-44ee-af89-e78960decfab" (UID: "31d9443c-5638-44ee-af89-e78960decfab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:32:01.301142 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.301107 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/777dacd7-0cef-42aa-b932-f25c78e7fb40-kube-api-access-fzflc" pod "777dacd7-0cef-42aa-b932-f25c78e7fb40" (UID: "777dacd7-0cef-42aa-b932-f25c78e7fb40"). InnerVolumeSpecName "kube-api-access-fzflc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:01.396918 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.396878 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/777dacd7-0cef-42aa-b932-f25c78e7fb40-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.396918 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.396916 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31d9443c-5638-44ee-af89-e78960decfab-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.396918 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.396925 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31d9443c-5638-44ee-af89-e78960decfab-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.397208 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.396936 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/777dacd7-0cef-42aa-b932-f25c78e7fb40-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.397208 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.396946 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31d9443c-5638-44ee-af89-e78960decfab-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.397208 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.396955 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4lw2\" (UniqueName: \"kubernetes.io/projected/31d9443c-5638-44ee-af89-e78960decfab-kube-api-access-x4lw2\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.397208 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.396964 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31d9443c-5638-44ee-af89-e78960decfab-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.397208 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.396973 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d9443c-5638-44ee-af89-e78960decfab-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.397208 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.396992 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/777dacd7-0cef-42aa-b932-f25c78e7fb40-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.397208 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.397001 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/777dacd7-0cef-42aa-b932-f25c78e7fb40-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.397208 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.397010 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzflc\" (UniqueName: \"kubernetes.io/projected/777dacd7-0cef-42aa-b932-f25c78e7fb40-kube-api-access-fzflc\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.568975 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.568940 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g" event={"ID":"109fb6e5-38ef-4030-9209-cda454c140ec","Type":"ContainerStarted","Data":"ccb6c5a3c4feddf4bbc9fe85326be19eea7846b188f31543b081678461e8346c"} Oct 31 13:32:01.568975 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.569022 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g" event={"ID":"109fb6e5-38ef-4030-9209-cda454c140ec","Type":"ContainerStarted","Data":"54452efb4994340b976e986882d1ba707c9295d71d65802f405407ba1bae49bc"} Oct 31 13:32:01.570560 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.570528 6841 generic.go:356] "Generic (PLEG): container finished" podID="777dacd7-0cef-42aa-b932-f25c78e7fb40" containerID="d818ed145097da59e8f3e72625ae399fddfaf70ef6f9b2c6a3aedef7011e61b6" exitCode=0 Oct 31 13:32:01.570711 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.570585 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" Oct 31 13:32:01.570711 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.570590 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" event={"ID":"777dacd7-0cef-42aa-b932-f25c78e7fb40","Type":"ContainerDied","Data":"d818ed145097da59e8f3e72625ae399fddfaf70ef6f9b2c6a3aedef7011e61b6"} Oct 31 13:32:01.570711 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.570692 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w" event={"ID":"777dacd7-0cef-42aa-b932-f25c78e7fb40","Type":"ContainerDied","Data":"4b17f3c5db0a87fe7345b62ef177f946cff70902604cb7ea2d7a9e9439274b81"} Oct 31 13:32:01.570888 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.570728 6841 scope.go:122] "RemoveContainer" containerID="d818ed145097da59e8f3e72625ae399fddfaf70ef6f9b2c6a3aedef7011e61b6" Oct 31 13:32:01.572196 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.572174 6841 generic.go:356] "Generic (PLEG): container finished" podID="31d9443c-5638-44ee-af89-e78960decfab" containerID="09ee9e5baf6944e6c7757bacf382ce9b36b9f45d6a43f4df7deca581cd3473ae" exitCode=0 Oct 31 13:32:01.572288 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.572214 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" event={"ID":"31d9443c-5638-44ee-af89-e78960decfab","Type":"ContainerDied","Data":"09ee9e5baf6944e6c7757bacf382ce9b36b9f45d6a43f4df7deca581cd3473ae"} Oct 31 13:32:01.572288 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.572230 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" Oct 31 13:32:01.572288 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.572251 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h" event={"ID":"31d9443c-5638-44ee-af89-e78960decfab","Type":"ContainerDied","Data":"6db4b41ff4e7979f1c90d3afa6d1a0605821b7d6c5607fd3dd389b541aabe057"} Oct 31 13:32:01.583473 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.583455 6841 scope.go:122] "RemoveContainer" containerID="d818ed145097da59e8f3e72625ae399fddfaf70ef6f9b2c6a3aedef7011e61b6" Oct 31 13:32:01.583898 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:32:01.583737 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d818ed145097da59e8f3e72625ae399fddfaf70ef6f9b2c6a3aedef7011e61b6\": container with ID starting with d818ed145097da59e8f3e72625ae399fddfaf70ef6f9b2c6a3aedef7011e61b6 not found: ID does not exist" containerID="d818ed145097da59e8f3e72625ae399fddfaf70ef6f9b2c6a3aedef7011e61b6" Oct 31 13:32:01.583898 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.583762 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d818ed145097da59e8f3e72625ae399fddfaf70ef6f9b2c6a3aedef7011e61b6"} err="failed to get container status \"d818ed145097da59e8f3e72625ae399fddfaf70ef6f9b2c6a3aedef7011e61b6\": rpc error: code = NotFound desc = could not find container \"d818ed145097da59e8f3e72625ae399fddfaf70ef6f9b2c6a3aedef7011e61b6\": container with ID starting with d818ed145097da59e8f3e72625ae399fddfaf70ef6f9b2c6a3aedef7011e61b6 not found: ID does not exist" Oct 31 13:32:01.583898 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.583778 6841 scope.go:122] "RemoveContainer" containerID="09ee9e5baf6944e6c7757bacf382ce9b36b9f45d6a43f4df7deca581cd3473ae" Oct 31 13:32:01.590477 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.590437 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g" podStartSLOduration=1.590423846 podStartE2EDuration="1.590423846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:32:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:32:01.588343037 +0000 UTC m=+3568.071465939" watchObservedRunningTime="2026-10-31 13:32:01.590423846 +0000 UTC m=+3568.073546748" Oct 31 13:32:01.597103 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.597080 6841 scope.go:122] "RemoveContainer" containerID="09ee9e5baf6944e6c7757bacf382ce9b36b9f45d6a43f4df7deca581cd3473ae" Oct 31 13:32:01.597381 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:32:01.597361 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09ee9e5baf6944e6c7757bacf382ce9b36b9f45d6a43f4df7deca581cd3473ae\": container with ID starting with 09ee9e5baf6944e6c7757bacf382ce9b36b9f45d6a43f4df7deca581cd3473ae not found: ID does not exist" containerID="09ee9e5baf6944e6c7757bacf382ce9b36b9f45d6a43f4df7deca581cd3473ae" Oct 31 13:32:01.597438 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.597388 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09ee9e5baf6944e6c7757bacf382ce9b36b9f45d6a43f4df7deca581cd3473ae"} err="failed to get container status \"09ee9e5baf6944e6c7757bacf382ce9b36b9f45d6a43f4df7deca581cd3473ae\": rpc error: code = NotFound desc = could not find container \"09ee9e5baf6944e6c7757bacf382ce9b36b9f45d6a43f4df7deca581cd3473ae\": container with ID starting with 09ee9e5baf6944e6c7757bacf382ce9b36b9f45d6a43f4df7deca581cd3473ae not found: ID does not exist" Oct 31 13:32:01.612165 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.612132 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w"] Oct 31 13:32:01.617331 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.617304 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-9h58w"] Oct 31 13:32:01.625901 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.625876 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h"] Oct 31 13:32:01.631609 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.631587 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-dk86h"] Oct 31 13:32:01.645924 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.645902 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31d9443c-5638-44ee-af89-e78960decfab" path="/var/lib/kubelet/pods/31d9443c-5638-44ee-af89-e78960decfab/volumes" Oct 31 13:32:01.646511 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.646495 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="777dacd7-0cef-42aa-b932-f25c78e7fb40" path="/var/lib/kubelet/pods/777dacd7-0cef-42aa-b932-f25c78e7fb40/volumes" Oct 31 13:32:01.737628 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:01.737590 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g" Oct 31 13:32:02.133609 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:02.133576 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:03.081234 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:03.081191 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:03.678933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:03.678900 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:04.413424 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:04.413394 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:04.638019 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:04.637964 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:04.842857 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:04.842821 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:05.240426 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:05.240381 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:06.015771 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:06.015729 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:06.560347 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:06.560311 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d"] Oct 31 13:32:06.560574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:06.560550 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d" podUID="c26c62d4-873b-422e-8740-457b84206ac3" containerName="maas-api" containerID="cri-o://04656f489e162cf216c0feb9c892fd7f785651121b34696a98baf3921771767f" gracePeriod=30 Oct 31 13:32:06.731101 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:06.731067 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:07.181765 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.181734 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d" Oct 31 13:32:07.190040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.189910 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:07.266665 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.266626 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c26c62d4-873b-422e-8740-457b84206ac3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c26c62d4-873b-422e-8740-457b84206ac3-maas-api-tls\") pod \"c26c62d4-873b-422e-8740-457b84206ac3\" (UID: \"c26c62d4-873b-422e-8740-457b84206ac3\") " Oct 31 13:32:07.266846 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.266707 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c26c62d4-873b-422e-8740-457b84206ac3-kube-api-access-swzzb\" (UniqueName: \"kubernetes.io/projected/c26c62d4-873b-422e-8740-457b84206ac3-kube-api-access-swzzb\") pod \"c26c62d4-873b-422e-8740-457b84206ac3\" (UID: \"c26c62d4-873b-422e-8740-457b84206ac3\") " Oct 31 13:32:07.266846 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.266835 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c26c62d4-873b-422e-8740-457b84206ac3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c26c62d4-873b-422e-8740-457b84206ac3-metrics-certs\") pod \"c26c62d4-873b-422e-8740-457b84206ac3\" (UID: \"c26c62d4-873b-422e-8740-457b84206ac3\") " Oct 31 13:32:07.266943 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.266878 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c26c62d4-873b-422e-8740-457b84206ac3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c26c62d4-873b-422e-8740-457b84206ac3-openshift-service-ca\") pod \"c26c62d4-873b-422e-8740-457b84206ac3\" (UID: \"c26c62d4-873b-422e-8740-457b84206ac3\") " Oct 31 13:32:07.267339 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.267309 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c26c62d4-873b-422e-8740-457b84206ac3-openshift-service-ca" pod "c26c62d4-873b-422e-8740-457b84206ac3" (UID: "c26c62d4-873b-422e-8740-457b84206ac3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:32:07.267672 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.267646 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c26c62d4-873b-422e-8740-457b84206ac3-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:07.269149 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.269121 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c26c62d4-873b-422e-8740-457b84206ac3-kube-api-access-swzzb" pod "c26c62d4-873b-422e-8740-457b84206ac3" (UID: "c26c62d4-873b-422e-8740-457b84206ac3"). InnerVolumeSpecName "kube-api-access-swzzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:07.269317 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.269299 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c26c62d4-873b-422e-8740-457b84206ac3-maas-api-tls" pod "c26c62d4-873b-422e-8740-457b84206ac3" (UID: "c26c62d4-873b-422e-8740-457b84206ac3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:32:07.270332 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.270309 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c26c62d4-873b-422e-8740-457b84206ac3-metrics-certs" pod "c26c62d4-873b-422e-8740-457b84206ac3" (UID: "c26c62d4-873b-422e-8740-457b84206ac3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:32:07.370088 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.369928 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c26c62d4-873b-422e-8740-457b84206ac3-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:07.370088 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.369961 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c26c62d4-873b-422e-8740-457b84206ac3-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:07.370088 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.369972 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swzzb\" (UniqueName: \"kubernetes.io/projected/c26c62d4-873b-422e-8740-457b84206ac3-kube-api-access-swzzb\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:07.639280 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.639192 6841 generic.go:356] "Generic (PLEG): container finished" podID="c26c62d4-873b-422e-8740-457b84206ac3" containerID="04656f489e162cf216c0feb9c892fd7f785651121b34696a98baf3921771767f" exitCode=0 Oct 31 13:32:07.639423 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.639295 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d" Oct 31 13:32:07.649258 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.649227 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d" event={"ID":"c26c62d4-873b-422e-8740-457b84206ac3","Type":"ContainerDied","Data":"04656f489e162cf216c0feb9c892fd7f785651121b34696a98baf3921771767f"} Oct 31 13:32:07.649401 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.649279 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d" event={"ID":"c26c62d4-873b-422e-8740-457b84206ac3","Type":"ContainerDied","Data":"ce160b1fbdfa5c797017984c876bb5cdb1d509e6f041fd4594c7ab66c1d5b315"} Oct 31 13:32:07.649401 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.649308 6841 scope.go:122] "RemoveContainer" containerID="04656f489e162cf216c0feb9c892fd7f785651121b34696a98baf3921771767f" Oct 31 13:32:07.664700 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.664677 6841 scope.go:122] "RemoveContainer" containerID="04656f489e162cf216c0feb9c892fd7f785651121b34696a98baf3921771767f" Oct 31 13:32:07.665181 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:32:07.665152 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04656f489e162cf216c0feb9c892fd7f785651121b34696a98baf3921771767f\": container with ID starting with 04656f489e162cf216c0feb9c892fd7f785651121b34696a98baf3921771767f not found: ID does not exist" containerID="04656f489e162cf216c0feb9c892fd7f785651121b34696a98baf3921771767f" Oct 31 13:32:07.665295 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.665189 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04656f489e162cf216c0feb9c892fd7f785651121b34696a98baf3921771767f"} err="failed to get container status \"04656f489e162cf216c0feb9c892fd7f785651121b34696a98baf3921771767f\": rpc error: code = NotFound desc = could not find container \"04656f489e162cf216c0feb9c892fd7f785651121b34696a98baf3921771767f\": container with ID starting with 04656f489e162cf216c0feb9c892fd7f785651121b34696a98baf3921771767f not found: ID does not exist" Oct 31 13:32:07.673114 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.673091 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d"] Oct 31 13:32:07.687242 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.687218 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4023ad-dbb58b646-bkn5d"] Oct 31 13:32:07.892012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.891902 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x"] Oct 31 13:32:07.892740 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.892723 6841 cpu_manager.go:414] "containerMap: removing container" podUID="c26c62d4-873b-422e-8740-457b84206ac3" containerName="maas-api" Oct 31 13:32:07.892798 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.892741 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c26c62d4-873b-422e-8740-457b84206ac3" containerName="maas-api" Oct 31 13:32:07.892798 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.892756 6841 cpu_manager.go:414] "containerMap: removing container" podUID="31d9443c-5638-44ee-af89-e78960decfab" containerName="istio-proxy" Oct 31 13:32:07.892798 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.892761 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31d9443c-5638-44ee-af89-e78960decfab" containerName="istio-proxy" Oct 31 13:32:07.892798 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.892780 6841 cpu_manager.go:414] "containerMap: removing container" podUID="777dacd7-0cef-42aa-b932-f25c78e7fb40" containerName="istio-proxy" Oct 31 13:32:07.892798 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.892786 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="777dacd7-0cef-42aa-b932-f25c78e7fb40" containerName="istio-proxy" Oct 31 13:32:07.893020 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.892932 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="777dacd7-0cef-42aa-b932-f25c78e7fb40" containerName="istio-proxy" Oct 31 13:32:07.893020 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.892955 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="c26c62d4-873b-422e-8740-457b84206ac3" containerName="maas-api" Oct 31 13:32:07.893020 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.892962 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="31d9443c-5638-44ee-af89-e78960decfab" containerName="istio-proxy" Oct 31 13:32:07.898301 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.898281 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x" Oct 31 13:32:07.917997 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.917966 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3cddb3\"/\"openshift-service-ca.crt\"" Oct 31 13:32:07.918266 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.918237 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3cddb3\"/\"kube-root-ca.crt\"" Oct 31 13:32:07.920413 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.920384 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3cddb3\"/\"default-dockercfg-wtslb\"" Oct 31 13:32:07.921077 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.921053 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3cddb3\"/\"test-model-3cddb3-kserve-self-signed-certs\"" Oct 31 13:32:07.924192 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:07.924171 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x"] Oct 31 13:32:08.083284 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:08.083249 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-tmp-dir\") pod \"test-model-3cddb3-kserve-75bf994797-jlw6x\" (UID: \"1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939\") " pod="ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x" Oct 31 13:32:08.083476 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:08.083318 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-model-cache\") pod \"test-model-3cddb3-kserve-75bf994797-jlw6x\" (UID: \"1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939\") " pod="ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x" Oct 31 13:32:08.083476 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:08.083400 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-home\") pod \"test-model-3cddb3-kserve-75bf994797-jlw6x\" (UID: \"1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939\") " pod="ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x" Oct 31 13:32:08.083476 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:08.083438 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-tls-certs\") pod \"test-model-3cddb3-kserve-75bf994797-jlw6x\" (UID: \"1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939\") " pod="ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x" Oct 31 13:32:08.083707 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:08.083686 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-dshm\") pod \"test-model-3cddb3-kserve-75bf994797-jlw6x\" (UID: \"1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939\") " pod="ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x" Oct 31 13:32:08.083853 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:08.083835 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnh5h\" (UniqueName: \"kubernetes.io/projected/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-kube-api-access-rnh5h\") pod \"test-model-3cddb3-kserve-75bf994797-jlw6x\" (UID: \"1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939\") " pod="ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x" Oct 31 13:32:08.160153 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:08.159962 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:08.186262 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:08.186222 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-dshm\") pod \"test-model-3cddb3-kserve-75bf994797-jlw6x\" (UID: \"1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939\") " pod="ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x" Oct 31 13:32:08.208008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:08.196293 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnh5h\" (UniqueName: \"kubernetes.io/projected/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-kube-api-access-rnh5h\") pod \"test-model-3cddb3-kserve-75bf994797-jlw6x\" (UID: \"1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939\") " pod="ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x" Oct 31 13:32:08.208636 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:08.208606 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-tmp-dir\") pod \"test-model-3cddb3-kserve-75bf994797-jlw6x\" (UID: \"1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939\") " pod="ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x" Oct 31 13:32:08.220778 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:08.220746 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-model-cache\") pod \"test-model-3cddb3-kserve-75bf994797-jlw6x\" (UID: \"1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939\") " pod="ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x" Oct 31 13:32:08.221075 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:08.221053 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-home\") pod \"test-model-3cddb3-kserve-75bf994797-jlw6x\" (UID: \"1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939\") " pod="ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x" Oct 31 13:32:08.221251 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:08.221233 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-tls-certs\") pod \"test-model-3cddb3-kserve-75bf994797-jlw6x\" (UID: \"1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939\") " pod="ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x" Oct 31 13:32:08.222488 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:08.222415 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-home\") pod \"test-model-3cddb3-kserve-75bf994797-jlw6x\" (UID: \"1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939\") " pod="ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x" Oct 31 13:32:08.222488 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:08.209199 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-tmp-dir\") pod \"test-model-3cddb3-kserve-75bf994797-jlw6x\" (UID: \"1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939\") " pod="ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x" Oct 31 13:32:08.239971 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:08.236865 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnh5h\" (UniqueName: \"kubernetes.io/projected/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-kube-api-access-rnh5h\") pod \"test-model-3cddb3-kserve-75bf994797-jlw6x\" (UID: \"1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939\") " pod="ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x" Oct 31 13:32:08.240469 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:08.240447 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-model-cache\") pod \"test-model-3cddb3-kserve-75bf994797-jlw6x\" (UID: \"1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939\") " pod="ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x" Oct 31 13:32:08.258001 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:08.251171 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-dshm\") pod \"test-model-3cddb3-kserve-75bf994797-jlw6x\" (UID: \"1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939\") " pod="ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x" Oct 31 13:32:08.264003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:08.263956 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-tls-certs\") pod \"test-model-3cddb3-kserve-75bf994797-jlw6x\" (UID: \"1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939\") " pod="ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x" Oct 31 13:32:08.377927 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:08.377885 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:08.511978 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:08.511946 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x" Oct 31 13:32:08.805909 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:32:08.805869 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f1f43cc_5fd8_4f7b_8bcc_67b32f2ac939.slice/crio-1d0b65542c8e38b7e798be4524ac7534ede3cae09ee3378703d1107f8681a49a WatchSource:0}: Error finding container 1d0b65542c8e38b7e798be4524ac7534ede3cae09ee3378703d1107f8681a49a: Status 404 returned error can't find the container with id 1d0b65542c8e38b7e798be4524ac7534ede3cae09ee3378703d1107f8681a49a Oct 31 13:32:08.807294 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:08.807262 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x"] Oct 31 13:32:09.239477 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:09.239439 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:09.645217 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:09.645128 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c26c62d4-873b-422e-8740-457b84206ac3" path="/var/lib/kubelet/pods/c26c62d4-873b-422e-8740-457b84206ac3/volumes" Oct 31 13:32:09.659947 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:09.659909 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x" event={"ID":"1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939","Type":"ContainerStarted","Data":"2560e79b2ef03481f4cc62ef0caaff2aa107ae6cad863161993219dccab661c7"} Oct 31 13:32:09.659947 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:09.659951 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x" event={"ID":"1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939","Type":"ContainerStarted","Data":"1d0b65542c8e38b7e798be4524ac7534ede3cae09ee3378703d1107f8681a49a"} Oct 31 13:32:09.682855 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:09.682796 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x" podStartSLOduration=2.682784124 podStartE2EDuration="2.682784124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:32:09.678446688 +0000 UTC m=+3576.161569592" watchObservedRunningTime="2026-10-31 13:32:09.682784124 +0000 UTC m=+3576.165907024" Oct 31 13:32:10.056157 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:10.056111 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:10.513020 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:10.512973 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x" Oct 31 13:32:10.734816 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:10.734783 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g" Oct 31 13:32:10.747734 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:10.747707 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g" Oct 31 13:32:11.035500 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:11.035457 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:11.308533 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:11.308445 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:11.527026 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:11.526958 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:11.687267 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:11.687189 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g" Oct 31 13:32:11.885171 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:11.885140 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:12.201027 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:12.200954 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:12.496946 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:12.492509 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:13.156365 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:13.156329 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:13.398148 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:13.398114 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:14.389730 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:14.389688 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:15.205307 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:15.205265 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:16.005547 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:16.005504 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:16.706518 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:16.706486 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:17.746967 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:17.746930 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:18.512065 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:18.512034 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x" Oct 31 13:32:18.524437 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:18.524408 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x" Oct 31 13:32:18.564604 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:18.564568 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:18.789519 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:18.789424 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:18.973746 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:18.973496 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x" Oct 31 13:32:19.982333 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:19.982297 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:20.037536 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:32:20.037501 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:32:21.050674 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:21.050642 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:21.466049 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:21.466011 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:21.765935 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:21.765852 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd"] Oct 31 13:32:21.766115 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:21.766066 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" podUID="d90d4670-3be8-4966-b1b5-cbe007b9a8ff" containerName="istio-proxy" containerID="cri-o://dc45023f3a87b31fa665dc43c1b003b845817952e88faa5d1c1701fdd259ec7e" gracePeriod=30 Oct 31 13:32:21.896596 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:21.896551 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:22.907455 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:22.904223 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:23.749375 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:23.749328 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:24.630897 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:24.630862 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:25.647128 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:25.647080 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:25.878588 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:25.878553 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:26.659323 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:26.659288 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:26.953352 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:26.953312 6841 generic.go:356] "Generic (PLEG): container finished" podID="d90d4670-3be8-4966-b1b5-cbe007b9a8ff" containerID="dc45023f3a87b31fa665dc43c1b003b845817952e88faa5d1c1701fdd259ec7e" exitCode=0 Oct 31 13:32:26.953525 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:26.953380 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" event={"ID":"d90d4670-3be8-4966-b1b5-cbe007b9a8ff","Type":"ContainerDied","Data":"dc45023f3a87b31fa665dc43c1b003b845817952e88faa5d1c1701fdd259ec7e"} Oct 31 13:32:27.110079 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:27.110051 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" Oct 31 13:32:27.210842 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:27.210750 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-istiod-ca-cert\") pod \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\" (UID: \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\") " Oct 31 13:32:27.211037 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:27.210851 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-istio-data\") pod \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\" (UID: \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\") " Oct 31 13:32:27.211037 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:27.210891 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-istio-token\" (UniqueName: \"kubernetes.io/projected/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-istio-token\") pod \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\" (UID: \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\") " Oct 31 13:32:27.211143 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:27.211063 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-credential-socket\") pod \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\" (UID: \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\") " Oct 31 13:32:27.211200 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:27.211153 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-istio-podinfo\") pod \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\" (UID: \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\") " Oct 31 13:32:27.211278 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:27.211255 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-kube-api-access-rbn6s\" (UniqueName: \"kubernetes.io/projected/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-kube-api-access-rbn6s\") pod \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\" (UID: \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\") " Oct 31 13:32:27.211389 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:27.211354 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-workload-certs\") pod \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\" (UID: \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\") " Oct 31 13:32:27.211450 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:27.211387 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-workload-socket\") pod \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\" (UID: \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\") " Oct 31 13:32:27.211450 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:27.211443 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-istio-envoy\") pod \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\" (UID: \"d90d4670-3be8-4966-b1b5-cbe007b9a8ff\") " Oct 31 13:32:27.212512 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:27.212141 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-workload-certs" pod "d90d4670-3be8-4966-b1b5-cbe007b9a8ff" (UID: "d90d4670-3be8-4966-b1b5-cbe007b9a8ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:27.212512 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:27.212303 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:27.212956 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:27.212765 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-workload-socket" pod "d90d4670-3be8-4966-b1b5-cbe007b9a8ff" (UID: "d90d4670-3be8-4966-b1b5-cbe007b9a8ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:27.214744 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:27.213839 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-credential-socket" pod "d90d4670-3be8-4966-b1b5-cbe007b9a8ff" (UID: "d90d4670-3be8-4966-b1b5-cbe007b9a8ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:27.214744 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:27.213930 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-istio-podinfo" pod "d90d4670-3be8-4966-b1b5-cbe007b9a8ff" (UID: "d90d4670-3be8-4966-b1b5-cbe007b9a8ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:32:27.215228 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:27.215205 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-istio-data" pod "d90d4670-3be8-4966-b1b5-cbe007b9a8ff" (UID: "d90d4670-3be8-4966-b1b5-cbe007b9a8ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:27.215535 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:27.215491 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-istiod-ca-cert" pod "d90d4670-3be8-4966-b1b5-cbe007b9a8ff" (UID: "d90d4670-3be8-4966-b1b5-cbe007b9a8ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:32:27.217472 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:27.217434 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-kube-api-access-rbn6s" pod "d90d4670-3be8-4966-b1b5-cbe007b9a8ff" (UID: "d90d4670-3be8-4966-b1b5-cbe007b9a8ff"). InnerVolumeSpecName "kube-api-access-rbn6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:27.219940 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:27.219905 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-istio-token" pod "d90d4670-3be8-4966-b1b5-cbe007b9a8ff" (UID: "d90d4670-3be8-4966-b1b5-cbe007b9a8ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:27.221000 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:27.220824 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-istio-envoy" pod "d90d4670-3be8-4966-b1b5-cbe007b9a8ff" (UID: "d90d4670-3be8-4966-b1b5-cbe007b9a8ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:27.314263 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:27.314228 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:27.314263 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:27.314260 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:27.314263 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:27.314270 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbn6s\" (UniqueName: \"kubernetes.io/projected/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-kube-api-access-rbn6s\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:27.314511 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:27.314280 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:27.314511 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:27.314288 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:27.314511 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:27.314298 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:27.314511 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:27.314306 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:27.314511 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:27.314317 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d90d4670-3be8-4966-b1b5-cbe007b9a8ff-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:27.588336 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:27.588228 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:27.964382 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:27.964348 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" Oct 31 13:32:27.964934 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:27.964383 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd" event={"ID":"d90d4670-3be8-4966-b1b5-cbe007b9a8ff","Type":"ContainerDied","Data":"7e3692154f93d7159df81e17079d98680aa4b40d3de2fb4c61e3172486535e88"} Oct 31 13:32:27.964934 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:27.964442 6841 scope.go:122] "RemoveContainer" containerID="dc45023f3a87b31fa665dc43c1b003b845817952e88faa5d1c1701fdd259ec7e" Oct 31 13:32:27.991693 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:27.991663 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd"] Oct 31 13:32:28.000078 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:28.000052 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-s84vd"] Oct 31 13:32:28.580915 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:28.580871 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:28.796997 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:28.796951 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:29.284127 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:29.284083 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:29.645404 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:29.645328 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d90d4670-3be8-4966-b1b5-cbe007b9a8ff" path="/var/lib/kubelet/pods/d90d4670-3be8-4966-b1b5-cbe007b9a8ff/volumes" Oct 31 13:32:30.128693 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:30.128662 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:30.586263 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:30.585175 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:31.308100 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:31.308065 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:32.037094 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:32.037054 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:32.963271 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:32.963237 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:33.742503 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:33.742469 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:34.554834 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:34.554796 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:35.411609 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:35.411556 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:36.161784 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:36.161734 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:37.088424 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:37.088387 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:37.799141 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:37.799106 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:38.527491 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:38.527451 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:39.312516 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:39.312485 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:39.899050 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:39.899000 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:40.112235 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:40.112191 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:40.374040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:40.373964 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:40.578458 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:40.578418 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:40.697638 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:40.697609 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp"] Oct 31 13:32:40.697876 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:40.697852 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" podUID="4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a" containerName="istio-proxy" containerID="cri-o://051bfea054619709544afe81b0c0a83a4440abdd9289e0d9b9b0bdaadb675215" gracePeriod=30 Oct 31 13:32:40.871589 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:40.871553 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:41.218811 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:41.218542 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:41.742977 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:41.742942 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:42.108459 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:42.108372 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:42.311649 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:42.311616 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:42.563784 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:42.563747 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:42.959421 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:42.959389 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:43.934402 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:43.934367 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:44.578570 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:44.578535 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:44.881872 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:44.881700 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:45.778589 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:45.778521 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:45.998252 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:45.998186 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:32:46.023730 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.023690 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-workload-socket\") pod \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\" (UID: \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\") " Oct 31 13:32:46.023917 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.023781 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-istio-podinfo\") pod \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\" (UID: \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\") " Oct 31 13:32:46.023917 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.023874 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-istiod-ca-cert\") pod \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\" (UID: \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\") " Oct 31 13:32:46.024117 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.023935 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-workload-certs\") pod \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\" (UID: \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\") " Oct 31 13:32:46.024117 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.024093 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-workload-socket" pod "4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a" (UID: "4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:46.024235 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.024135 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-istio-envoy\") pod \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\" (UID: \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\") " Oct 31 13:32:46.024235 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.024200 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-kube-api-access-s8b88\" (UniqueName: \"kubernetes.io/projected/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-kube-api-access-s8b88\") pod \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\" (UID: \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\") " Oct 31 13:32:46.024321 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.024277 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-istio-data\") pod \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\" (UID: \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\") " Oct 31 13:32:46.024321 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.024277 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-workload-certs" pod "4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a" (UID: "4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:46.024431 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.024342 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-istiod-ca-cert" pod "4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a" (UID: "4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:32:46.024431 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.024353 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-istio-token\" (UniqueName: \"kubernetes.io/projected/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-istio-token\") pod \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\" (UID: \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\") " Oct 31 13:32:46.024532 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.024438 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-credential-socket\") pod \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\" (UID: \"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a\") " Oct 31 13:32:46.025531 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.024631 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-istio-data" pod "4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a" (UID: "4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:46.025531 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.024859 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-credential-socket" pod "4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a" (UID: "4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:46.025531 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.025409 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:46.025531 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.025432 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:46.025531 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.025445 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:46.025531 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.025459 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:46.025531 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.025472 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:46.027248 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.027222 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-istio-podinfo" pod "4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a" (UID: "4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:32:46.027483 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.027449 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-istio-envoy" pod "4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a" (UID: "4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:46.028058 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.028032 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-kube-api-access-s8b88" pod "4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a" (UID: "4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a"). InnerVolumeSpecName "kube-api-access-s8b88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:46.028745 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.028722 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-istio-token" pod "4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a" (UID: "4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:46.127546 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.127518 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:46.127546 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.127543 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:46.127756 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.127556 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:46.127756 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.127566 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8b88\" (UniqueName: \"kubernetes.io/projected/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a-kube-api-access-s8b88\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:46.154841 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.154810 6841 generic.go:356] "Generic (PLEG): container finished" podID="4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a" containerID="051bfea054619709544afe81b0c0a83a4440abdd9289e0d9b9b0bdaadb675215" exitCode=0 Oct 31 13:32:46.155051 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.154860 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" Oct 31 13:32:46.155051 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.154886 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" event={"ID":"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a","Type":"ContainerDied","Data":"051bfea054619709544afe81b0c0a83a4440abdd9289e0d9b9b0bdaadb675215"} Oct 31 13:32:46.155051 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.154947 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp" event={"ID":"4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a","Type":"ContainerDied","Data":"402c95be22d7ca72ec239a81dc805e082c162f0c8e0f85ccbcf72a196f1e1c78"} Oct 31 13:32:46.155051 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.154991 6841 scope.go:122] "RemoveContainer" containerID="051bfea054619709544afe81b0c0a83a4440abdd9289e0d9b9b0bdaadb675215" Oct 31 13:32:46.168189 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.168164 6841 scope.go:122] "RemoveContainer" containerID="63c97c8347b67d1012a4027d06e9e76aebdbe0a24a2dc927092c2205621636bb" Oct 31 13:32:46.178793 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.178772 6841 scope.go:122] "RemoveContainer" containerID="051bfea054619709544afe81b0c0a83a4440abdd9289e0d9b9b0bdaadb675215" Oct 31 13:32:46.179118 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:32:46.179094 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"051bfea054619709544afe81b0c0a83a4440abdd9289e0d9b9b0bdaadb675215\": container with ID starting with 051bfea054619709544afe81b0c0a83a4440abdd9289e0d9b9b0bdaadb675215 not found: ID does not exist" containerID="051bfea054619709544afe81b0c0a83a4440abdd9289e0d9b9b0bdaadb675215" Oct 31 13:32:46.179218 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.179160 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"051bfea054619709544afe81b0c0a83a4440abdd9289e0d9b9b0bdaadb675215"} err="failed to get container status \"051bfea054619709544afe81b0c0a83a4440abdd9289e0d9b9b0bdaadb675215\": rpc error: code = NotFound desc = could not find container \"051bfea054619709544afe81b0c0a83a4440abdd9289e0d9b9b0bdaadb675215\": container with ID starting with 051bfea054619709544afe81b0c0a83a4440abdd9289e0d9b9b0bdaadb675215 not found: ID does not exist" Oct 31 13:32:46.179218 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.179185 6841 scope.go:122] "RemoveContainer" containerID="63c97c8347b67d1012a4027d06e9e76aebdbe0a24a2dc927092c2205621636bb" Oct 31 13:32:46.179559 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:32:46.179531 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63c97c8347b67d1012a4027d06e9e76aebdbe0a24a2dc927092c2205621636bb\": container with ID starting with 63c97c8347b67d1012a4027d06e9e76aebdbe0a24a2dc927092c2205621636bb not found: ID does not exist" containerID="63c97c8347b67d1012a4027d06e9e76aebdbe0a24a2dc927092c2205621636bb" Oct 31 13:32:46.179804 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.179763 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63c97c8347b67d1012a4027d06e9e76aebdbe0a24a2dc927092c2205621636bb"} err="failed to get container status \"63c97c8347b67d1012a4027d06e9e76aebdbe0a24a2dc927092c2205621636bb\": rpc error: code = NotFound desc = could not find container \"63c97c8347b67d1012a4027d06e9e76aebdbe0a24a2dc927092c2205621636bb\": container with ID starting with 63c97c8347b67d1012a4027d06e9e76aebdbe0a24a2dc927092c2205621636bb not found: ID does not exist" Oct 31 13:32:46.185132 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.185112 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp"] Oct 31 13:32:46.192294 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.192274 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-384dbe-openshift-default-5b58f4999c-7pjsp"] Oct 31 13:32:46.593378 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:46.593334 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:47.394115 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:47.394075 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:47.644732 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:47.644658 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a" path="/var/lib/kubelet/pods/4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a/volumes" Oct 31 13:32:48.226739 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:48.226693 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:49.016711 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:49.016665 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:49.780753 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:49.780718 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:50.036754 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:32:50.036675 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:32:50.620755 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:50.620661 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:51.520855 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:51.520820 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:52.515034 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:52.514996 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:53.533832 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:53.533770 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:54.534591 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:54.534552 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:55.400008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:55.398712 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:55.765797 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:55.765602 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:56.018411 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:56.018326 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:56.463007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:56.460437 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:56.693036 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:56.692096 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7"] Oct 31 13:32:56.714434 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:56.714214 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:56.934007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:56.933542 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:57.331457 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:57.331419 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" podUID="c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f" containerName="istio-proxy" containerID="cri-o://ffebabf8fa00e2f3db0f57152935bac2b6c85f742bcb5638725aa7bb053c6f0b" gracePeriod=30 Oct 31 13:32:57.620547 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:57.620465 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:58.069330 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:58.069293 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:58.395539 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:58.395444 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:58.632658 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:58.632577 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:58.857796 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:58.857749 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:32:59.740006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:32:59.736683 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:00.058951 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:00.058864 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:00.932976 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:00.932918 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:01.719525 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:01.719486 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:02.476460 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:02.476420 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:02.663522 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:02.663492 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:33:02.705449 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:02.705414 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-istio-token\" (UniqueName: \"kubernetes.io/projected/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-istio-token\") pod \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\" (UID: \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\") " Oct 31 13:33:02.705640 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:02.705493 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-credential-socket\") pod \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\" (UID: \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\") " Oct 31 13:33:02.705640 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:02.705538 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-workload-certs\") pod \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\" (UID: \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\") " Oct 31 13:33:02.705640 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:02.705580 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-istio-envoy\") pod \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\" (UID: \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\") " Oct 31 13:33:02.705826 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:02.705673 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-istiod-ca-cert\") pod \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\" (UID: \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\") " Oct 31 13:33:02.705826 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:02.705762 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-istio-podinfo\") pod \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\" (UID: \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\") " Oct 31 13:33:02.705826 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:02.705796 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-istio-data\") pod \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\" (UID: \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\") " Oct 31 13:33:02.706061 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:02.705846 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-credential-socket" pod "c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f" (UID: "c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:33:02.706061 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:02.705890 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-workload-certs" pod "c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f" (UID: "c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:33:02.706061 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:02.705920 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-kube-api-access-vmhmr\" (UniqueName: \"kubernetes.io/projected/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-kube-api-access-vmhmr\") pod \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\" (UID: \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\") " Oct 31 13:33:02.706243 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:02.706106 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-istio-data" pod "c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f" (UID: "c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:33:02.706243 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:02.706158 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-workload-socket\") pod \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\" (UID: \"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f\") " Oct 31 13:33:02.706568 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:02.706534 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-istiod-ca-cert" pod "c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f" (UID: "c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:33:02.706708 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:02.706590 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-workload-socket" pod "c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f" (UID: "c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:33:02.706916 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:02.706897 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:33:02.706916 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:02.706918 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:33:02.707138 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:02.706927 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:33:02.707138 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:02.706936 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:33:02.707138 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:02.706946 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:33:02.708123 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:02.708101 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-istio-podinfo" pod "c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f" (UID: "c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:33:02.708270 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:02.708253 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-istio-envoy" pod "c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f" (UID: "c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:33:02.708419 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:02.708390 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-istio-token" pod "c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f" (UID: "c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:33:02.708511 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:02.708420 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-kube-api-access-vmhmr" pod "c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f" (UID: "c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f"). InnerVolumeSpecName "kube-api-access-vmhmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:33:02.808050 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:02.808019 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:33:02.808050 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:02.808050 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmhmr\" (UniqueName: \"kubernetes.io/projected/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-kube-api-access-vmhmr\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:33:02.808242 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:02.808061 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:33:02.808242 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:02.808071 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:33:03.330179 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:03.330112 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:03.388429 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:03.388397 6841 generic.go:356] "Generic (PLEG): container finished" podID="c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f" containerID="ffebabf8fa00e2f3db0f57152935bac2b6c85f742bcb5638725aa7bb053c6f0b" exitCode=0 Oct 31 13:33:03.388594 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:03.388449 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" Oct 31 13:33:03.388594 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:03.388478 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" event={"ID":"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f","Type":"ContainerDied","Data":"ffebabf8fa00e2f3db0f57152935bac2b6c85f742bcb5638725aa7bb053c6f0b"} Oct 31 13:33:03.388594 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:03.388536 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7" event={"ID":"c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f","Type":"ContainerDied","Data":"ecc9af2001de9e8cb8063d6e950430174edde5a65e154c28f82d9ab4897ec86a"} Oct 31 13:33:03.388594 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:03.388567 6841 scope.go:122] "RemoveContainer" containerID="ffebabf8fa00e2f3db0f57152935bac2b6c85f742bcb5638725aa7bb053c6f0b" Oct 31 13:33:03.390966 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:03.390940 6841 generic.go:356] "Generic (PLEG): container finished" podID="c7a4d6d4-06ba-4385-94cc-66f96e07232a" containerID="116ceac509139ce93124e720a1f41afb0822806e9fb44e48bb37f30562e8232a" exitCode=0 Oct 31 13:33:03.391135 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:03.391038 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" event={"ID":"c7a4d6d4-06ba-4385-94cc-66f96e07232a","Type":"ContainerDied","Data":"116ceac509139ce93124e720a1f41afb0822806e9fb44e48bb37f30562e8232a"} Oct 31 13:33:03.391594 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:03.391575 6841 scope.go:122] "RemoveContainer" containerID="116ceac509139ce93124e720a1f41afb0822806e9fb44e48bb37f30562e8232a" Oct 31 13:33:03.393926 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:03.393890 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:33:03.394075 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:03.394000 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:33:03.394075 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:03.394055 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:33:03.404824 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:03.404802 6841 scope.go:122] "RemoveContainer" containerID="79b46d3cdeee3c8947262d926fcde8e152aabb749f88e4d1d46db61ea61822a5" Oct 31 13:33:03.418346 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:03.418321 6841 scope.go:122] "RemoveContainer" containerID="ffebabf8fa00e2f3db0f57152935bac2b6c85f742bcb5638725aa7bb053c6f0b" Oct 31 13:33:03.418635 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:33:03.418613 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffebabf8fa00e2f3db0f57152935bac2b6c85f742bcb5638725aa7bb053c6f0b\": container with ID starting with ffebabf8fa00e2f3db0f57152935bac2b6c85f742bcb5638725aa7bb053c6f0b not found: ID does not exist" containerID="ffebabf8fa00e2f3db0f57152935bac2b6c85f742bcb5638725aa7bb053c6f0b" Oct 31 13:33:03.418736 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:03.418639 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffebabf8fa00e2f3db0f57152935bac2b6c85f742bcb5638725aa7bb053c6f0b"} err="failed to get container status \"ffebabf8fa00e2f3db0f57152935bac2b6c85f742bcb5638725aa7bb053c6f0b\": rpc error: code = NotFound desc = could not find container \"ffebabf8fa00e2f3db0f57152935bac2b6c85f742bcb5638725aa7bb053c6f0b\": container with ID starting with ffebabf8fa00e2f3db0f57152935bac2b6c85f742bcb5638725aa7bb053c6f0b not found: ID does not exist" Oct 31 13:33:03.418736 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:03.418658 6841 scope.go:122] "RemoveContainer" containerID="79b46d3cdeee3c8947262d926fcde8e152aabb749f88e4d1d46db61ea61822a5" Oct 31 13:33:03.419015 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:33:03.418960 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79b46d3cdeee3c8947262d926fcde8e152aabb749f88e4d1d46db61ea61822a5\": container with ID starting with 79b46d3cdeee3c8947262d926fcde8e152aabb749f88e4d1d46db61ea61822a5 not found: ID does not exist" containerID="79b46d3cdeee3c8947262d926fcde8e152aabb749f88e4d1d46db61ea61822a5" Oct 31 13:33:03.419089 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:03.419022 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79b46d3cdeee3c8947262d926fcde8e152aabb749f88e4d1d46db61ea61822a5"} err="failed to get container status \"79b46d3cdeee3c8947262d926fcde8e152aabb749f88e4d1d46db61ea61822a5\": rpc error: code = NotFound desc = could not find container \"79b46d3cdeee3c8947262d926fcde8e152aabb749f88e4d1d46db61ea61822a5\": container with ID starting with 79b46d3cdeee3c8947262d926fcde8e152aabb749f88e4d1d46db61ea61822a5 not found: ID does not exist" Oct 31 13:33:03.443622 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:03.443593 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7"] Oct 31 13:33:03.452427 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:03.451840 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c6c445-openshift-default-5744698459-fgrv7"] Oct 31 13:33:03.646068 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:03.645965 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f" path="/var/lib/kubelet/pods/c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f/volumes" Oct 31 13:33:04.181217 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:04.181182 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:04.401864 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:04.401827 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" event={"ID":"c7a4d6d4-06ba-4385-94cc-66f96e07232a","Type":"ContainerStarted","Data":"480da5b0e6d30ecb1fad6b5158c5143fa7950fd60a6dcdd6e8800d19391918c5"} Oct 31 13:33:04.865137 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:04.861868 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:05.390164 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:05.390125 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:33:05.391547 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:05.391516 6841 patch_prober.go:29] interesting pod/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 13:33:05.391693 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:05.391564 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" podUID="c7a4d6d4-06ba-4385-94cc-66f96e07232a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 13:33:05.409060 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:05.409030 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:33:05.666107 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:05.663451 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:06.310855 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:06.309658 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:06.390515 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:06.390475 6841 patch_prober.go:29] interesting pod/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 13:33:06.390698 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:06.390533 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" podUID="c7a4d6d4-06ba-4385-94cc-66f96e07232a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 13:33:07.133340 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:07.133308 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:07.390497 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:07.390411 6841 patch_prober.go:29] interesting pod/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 13:33:07.390497 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:07.390471 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" podUID="c7a4d6d4-06ba-4385-94cc-66f96e07232a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 13:33:08.210049 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:08.210010 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:08.390941 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:08.390900 6841 patch_prober.go:29] interesting pod/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 13:33:08.391449 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:08.391013 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" podUID="c7a4d6d4-06ba-4385-94cc-66f96e07232a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 13:33:08.615822 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:08.615743 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:08.803095 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:08.803057 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8"] Oct 31 13:33:08.803867 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:08.803846 6841 cpu_manager.go:414] "containerMap: removing container" podUID="d90d4670-3be8-4966-b1b5-cbe007b9a8ff" containerName="istio-proxy" Oct 31 13:33:08.803867 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:08.803864 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d90d4670-3be8-4966-b1b5-cbe007b9a8ff" containerName="istio-proxy" Oct 31 13:33:08.804069 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:08.803898 6841 cpu_manager.go:414] "containerMap: removing container" podUID="c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f" containerName="istio-proxy" Oct 31 13:33:08.804069 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:08.803903 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f" containerName="istio-proxy" Oct 31 13:33:08.804069 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:08.803912 6841 cpu_manager.go:414] "containerMap: removing container" podUID="c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f" containerName="istio-proxy" Oct 31 13:33:08.804069 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:08.803918 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f" containerName="istio-proxy" Oct 31 13:33:08.804069 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:08.803930 6841 cpu_manager.go:414] "containerMap: removing container" podUID="4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a" containerName="istio-proxy" Oct 31 13:33:08.804069 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:08.803935 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a" containerName="istio-proxy" Oct 31 13:33:08.804069 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:08.804071 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a" containerName="istio-proxy" Oct 31 13:33:08.804307 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:08.804086 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="d90d4670-3be8-4966-b1b5-cbe007b9a8ff" containerName="istio-proxy" Oct 31 13:33:08.804307 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:08.804098 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f" containerName="istio-proxy" Oct 31 13:33:08.804307 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:08.804104 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9d2d6ca-9ce2-421e-919c-7d5c8200ad4f" containerName="istio-proxy" Oct 31 13:33:08.809129 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:08.809105 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" Oct 31 13:33:08.819707 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:08.819687 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-3cddb3-openshift-default-dockercfg-9cn7x\"" Oct 31 13:33:08.825078 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:08.825054 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8"] Oct 31 13:33:08.970021 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:08.969960 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5c84160-0130-4f19-a32f-1b1b0f038198-workload-certs\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8\" (UID: \"b5c84160-0130-4f19-a32f-1b1b0f038198\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" Oct 31 13:33:08.970235 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:08.970073 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vvsc\" (UniqueName: \"kubernetes.io/projected/b5c84160-0130-4f19-a32f-1b1b0f038198-kube-api-access-4vvsc\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8\" (UID: \"b5c84160-0130-4f19-a32f-1b1b0f038198\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" Oct 31 13:33:08.970235 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:08.970177 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5c84160-0130-4f19-a32f-1b1b0f038198-istio-data\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8\" (UID: \"b5c84160-0130-4f19-a32f-1b1b0f038198\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" Oct 31 13:33:08.970235 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:08.970202 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5c84160-0130-4f19-a32f-1b1b0f038198-istiod-ca-cert\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8\" (UID: \"b5c84160-0130-4f19-a32f-1b1b0f038198\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" Oct 31 13:33:08.970432 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:08.970363 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5c84160-0130-4f19-a32f-1b1b0f038198-istio-podinfo\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8\" (UID: \"b5c84160-0130-4f19-a32f-1b1b0f038198\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" Oct 31 13:33:08.970432 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:08.970410 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5c84160-0130-4f19-a32f-1b1b0f038198-credential-socket\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8\" (UID: \"b5c84160-0130-4f19-a32f-1b1b0f038198\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" Oct 31 13:33:08.970533 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:08.970470 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5c84160-0130-4f19-a32f-1b1b0f038198-istio-token\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8\" (UID: \"b5c84160-0130-4f19-a32f-1b1b0f038198\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" Oct 31 13:33:08.970588 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:08.970570 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5c84160-0130-4f19-a32f-1b1b0f038198-workload-socket\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8\" (UID: \"b5c84160-0130-4f19-a32f-1b1b0f038198\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" Oct 31 13:33:08.970662 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:08.970650 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5c84160-0130-4f19-a32f-1b1b0f038198-istio-envoy\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8\" (UID: \"b5c84160-0130-4f19-a32f-1b1b0f038198\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" Oct 31 13:33:09.070101 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:09.067645 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:09.072318 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:09.072282 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5c84160-0130-4f19-a32f-1b1b0f038198-workload-certs\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8\" (UID: \"b5c84160-0130-4f19-a32f-1b1b0f038198\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" Oct 31 13:33:09.072484 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:09.072338 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vvsc\" (UniqueName: \"kubernetes.io/projected/b5c84160-0130-4f19-a32f-1b1b0f038198-kube-api-access-4vvsc\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8\" (UID: \"b5c84160-0130-4f19-a32f-1b1b0f038198\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" Oct 31 13:33:09.072484 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:09.072425 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5c84160-0130-4f19-a32f-1b1b0f038198-istio-data\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8\" (UID: \"b5c84160-0130-4f19-a32f-1b1b0f038198\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" Oct 31 13:33:09.072484 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:09.072460 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5c84160-0130-4f19-a32f-1b1b0f038198-istiod-ca-cert\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8\" (UID: \"b5c84160-0130-4f19-a32f-1b1b0f038198\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" Oct 31 13:33:09.072606 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:09.072554 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5c84160-0130-4f19-a32f-1b1b0f038198-istio-podinfo\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8\" (UID: \"b5c84160-0130-4f19-a32f-1b1b0f038198\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" Oct 31 13:33:09.072606 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:09.072583 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5c84160-0130-4f19-a32f-1b1b0f038198-credential-socket\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8\" (UID: \"b5c84160-0130-4f19-a32f-1b1b0f038198\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" Oct 31 13:33:09.072683 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:09.072608 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5c84160-0130-4f19-a32f-1b1b0f038198-istio-token\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8\" (UID: \"b5c84160-0130-4f19-a32f-1b1b0f038198\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" Oct 31 13:33:09.072725 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:09.072683 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5c84160-0130-4f19-a32f-1b1b0f038198-workload-socket\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8\" (UID: \"b5c84160-0130-4f19-a32f-1b1b0f038198\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" Oct 31 13:33:09.072799 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:09.072770 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5c84160-0130-4f19-a32f-1b1b0f038198-workload-certs\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8\" (UID: \"b5c84160-0130-4f19-a32f-1b1b0f038198\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" Oct 31 13:33:09.072841 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:09.072793 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5c84160-0130-4f19-a32f-1b1b0f038198-istio-envoy\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8\" (UID: \"b5c84160-0130-4f19-a32f-1b1b0f038198\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" Oct 31 13:33:09.072841 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:09.072826 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5c84160-0130-4f19-a32f-1b1b0f038198-istio-data\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8\" (UID: \"b5c84160-0130-4f19-a32f-1b1b0f038198\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" Oct 31 13:33:09.073337 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:09.073314 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5c84160-0130-4f19-a32f-1b1b0f038198-credential-socket\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8\" (UID: \"b5c84160-0130-4f19-a32f-1b1b0f038198\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" Oct 31 13:33:09.073469 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:09.073381 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5c84160-0130-4f19-a32f-1b1b0f038198-workload-socket\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8\" (UID: \"b5c84160-0130-4f19-a32f-1b1b0f038198\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" Oct 31 13:33:09.073626 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:09.073605 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5c84160-0130-4f19-a32f-1b1b0f038198-istiod-ca-cert\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8\" (UID: \"b5c84160-0130-4f19-a32f-1b1b0f038198\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" Oct 31 13:33:09.075591 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:09.075543 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5c84160-0130-4f19-a32f-1b1b0f038198-istio-envoy\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8\" (UID: \"b5c84160-0130-4f19-a32f-1b1b0f038198\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" Oct 31 13:33:09.075995 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:09.075960 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5c84160-0130-4f19-a32f-1b1b0f038198-istio-podinfo\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8\" (UID: \"b5c84160-0130-4f19-a32f-1b1b0f038198\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" Oct 31 13:33:09.084830 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:09.084800 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5c84160-0130-4f19-a32f-1b1b0f038198-istio-token\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8\" (UID: \"b5c84160-0130-4f19-a32f-1b1b0f038198\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" Oct 31 13:33:09.085436 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:09.085414 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vvsc\" (UniqueName: \"kubernetes.io/projected/b5c84160-0130-4f19-a32f-1b1b0f038198-kube-api-access-4vvsc\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8\" (UID: \"b5c84160-0130-4f19-a32f-1b1b0f038198\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" Oct 31 13:33:09.129385 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:09.129355 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" Oct 31 13:33:09.390380 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:09.390343 6841 patch_prober.go:29] interesting pod/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 13:33:09.390560 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:09.390411 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" podUID="c7a4d6d4-06ba-4385-94cc-66f96e07232a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 13:33:09.472782 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:09.472601 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8"] Oct 31 13:33:09.474137 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:33:09.474108 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5c84160_0130_4f19_a32f_1b1b0f038198.slice/crio-6c216900131950a3c18e163d8f434f6abf65172f16d59d253a7187eb7769f99d WatchSource:0}: Error finding container 6c216900131950a3c18e163d8f434f6abf65172f16d59d253a7187eb7769f99d: Status 404 returned error can't find the container with id 6c216900131950a3c18e163d8f434f6abf65172f16d59d253a7187eb7769f99d Oct 31 13:33:09.479630 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:09.479593 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:33:09.479744 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:09.479681 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:33:09.479744 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:09.479727 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:33:09.819410 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:09.818531 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:10.395282 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:10.395251 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:33:10.396463 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:10.396443 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:33:10.495005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:10.494793 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" event={"ID":"b5c84160-0130-4f19-a32f-1b1b0f038198","Type":"ContainerStarted","Data":"12a237c2e1ebeadf6e51599ba3e65653a56f07b7b5673a82858e92d18b979baa"} Oct 31 13:33:10.495005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:10.494854 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" event={"ID":"b5c84160-0130-4f19-a32f-1b1b0f038198","Type":"ContainerStarted","Data":"6c216900131950a3c18e163d8f434f6abf65172f16d59d253a7187eb7769f99d"} Oct 31 13:33:10.526326 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:10.526276 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" podStartSLOduration=2.526261239 podStartE2EDuration="2.526261239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:33:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:33:10.521937027 +0000 UTC m=+3637.005059961" watchObservedRunningTime="2026-10-31 13:33:10.526261239 +0000 UTC m=+3637.009384201" Oct 31 13:33:10.635008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:10.623311 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:11.130190 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:11.130156 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" Oct 31 13:33:11.132243 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:11.132218 6841 patch_prober.go:29] interesting pod/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 31 13:33:11.132381 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:11.132266 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" podUID="b5c84160-0130-4f19-a32f-1b1b0f038198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 31 13:33:11.475290 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:11.475227 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht"] Oct 31 13:33:11.476178 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:11.476159 6841 cpu_manager.go:414] "containerMap: removing container" podUID="4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a" containerName="istio-proxy" Oct 31 13:33:11.476236 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:11.476179 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a" containerName="istio-proxy" Oct 31 13:33:11.476330 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:11.476319 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dfe8ea5-8d8c-4b1e-ae0b-d7b7483b6b1a" containerName="istio-proxy" Oct 31 13:33:11.479745 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:11.479721 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht" Oct 31 13:33:11.486427 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:11.486363 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-50b4aa\"/\"test-model-50b4aa-kserve-self-signed-certs\"" Oct 31 13:33:11.487996 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:11.487951 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-50b4aa\"/\"openshift-service-ca.crt\"" Oct 31 13:33:11.488110 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:11.488029 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-50b4aa\"/\"kube-root-ca.crt\"" Oct 31 13:33:11.488172 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:11.488109 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-50b4aa\"/\"default-dockercfg-r2p5k\"" Oct 31 13:33:11.502583 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:11.502553 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht"] Oct 31 13:33:11.523078 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:11.523052 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:11.567513 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:11.552077 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b413db83-3583-447b-ab12-b6a73a4769be-tmp-dir\") pod \"test-model-50b4aa-kserve-5b55694797-bvhht\" (UID: \"b413db83-3583-447b-ab12-b6a73a4769be\") " pod="ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht" Oct 31 13:33:11.567725 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:11.567544 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b413db83-3583-447b-ab12-b6a73a4769be-tls-certs\") pod \"test-model-50b4aa-kserve-5b55694797-bvhht\" (UID: \"b413db83-3583-447b-ab12-b6a73a4769be\") " pod="ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht" Oct 31 13:33:11.567799 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:11.567741 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b413db83-3583-447b-ab12-b6a73a4769be-dshm\") pod \"test-model-50b4aa-kserve-5b55694797-bvhht\" (UID: \"b413db83-3583-447b-ab12-b6a73a4769be\") " pod="ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht" Oct 31 13:33:11.568038 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:11.568013 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b413db83-3583-447b-ab12-b6a73a4769be-home\") pod \"test-model-50b4aa-kserve-5b55694797-bvhht\" (UID: \"b413db83-3583-447b-ab12-b6a73a4769be\") " pod="ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht" Oct 31 13:33:11.568198 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:11.568144 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b413db83-3583-447b-ab12-b6a73a4769be-model-cache\") pod \"test-model-50b4aa-kserve-5b55694797-bvhht\" (UID: \"b413db83-3583-447b-ab12-b6a73a4769be\") " pod="ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht" Oct 31 13:33:11.568287 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:11.568213 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ghzz\" (UniqueName: \"kubernetes.io/projected/b413db83-3583-447b-ab12-b6a73a4769be-kube-api-access-4ghzz\") pod \"test-model-50b4aa-kserve-5b55694797-bvhht\" (UID: \"b413db83-3583-447b-ab12-b6a73a4769be\") " pod="ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht" Oct 31 13:33:11.683024 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:11.682426 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b413db83-3583-447b-ab12-b6a73a4769be-model-cache\") pod \"test-model-50b4aa-kserve-5b55694797-bvhht\" (UID: \"b413db83-3583-447b-ab12-b6a73a4769be\") " pod="ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht" Oct 31 13:33:11.683024 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:11.682506 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ghzz\" (UniqueName: \"kubernetes.io/projected/b413db83-3583-447b-ab12-b6a73a4769be-kube-api-access-4ghzz\") pod \"test-model-50b4aa-kserve-5b55694797-bvhht\" (UID: \"b413db83-3583-447b-ab12-b6a73a4769be\") " pod="ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht" Oct 31 13:33:11.683024 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:11.682671 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b413db83-3583-447b-ab12-b6a73a4769be-tmp-dir\") pod \"test-model-50b4aa-kserve-5b55694797-bvhht\" (UID: \"b413db83-3583-447b-ab12-b6a73a4769be\") " pod="ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht" Oct 31 13:33:11.683024 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:11.682768 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b413db83-3583-447b-ab12-b6a73a4769be-tls-certs\") pod \"test-model-50b4aa-kserve-5b55694797-bvhht\" (UID: \"b413db83-3583-447b-ab12-b6a73a4769be\") " pod="ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht" Oct 31 13:33:11.683024 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:11.682910 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b413db83-3583-447b-ab12-b6a73a4769be-dshm\") pod \"test-model-50b4aa-kserve-5b55694797-bvhht\" (UID: \"b413db83-3583-447b-ab12-b6a73a4769be\") " pod="ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht" Oct 31 13:33:11.683024 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:11.682968 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b413db83-3583-447b-ab12-b6a73a4769be-home\") pod \"test-model-50b4aa-kserve-5b55694797-bvhht\" (UID: \"b413db83-3583-447b-ab12-b6a73a4769be\") " pod="ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht" Oct 31 13:33:11.686021 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:11.683647 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b413db83-3583-447b-ab12-b6a73a4769be-home\") pod \"test-model-50b4aa-kserve-5b55694797-bvhht\" (UID: \"b413db83-3583-447b-ab12-b6a73a4769be\") " pod="ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht" Oct 31 13:33:11.686021 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:11.683925 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b413db83-3583-447b-ab12-b6a73a4769be-model-cache\") pod \"test-model-50b4aa-kserve-5b55694797-bvhht\" (UID: \"b413db83-3583-447b-ab12-b6a73a4769be\") " pod="ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht" Oct 31 13:33:11.686021 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:11.684489 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b413db83-3583-447b-ab12-b6a73a4769be-tmp-dir\") pod \"test-model-50b4aa-kserve-5b55694797-bvhht\" (UID: \"b413db83-3583-447b-ab12-b6a73a4769be\") " pod="ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht" Oct 31 13:33:11.718650 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:11.715373 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b413db83-3583-447b-ab12-b6a73a4769be-dshm\") pod \"test-model-50b4aa-kserve-5b55694797-bvhht\" (UID: \"b413db83-3583-447b-ab12-b6a73a4769be\") " pod="ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht" Oct 31 13:33:11.747647 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:11.736166 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b413db83-3583-447b-ab12-b6a73a4769be-tls-certs\") pod \"test-model-50b4aa-kserve-5b55694797-bvhht\" (UID: \"b413db83-3583-447b-ab12-b6a73a4769be\") " pod="ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht" Oct 31 13:33:11.771610 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:11.766609 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ghzz\" (UniqueName: \"kubernetes.io/projected/b413db83-3583-447b-ab12-b6a73a4769be-kube-api-access-4ghzz\") pod \"test-model-50b4aa-kserve-5b55694797-bvhht\" (UID: \"b413db83-3583-447b-ab12-b6a73a4769be\") " pod="ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht" Oct 31 13:33:11.817007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:11.812626 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:11.817007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:11.814287 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht" Oct 31 13:33:12.130108 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:12.130025 6841 patch_prober.go:29] interesting pod/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 31 13:33:12.130108 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:12.130083 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" podUID="b5c84160-0130-4f19-a32f-1b1b0f038198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 31 13:33:12.171023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:12.170974 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht"] Oct 31 13:33:12.171243 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:33:12.171219 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb413db83_3583_447b_ab12_b6a73a4769be.slice/crio-1e44e4d3e70e498f819a2b807592e8a4e9aa4d33c52e1301e66150ccd30e3377 WatchSource:0}: Error finding container 1e44e4d3e70e498f819a2b807592e8a4e9aa4d33c52e1301e66150ccd30e3377: Status 404 returned error can't find the container with id 1e44e4d3e70e498f819a2b807592e8a4e9aa4d33c52e1301e66150ccd30e3377 Oct 31 13:33:12.514624 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:12.514583 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:12.524666 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:12.524633 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht" event={"ID":"b413db83-3583-447b-ab12-b6a73a4769be","Type":"ContainerStarted","Data":"bf9f4700bd80a86e85c6358bd9a2d882a9949f67f0e29b79c6af11a949917009"} Oct 31 13:33:12.524837 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:12.524673 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht" event={"ID":"b413db83-3583-447b-ab12-b6a73a4769be","Type":"ContainerStarted","Data":"1e44e4d3e70e498f819a2b807592e8a4e9aa4d33c52e1301e66150ccd30e3377"} Oct 31 13:33:12.561049 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:12.560961 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht" podStartSLOduration=1.5609430450000001 podStartE2EDuration="1.560943045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:33:12.560419774 +0000 UTC m=+3639.043542676" watchObservedRunningTime="2026-10-31 13:33:12.560943045 +0000 UTC m=+3639.044065950" Oct 31 13:33:12.814879 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:12.814800 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht" Oct 31 13:33:13.130518 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:13.130439 6841 patch_prober.go:29] interesting pod/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 31 13:33:13.130518 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:13.130493 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" podUID="b5c84160-0130-4f19-a32f-1b1b0f038198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 31 13:33:13.283100 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:13.283064 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:14.017856 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:14.017820 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:14.130199 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:14.130166 6841 patch_prober.go:29] interesting pod/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 31 13:33:14.130378 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:14.130232 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" podUID="b5c84160-0130-4f19-a32f-1b1b0f038198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 31 13:33:14.915619 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:14.915580 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:15.137017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:15.136824 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:15.140030 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:15.137609 6841 patch_prober.go:29] interesting pod/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 31 13:33:15.140030 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:15.137699 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" podUID="b5c84160-0130-4f19-a32f-1b1b0f038198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 31 13:33:15.708008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:15.705815 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:16.130469 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:16.130380 6841 patch_prober.go:29] interesting pod/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 31 13:33:16.130469 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:16.130437 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" podUID="b5c84160-0130-4f19-a32f-1b1b0f038198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 31 13:33:16.470094 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:16.469971 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:17.151903 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:17.151864 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" Oct 31 13:33:17.153716 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:17.153663 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" Oct 31 13:33:17.161714 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:17.159550 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" Oct 31 13:33:17.231766 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:17.231723 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:17.910428 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:17.910383 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:18.595564 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:18.595521 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:19.326711 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.326674 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz"] Oct 31 13:33:19.331700 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.331679 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" Oct 31 13:33:19.341568 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.340789 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-691f1e-openshift-default-dockercfg-bjw8b\"" Oct 31 13:33:19.344947 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.344920 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz"] Oct 31 13:33:19.470815 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.470774 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-istiod-ca-cert\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz\" (UID: \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" Oct 31 13:33:19.470815 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.470818 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-workload-certs\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz\" (UID: \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" Oct 31 13:33:19.471077 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.470853 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-istio-podinfo\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz\" (UID: \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" Oct 31 13:33:19.471122 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.471096 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-istio-envoy\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz\" (UID: \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" Oct 31 13:33:19.471236 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.471218 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-istio-data\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz\" (UID: \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" Oct 31 13:33:19.471275 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.471257 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-credential-socket\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz\" (UID: \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" Oct 31 13:33:19.471402 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.471389 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-workload-socket\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz\" (UID: \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" Oct 31 13:33:19.471437 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.471412 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxcgh\" (UniqueName: \"kubernetes.io/projected/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-kube-api-access-dxcgh\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz\" (UID: \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" Oct 31 13:33:19.471469 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.471450 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-istio-token\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz\" (UID: \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" Oct 31 13:33:19.545735 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.545698 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:19.573368 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.573329 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-workload-socket\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz\" (UID: \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" Oct 31 13:33:19.573571 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.573373 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxcgh\" (UniqueName: \"kubernetes.io/projected/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-kube-api-access-dxcgh\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz\" (UID: \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" Oct 31 13:33:19.573571 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.573401 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-istio-token\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz\" (UID: \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" Oct 31 13:33:19.573571 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.573482 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-istiod-ca-cert\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz\" (UID: \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" Oct 31 13:33:19.573571 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.573519 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-workload-certs\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz\" (UID: \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" Oct 31 13:33:19.573571 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.573559 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-istio-podinfo\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz\" (UID: \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" Oct 31 13:33:19.573886 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.573746 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-istio-envoy\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz\" (UID: \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" Oct 31 13:33:19.573886 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.573772 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-workload-socket\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz\" (UID: \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" Oct 31 13:33:19.574029 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.573908 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-istio-data\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz\" (UID: \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" Oct 31 13:33:19.574029 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.573939 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-credential-socket\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz\" (UID: \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" Oct 31 13:33:19.574029 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.573948 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-workload-certs\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz\" (UID: \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" Oct 31 13:33:19.574207 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.574180 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-istiod-ca-cert\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz\" (UID: \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" Oct 31 13:33:19.574291 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.574268 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-credential-socket\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz\" (UID: \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" Oct 31 13:33:19.574341 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.574293 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-istio-data\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz\" (UID: \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" Oct 31 13:33:19.576202 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.576178 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-istio-envoy\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz\" (UID: \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" Oct 31 13:33:19.576312 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.576284 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-istio-podinfo\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz\" (UID: \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" Oct 31 13:33:19.586741 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.586688 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxcgh\" (UniqueName: \"kubernetes.io/projected/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-kube-api-access-dxcgh\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz\" (UID: \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" Oct 31 13:33:19.587188 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.587171 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-istio-token\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz\" (UID: \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" Oct 31 13:33:19.657719 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.657690 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" Oct 31 13:33:19.964900 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.964871 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz"] Oct 31 13:33:19.965551 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:33:19.965525 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e6784e6_2aeb_4b2d_b1f8_97f903337e82.slice/crio-e220e5f6e772d3e6b9653e9a9d07ac61d4a7a895f5c29e9f3a44d04d8dcc5fae WatchSource:0}: Error finding container e220e5f6e772d3e6b9653e9a9d07ac61d4a7a895f5c29e9f3a44d04d8dcc5fae: Status 404 returned error can't find the container with id e220e5f6e772d3e6b9653e9a9d07ac61d4a7a895f5c29e9f3a44d04d8dcc5fae Oct 31 13:33:19.969239 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.969201 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:33:19.969338 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.969288 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:33:19.969338 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:19.969330 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:33:20.039525 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:33:20.039492 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:33:20.632996 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:20.632953 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" event={"ID":"0e6784e6-2aeb-4b2d-b1f8-97f903337e82","Type":"ContainerStarted","Data":"58211cd76607bd49c2f389efdbb9a862de0151b95845dea76c651d897c26eb55"} Oct 31 13:33:20.633511 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:20.633031 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" event={"ID":"0e6784e6-2aeb-4b2d-b1f8-97f903337e82","Type":"ContainerStarted","Data":"e220e5f6e772d3e6b9653e9a9d07ac61d4a7a895f5c29e9f3a44d04d8dcc5fae"} Oct 31 13:33:20.668009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:20.667804 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" Oct 31 13:33:20.668960 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:20.668924 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:21.416079 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:21.415932 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:21.676233 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:21.676138 6841 patch_prober.go:29] interesting pod/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 31 13:33:21.676233 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:21.676213 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" podUID="0e6784e6-2aeb-4b2d-b1f8-97f903337e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 31 13:33:21.816334 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:21.816296 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht" Oct 31 13:33:21.831078 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:21.831052 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht" Oct 31 13:33:21.869975 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:21.869927 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" podStartSLOduration=2.869911777 podStartE2EDuration="2.869911777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:33:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:33:20.667760403 +0000 UTC m=+3647.150883305" watchObservedRunningTime="2026-10-31 13:33:21.869911777 +0000 UTC m=+3648.353034679" Oct 31 13:33:22.226345 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:22.226306 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:22.662068 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:22.661397 6841 patch_prober.go:29] interesting pod/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 31 13:33:22.662068 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:22.661464 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" podUID="0e6784e6-2aeb-4b2d-b1f8-97f903337e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 31 13:33:22.698860 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:22.698830 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht" Oct 31 13:33:22.914166 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:22.912604 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:23.679006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:23.672104 6841 patch_prober.go:29] interesting pod/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 31 13:33:23.679006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:23.672170 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" podUID="0e6784e6-2aeb-4b2d-b1f8-97f903337e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 31 13:33:23.726924 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:23.724594 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:24.434635 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:24.434459 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:24.660006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:24.659356 6841 patch_prober.go:29] interesting pod/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 31 13:33:24.660006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:24.659434 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" podUID="0e6784e6-2aeb-4b2d-b1f8-97f903337e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 31 13:33:25.303362 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:25.303328 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:25.666186 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:25.666094 6841 patch_prober.go:29] interesting pod/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 31 13:33:25.666486 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:25.666451 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" podUID="0e6784e6-2aeb-4b2d-b1f8-97f903337e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 31 13:33:26.008824 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:26.008776 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:26.664657 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:26.664618 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" Oct 31 13:33:26.665324 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:26.665270 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" Oct 31 13:33:26.666072 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:26.666052 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" Oct 31 13:33:26.824819 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:26.824774 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:27.718737 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:27.718701 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:28.397000 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:28.396949 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:29.129913 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:29.129879 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:29.908394 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:29.908354 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:30.611724 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:30.611687 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:31.364609 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:31.364577 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:32.112400 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:32.112359 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:32.869306 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:32.869260 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:33.642566 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:33.642536 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:34.502442 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:34.502410 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:35.531799 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:35.531746 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:36.222570 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:36.222535 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:36.947134 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:36.947098 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:37.899763 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:37.899720 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:38.807472 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:38.807437 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:39.715437 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:39.715369 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:39.998428 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:39.998311 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:40.865349 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:40.865289 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:41.822704 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:41.822663 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:42.623585 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:42.623550 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:43.830519 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:43.830478 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:44.596589 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:44.596555 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:45.372117 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:45.372075 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:46.113323 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:46.113290 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:46.963577 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:46.963545 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:47.829386 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:47.829350 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:48.531024 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:48.530951 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:49.223640 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:49.223608 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:50.036568 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:33:50.036516 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:33:50.256531 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:50.256496 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:51.046228 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:51.046191 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:51.911811 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:51.911773 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:52.597809 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:52.597775 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:53.356189 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:53.356156 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:54.115491 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:54.115450 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:54.802310 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:54.802270 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:55.530210 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:55.530172 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:56.400432 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:56.400381 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:57.210320 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:57.210285 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:57.964433 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:57.964398 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:58.711831 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:58.711795 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:33:59.727279 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:33:59.727241 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:00.555654 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:00.555611 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:01.294306 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:01.294266 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:02.086925 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:02.086886 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:02.734582 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:02.734542 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:02.933302 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:02.933268 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:03.704469 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:03.704433 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:04.506259 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:04.506225 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:05.388176 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:05.388138 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:06.122609 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:06.122576 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:06.806845 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:06.806809 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:07.683184 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:07.683146 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:08.387250 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:08.387210 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:09.097933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:09.097854 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:09.866087 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:09.866054 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:10.645464 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:10.645426 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:11.350805 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:11.350774 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:12.308608 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:12.308521 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:13.042681 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:13.042635 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:13.928276 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:13.928240 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:14.814375 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:14.814340 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:15.546149 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:15.546111 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:16.327624 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:16.327581 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:17.037901 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:17.037861 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:17.965825 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:17.965790 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:18.730245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:18.730211 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:19.714966 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:19.714924 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:20.037797 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:34:20.037706 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:34:20.538444 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:20.538411 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:21.273262 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:21.273226 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:22.132837 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:22.132803 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:23.182453 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:23.182416 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:23.943848 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:23.943792 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:24.150361 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:24.150325 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:24.935027 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:24.934972 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:25.608149 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:25.608113 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:26.346417 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:26.346376 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:27.130009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:27.129959 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:27.975917 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:27.975882 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:28.700176 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:28.700142 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:29.518058 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:29.517972 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:30.252113 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:30.252078 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:31.029379 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:31.029343 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:31.836248 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:31.836208 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:32.625686 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:32.625634 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:33.441100 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:33.441060 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:34.191556 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:34.191516 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:35.119174 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:35.119139 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:35.800677 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:35.800640 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:36.540847 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:36.540809 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:37.376611 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:37.376576 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:37.999341 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:37.999301 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:38.792881 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:38.792846 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:39.137784 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:39.137690 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:39.438185 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:39.438096 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:39.846352 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:39.846310 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:40.128786 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:40.128694 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:40.652373 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:40.652336 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:41.951910 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:41.951877 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:42.859669 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:42.859633 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:43.061600 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:43.061564 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:43.659233 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:43.659199 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:44.460097 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:44.460052 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:45.230900 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:45.230862 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:45.989439 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:45.989402 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:46.691612 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:46.691578 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:47.431954 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:47.431908 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:48.151302 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:48.151262 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:49.003225 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:49.003190 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:49.287354 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:49.287268 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:49.977298 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:49.977264 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:50.037335 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:34:50.037299 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:34:50.799006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:50.795144 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:51.501787 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:51.501747 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:52.226184 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:52.226132 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:52.991442 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:52.991404 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:53.701265 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:53.701229 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:53.967738 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:53.967651 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd"] Oct 31 13:34:54.133570 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:54.133535 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:54.724872 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:54.724835 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:55.332274 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:55.332238 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:55.693183 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:55.693150 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:56.550852 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:56.550812 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:57.358682 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:57.358641 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:58.125907 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:58.125868 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:59.067602 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:59.067566 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:34:59.850218 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:34:59.850153 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:00.640433 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:00.640307 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:00.740258 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:00.739291 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx"] Oct 31 13:35:00.740258 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:00.739552 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" podUID="41ec82c1-2f15-4a66-8211-330a7a5029cf" containerName="istio-proxy" containerID="cri-o://600ec71a0cd3c632ef9326206b93a90e760fb443216f4534e6e92271b8b3b25c" gracePeriod=30 Oct 31 13:35:01.494470 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:01.494437 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:01.727258 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:01.727227 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:02.313780 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:02.313746 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:03.227894 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:03.227836 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:04.174456 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:04.174418 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:04.904126 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:04.904092 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:05.383142 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:05.383102 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:05.733305 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:05.733270 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:05.944669 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:05.944638 6841 generic.go:356] "Generic (PLEG): container finished" podID="41ec82c1-2f15-4a66-8211-330a7a5029cf" containerID="600ec71a0cd3c632ef9326206b93a90e760fb443216f4534e6e92271b8b3b25c" exitCode=0 Oct 31 13:35:05.944669 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:05.944663 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" event={"ID":"41ec82c1-2f15-4a66-8211-330a7a5029cf","Type":"ContainerDied","Data":"600ec71a0cd3c632ef9326206b93a90e760fb443216f4534e6e92271b8b3b25c"} Oct 31 13:35:06.059115 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:06.058921 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:06.125437 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:06.125408 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" Oct 31 13:35:06.250716 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:06.250687 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41ec82c1-2f15-4a66-8211-330a7a5029cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41ec82c1-2f15-4a66-8211-330a7a5029cf-istio-data\") pod \"41ec82c1-2f15-4a66-8211-330a7a5029cf\" (UID: \"41ec82c1-2f15-4a66-8211-330a7a5029cf\") " Oct 31 13:35:06.250890 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:06.250758 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41ec82c1-2f15-4a66-8211-330a7a5029cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41ec82c1-2f15-4a66-8211-330a7a5029cf-workload-certs\") pod \"41ec82c1-2f15-4a66-8211-330a7a5029cf\" (UID: \"41ec82c1-2f15-4a66-8211-330a7a5029cf\") " Oct 31 13:35:06.250890 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:06.250867 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41ec82c1-2f15-4a66-8211-330a7a5029cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41ec82c1-2f15-4a66-8211-330a7a5029cf-istio-envoy\") pod \"41ec82c1-2f15-4a66-8211-330a7a5029cf\" (UID: \"41ec82c1-2f15-4a66-8211-330a7a5029cf\") " Oct 31 13:35:06.251051 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:06.250923 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41ec82c1-2f15-4a66-8211-330a7a5029cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ec82c1-2f15-4a66-8211-330a7a5029cf-credential-socket\") pod \"41ec82c1-2f15-4a66-8211-330a7a5029cf\" (UID: \"41ec82c1-2f15-4a66-8211-330a7a5029cf\") " Oct 31 13:35:06.251051 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:06.250948 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41ec82c1-2f15-4a66-8211-330a7a5029cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ec82c1-2f15-4a66-8211-330a7a5029cf-workload-socket\") pod \"41ec82c1-2f15-4a66-8211-330a7a5029cf\" (UID: \"41ec82c1-2f15-4a66-8211-330a7a5029cf\") " Oct 31 13:35:06.251051 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:06.251033 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ec82c1-2f15-4a66-8211-330a7a5029cf-istio-data" pod "41ec82c1-2f15-4a66-8211-330a7a5029cf" (UID: "41ec82c1-2f15-4a66-8211-330a7a5029cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:35:06.251250 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:06.251053 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ec82c1-2f15-4a66-8211-330a7a5029cf-workload-certs" pod "41ec82c1-2f15-4a66-8211-330a7a5029cf" (UID: "41ec82c1-2f15-4a66-8211-330a7a5029cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:35:06.251250 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:06.251066 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41ec82c1-2f15-4a66-8211-330a7a5029cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41ec82c1-2f15-4a66-8211-330a7a5029cf-istio-podinfo\") pod \"41ec82c1-2f15-4a66-8211-330a7a5029cf\" (UID: \"41ec82c1-2f15-4a66-8211-330a7a5029cf\") " Oct 31 13:35:06.251250 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:06.251130 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41ec82c1-2f15-4a66-8211-330a7a5029cf-kube-api-access-mdcxd\" (UniqueName: \"kubernetes.io/projected/41ec82c1-2f15-4a66-8211-330a7a5029cf-kube-api-access-mdcxd\") pod \"41ec82c1-2f15-4a66-8211-330a7a5029cf\" (UID: \"41ec82c1-2f15-4a66-8211-330a7a5029cf\") " Oct 31 13:35:06.251250 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:06.251158 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41ec82c1-2f15-4a66-8211-330a7a5029cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41ec82c1-2f15-4a66-8211-330a7a5029cf-istiod-ca-cert\") pod \"41ec82c1-2f15-4a66-8211-330a7a5029cf\" (UID: \"41ec82c1-2f15-4a66-8211-330a7a5029cf\") " Oct 31 13:35:06.251250 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:06.251229 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41ec82c1-2f15-4a66-8211-330a7a5029cf-istio-token\" (UniqueName: \"kubernetes.io/projected/41ec82c1-2f15-4a66-8211-330a7a5029cf-istio-token\") pod \"41ec82c1-2f15-4a66-8211-330a7a5029cf\" (UID: \"41ec82c1-2f15-4a66-8211-330a7a5029cf\") " Oct 31 13:35:06.251250 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:06.251230 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ec82c1-2f15-4a66-8211-330a7a5029cf-credential-socket" pod "41ec82c1-2f15-4a66-8211-330a7a5029cf" (UID: "41ec82c1-2f15-4a66-8211-330a7a5029cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:35:06.251588 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:06.251340 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ec82c1-2f15-4a66-8211-330a7a5029cf-workload-socket" pod "41ec82c1-2f15-4a66-8211-330a7a5029cf" (UID: "41ec82c1-2f15-4a66-8211-330a7a5029cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:35:06.251588 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:06.251548 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41ec82c1-2f15-4a66-8211-330a7a5029cf-istiod-ca-cert" pod "41ec82c1-2f15-4a66-8211-330a7a5029cf" (UID: "41ec82c1-2f15-4a66-8211-330a7a5029cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:35:06.252179 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:06.252151 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ec82c1-2f15-4a66-8211-330a7a5029cf-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:06.252319 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:06.252184 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ec82c1-2f15-4a66-8211-330a7a5029cf-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:06.252319 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:06.252200 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41ec82c1-2f15-4a66-8211-330a7a5029cf-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:06.252319 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:06.252216 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41ec82c1-2f15-4a66-8211-330a7a5029cf-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:06.252319 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:06.252231 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41ec82c1-2f15-4a66-8211-330a7a5029cf-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:06.253502 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:06.253469 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41ec82c1-2f15-4a66-8211-330a7a5029cf-kube-api-access-mdcxd" pod "41ec82c1-2f15-4a66-8211-330a7a5029cf" (UID: "41ec82c1-2f15-4a66-8211-330a7a5029cf"). InnerVolumeSpecName "kube-api-access-mdcxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:35:06.253610 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:06.253479 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41ec82c1-2f15-4a66-8211-330a7a5029cf-istio-podinfo" pod "41ec82c1-2f15-4a66-8211-330a7a5029cf" (UID: "41ec82c1-2f15-4a66-8211-330a7a5029cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:35:06.253659 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:06.253617 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ec82c1-2f15-4a66-8211-330a7a5029cf-istio-envoy" pod "41ec82c1-2f15-4a66-8211-330a7a5029cf" (UID: "41ec82c1-2f15-4a66-8211-330a7a5029cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:35:06.253748 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:06.253733 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41ec82c1-2f15-4a66-8211-330a7a5029cf-istio-token" pod "41ec82c1-2f15-4a66-8211-330a7a5029cf" (UID: "41ec82c1-2f15-4a66-8211-330a7a5029cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:35:06.354178 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:06.354102 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41ec82c1-2f15-4a66-8211-330a7a5029cf-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:06.354178 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:06.354132 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41ec82c1-2f15-4a66-8211-330a7a5029cf-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:06.354178 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:06.354144 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdcxd\" (UniqueName: \"kubernetes.io/projected/41ec82c1-2f15-4a66-8211-330a7a5029cf-kube-api-access-mdcxd\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:06.354178 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:06.354155 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41ec82c1-2f15-4a66-8211-330a7a5029cf-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:06.929911 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:06.929873 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:06.955600 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:06.955570 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" Oct 31 13:35:06.956120 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:06.955575 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx" event={"ID":"41ec82c1-2f15-4a66-8211-330a7a5029cf","Type":"ContainerDied","Data":"f46c8e23e05986d1821e196d613fa09801a4b5838b270562cd6c5b6a1e5dee07"} Oct 31 13:35:06.956120 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:06.955702 6841 scope.go:122] "RemoveContainer" containerID="600ec71a0cd3c632ef9326206b93a90e760fb443216f4534e6e92271b8b3b25c" Oct 31 13:35:06.993217 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:06.993192 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx"] Oct 31 13:35:07.002417 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:07.002391 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-cs4vx"] Oct 31 13:35:07.612415 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:07.612382 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:07.645487 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:07.645452 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41ec82c1-2f15-4a66-8211-330a7a5029cf" path="/var/lib/kubelet/pods/41ec82c1-2f15-4a66-8211-330a7a5029cf/volumes" Oct 31 13:35:08.433260 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:08.433221 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:08.997444 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:08.997391 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd" podUID="3ff4f1a8-99b6-4d81-8772-7ea00af5bbad" containerName="main" containerID="cri-o://03fab0677cb7639c413a34c1bc771d08ff9f19e02dad31a9f4699d8820d84368" gracePeriod=45 Oct 31 13:35:09.141140 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:09.141109 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:09.287272 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:09.287244 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd" Oct 31 13:35:09.309440 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:09.309412 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-tmp-dir\") pod \"3ff4f1a8-99b6-4d81-8772-7ea00af5bbad\" (UID: \"3ff4f1a8-99b6-4d81-8772-7ea00af5bbad\") " Oct 31 13:35:09.309619 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:09.309533 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-dshm\") pod \"3ff4f1a8-99b6-4d81-8772-7ea00af5bbad\" (UID: \"3ff4f1a8-99b6-4d81-8772-7ea00af5bbad\") " Oct 31 13:35:09.309619 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:09.309568 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-model-cache\") pod \"3ff4f1a8-99b6-4d81-8772-7ea00af5bbad\" (UID: \"3ff4f1a8-99b6-4d81-8772-7ea00af5bbad\") " Oct 31 13:35:09.309619 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:09.309611 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-home\" (UniqueName: \"kubernetes.io/empty-dir/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-home\") pod \"3ff4f1a8-99b6-4d81-8772-7ea00af5bbad\" (UID: \"3ff4f1a8-99b6-4d81-8772-7ea00af5bbad\") " Oct 31 13:35:09.309794 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:09.309712 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-tls-certs\" (UniqueName: \"kubernetes.io/secret/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-tls-certs\") pod \"3ff4f1a8-99b6-4d81-8772-7ea00af5bbad\" (UID: \"3ff4f1a8-99b6-4d81-8772-7ea00af5bbad\") " Oct 31 13:35:09.309794 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:09.309706 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-tmp-dir" pod "3ff4f1a8-99b6-4d81-8772-7ea00af5bbad" (UID: "3ff4f1a8-99b6-4d81-8772-7ea00af5bbad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:35:09.309912 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:09.309840 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-kube-api-access-4qqlz\" (UniqueName: \"kubernetes.io/projected/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-kube-api-access-4qqlz\") pod \"3ff4f1a8-99b6-4d81-8772-7ea00af5bbad\" (UID: \"3ff4f1a8-99b6-4d81-8772-7ea00af5bbad\") " Oct 31 13:35:09.309912 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:09.309842 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-model-cache" pod "3ff4f1a8-99b6-4d81-8772-7ea00af5bbad" (UID: "3ff4f1a8-99b6-4d81-8772-7ea00af5bbad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:35:09.309912 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:09.309853 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-home" pod "3ff4f1a8-99b6-4d81-8772-7ea00af5bbad" (UID: "3ff4f1a8-99b6-4d81-8772-7ea00af5bbad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:35:09.310744 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:09.310721 6841 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-tmp-dir\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:09.310813 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:09.310753 6841 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-model-cache\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:09.310813 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:09.310768 6841 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-home\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:09.311888 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:09.311869 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-dshm" pod "3ff4f1a8-99b6-4d81-8772-7ea00af5bbad" (UID: "3ff4f1a8-99b6-4d81-8772-7ea00af5bbad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:35:09.311962 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:09.311888 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-tls-certs" pod "3ff4f1a8-99b6-4d81-8772-7ea00af5bbad" (UID: "3ff4f1a8-99b6-4d81-8772-7ea00af5bbad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:35:09.312062 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:09.312042 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-kube-api-access-4qqlz" pod "3ff4f1a8-99b6-4d81-8772-7ea00af5bbad" (UID: "3ff4f1a8-99b6-4d81-8772-7ea00af5bbad"). InnerVolumeSpecName "kube-api-access-4qqlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:35:09.412706 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:09.412671 6841 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-dshm\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:09.412706 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:09.412701 6841 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-tls-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:09.412706 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:09.412712 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qqlz\" (UniqueName: \"kubernetes.io/projected/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad-kube-api-access-4qqlz\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:10.025205 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:10.025163 6841 generic.go:356] "Generic (PLEG): container finished" podID="3ff4f1a8-99b6-4d81-8772-7ea00af5bbad" containerID="03fab0677cb7639c413a34c1bc771d08ff9f19e02dad31a9f4699d8820d84368" exitCode=0 Oct 31 13:35:10.025877 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:10.025840 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd" event={"ID":"3ff4f1a8-99b6-4d81-8772-7ea00af5bbad","Type":"ContainerDied","Data":"03fab0677cb7639c413a34c1bc771d08ff9f19e02dad31a9f4699d8820d84368"} Oct 31 13:35:10.026047 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:10.025894 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd" event={"ID":"3ff4f1a8-99b6-4d81-8772-7ea00af5bbad","Type":"ContainerDied","Data":"19732f129087bf36f864a08aab9a885ad46edf8a7b94d08ab10f1591a3a0a083"} Oct 31 13:35:10.026047 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:10.025918 6841 scope.go:122] "RemoveContainer" containerID="03fab0677cb7639c413a34c1bc771d08ff9f19e02dad31a9f4699d8820d84368" Oct 31 13:35:10.026212 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:10.026136 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd" Oct 31 13:35:10.042400 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:10.042365 6841 scope.go:122] "RemoveContainer" containerID="03fab0677cb7639c413a34c1bc771d08ff9f19e02dad31a9f4699d8820d84368" Oct 31 13:35:10.042741 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:35:10.042710 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03fab0677cb7639c413a34c1bc771d08ff9f19e02dad31a9f4699d8820d84368\": container with ID starting with 03fab0677cb7639c413a34c1bc771d08ff9f19e02dad31a9f4699d8820d84368 not found: ID does not exist" containerID="03fab0677cb7639c413a34c1bc771d08ff9f19e02dad31a9f4699d8820d84368" Oct 31 13:35:10.042883 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:10.042750 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03fab0677cb7639c413a34c1bc771d08ff9f19e02dad31a9f4699d8820d84368"} err="failed to get container status \"03fab0677cb7639c413a34c1bc771d08ff9f19e02dad31a9f4699d8820d84368\": rpc error: code = NotFound desc = could not find container \"03fab0677cb7639c413a34c1bc771d08ff9f19e02dad31a9f4699d8820d84368\": container with ID starting with 03fab0677cb7639c413a34c1bc771d08ff9f19e02dad31a9f4699d8820d84368 not found: ID does not exist" Oct 31 13:35:10.053680 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:10.053652 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:10.057289 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:10.057264 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd"] Oct 31 13:35:10.065759 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:10.065736 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a39960/test-model-a39960-kserve-8674d5c7c5-r22xd"] Oct 31 13:35:10.837875 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:10.837840 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:11.644113 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:11.644076 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ff4f1a8-99b6-4d81-8772-7ea00af5bbad" path="/var/lib/kubelet/pods/3ff4f1a8-99b6-4d81-8772-7ea00af5bbad/volumes" Oct 31 13:35:11.859397 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:11.859364 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:12.659116 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:12.659081 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:12.910134 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:12.910044 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:13.389153 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:13.389118 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:13.605946 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:13.605887 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:14.296262 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:14.296222 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:15.108519 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:15.108486 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:15.915429 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:15.915386 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:16.688039 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:16.687959 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:17.441258 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:17.441225 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:18.326097 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:18.326059 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:19.069148 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:19.069110 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:19.288500 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:19.288462 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:19.956379 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:19.956345 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:20.045570 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:35:20.045535 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:35:20.560192 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:20.560132 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:20.813408 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:20.813311 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:21.611413 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:21.611378 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:22.338883 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:22.338845 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:23.154333 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:23.154300 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:23.930936 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:23.930900 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:24.846007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:24.836517 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:26.167790 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:26.167737 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:27.040930 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:27.040893 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:27.045170 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:27.045130 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fk2tx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:35:27.045346 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:27.045218 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" podUID="aa7e0a48-9dc5-4735-b1ae-cafd28ff982f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:35:27.361889 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:27.359818 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:28.124997 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:28.124936 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:28.340845 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:28.340810 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:28.790647 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:28.790618 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:29.144807 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:29.144708 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:29.347308 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:29.347268 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:29.970274 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:29.970244 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:30.325377 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:30.325297 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:30.875660 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:30.875629 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:31.632794 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:31.632758 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:31.844858 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:31.844818 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:32.402394 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:32.402352 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:33.101508 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:33.101471 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:33.808893 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:33.808167 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:34.678395 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:34.678355 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:35.201190 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:35.201157 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a13836-bb4868df9-9qxgz"] Oct 31 13:35:35.404454 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:35.404420 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:35.614114 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:35.614028 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:36.373344 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:36.373307 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:37.048527 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.048494 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn"] Oct 31 13:35:37.049462 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.049438 6841 cpu_manager.go:414] "containerMap: removing container" podUID="3ff4f1a8-99b6-4d81-8772-7ea00af5bbad" containerName="main" Oct 31 13:35:37.049462 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.049459 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff4f1a8-99b6-4d81-8772-7ea00af5bbad" containerName="main" Oct 31 13:35:37.049707 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.049561 6841 cpu_manager.go:414] "containerMap: removing container" podUID="41ec82c1-2f15-4a66-8211-330a7a5029cf" containerName="istio-proxy" Oct 31 13:35:37.049707 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.049571 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41ec82c1-2f15-4a66-8211-330a7a5029cf" containerName="istio-proxy" Oct 31 13:35:37.049832 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.049725 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="41ec82c1-2f15-4a66-8211-330a7a5029cf" containerName="istio-proxy" Oct 31 13:35:37.049832 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.049736 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ff4f1a8-99b6-4d81-8772-7ea00af5bbad" containerName="main" Oct 31 13:35:37.063869 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.063834 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" Oct 31 13:35:37.143350 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.143316 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:37.149143 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.149096 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn"] Oct 31 13:35:37.165552 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.165516 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxwdn\" (UID: \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" Oct 31 13:35:37.165699 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.165584 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxwdn\" (UID: \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" Oct 31 13:35:37.165809 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.165781 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxwdn\" (UID: \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" Oct 31 13:35:37.165870 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.165853 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxwdn\" (UID: \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" Oct 31 13:35:37.165960 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.165941 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kww6\" (UniqueName: \"kubernetes.io/projected/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-kube-api-access-8kww6\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxwdn\" (UID: \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" Oct 31 13:35:37.166107 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.166084 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxwdn\" (UID: \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" Oct 31 13:35:37.166213 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.166187 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxwdn\" (UID: \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" Oct 31 13:35:37.166262 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.166246 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxwdn\" (UID: \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" Oct 31 13:35:37.166418 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.166382 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxwdn\" (UID: \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" Oct 31 13:35:37.267891 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.267858 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kww6\" (UniqueName: \"kubernetes.io/projected/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-kube-api-access-8kww6\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxwdn\" (UID: \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" Oct 31 13:35:37.268093 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.267931 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxwdn\" (UID: \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" Oct 31 13:35:37.268093 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.267965 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxwdn\" (UID: \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" Oct 31 13:35:37.268093 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.268007 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxwdn\" (UID: \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" Oct 31 13:35:37.268093 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.268058 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxwdn\" (UID: \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" Oct 31 13:35:37.268308 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.268107 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxwdn\" (UID: \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" Oct 31 13:35:37.268308 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.268147 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxwdn\" (UID: \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" Oct 31 13:35:37.268308 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.268254 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxwdn\" (UID: \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" Oct 31 13:35:37.268485 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.268397 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxwdn\" (UID: \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" Oct 31 13:35:37.268485 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.268408 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxwdn\" (UID: \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" Oct 31 13:35:37.268485 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.268441 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxwdn\" (UID: \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" Oct 31 13:35:37.268653 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.268590 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxwdn\" (UID: \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" Oct 31 13:35:37.268805 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.268779 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxwdn\" (UID: \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" Oct 31 13:35:37.268955 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.268939 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxwdn\" (UID: \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" Oct 31 13:35:37.270468 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.270450 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxwdn\" (UID: \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" Oct 31 13:35:37.270703 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.270685 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxwdn\" (UID: \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" Oct 31 13:35:37.280286 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.280262 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxwdn\" (UID: \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" Oct 31 13:35:37.280502 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.280485 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kww6\" (UniqueName: \"kubernetes.io/projected/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-kube-api-access-8kww6\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxwdn\" (UID: \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" Oct 31 13:35:37.343948 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.343799 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:37.462363 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.462331 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" Oct 31 13:35:37.758532 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:35:37.758502 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac22634_1f31_475f_ae5b_7bb13eb7c6b6.slice/crio-5bc4b313e0a7f6cbad8e4480daaa215d8aec4442d154db42e2ed72e94f0f2b06 WatchSource:0}: Error finding container 5bc4b313e0a7f6cbad8e4480daaa215d8aec4442d154db42e2ed72e94f0f2b06: Status 404 returned error can't find the container with id 5bc4b313e0a7f6cbad8e4480daaa215d8aec4442d154db42e2ed72e94f0f2b06 Oct 31 13:35:37.758744 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.758723 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn"] Oct 31 13:35:37.761842 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.761804 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:35:37.761960 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.761910 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:35:37.761960 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:37.761948 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:35:38.111538 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:38.111455 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:38.364777 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:38.364694 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" event={"ID":"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6","Type":"ContainerStarted","Data":"c3e5a5502f6547128ea04d6f9cc100477289bd0147dbf50dc383ade051a3dac6"} Oct 31 13:35:38.364777 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:38.364759 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" event={"ID":"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6","Type":"ContainerStarted","Data":"5bc4b313e0a7f6cbad8e4480daaa215d8aec4442d154db42e2ed72e94f0f2b06"} Oct 31 13:35:38.393404 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:38.393349 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" podStartSLOduration=1.3933340570000001 podStartE2EDuration="1.393334057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:35:38.390490943 +0000 UTC m=+3784.873613843" watchObservedRunningTime="2026-10-31 13:35:38.393334057 +0000 UTC m=+3784.876456957" Oct 31 13:35:38.462582 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:38.462548 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" Oct 31 13:35:38.464560 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:38.464532 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-sxwdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Oct 31 13:35:38.464698 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:38.464592 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" podUID="6ac22634-1f31-475f-ae5b-7bb13eb7c6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Oct 31 13:35:39.287233 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:39.287027 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:39.466803 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:39.466760 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-sxwdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Oct 31 13:35:39.467295 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:39.466830 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" podUID="6ac22634-1f31-475f-ae5b-7bb13eb7c6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Oct 31 13:35:39.564855 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:39.564768 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:39.770503 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:39.770441 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:40.195203 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:40.195168 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:40.201930 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:40.201900 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-a13836-bb4868df9-9qxgz" podUID="61c9f172-a17d-4531-8626-6bd1433a9a39" containerName="payload-processing" containerID="cri-o://1652a9dab04fee18b97355cbd9745471cfcdba9d1e880d91fe93575c3b5d8d4c" gracePeriod=25 Oct 31 13:35:40.416607 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:40.416512 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:40.466619 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:40.466494 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-sxwdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Oct 31 13:35:40.466619 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:40.466592 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" podUID="6ac22634-1f31-475f-ae5b-7bb13eb7c6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Oct 31 13:35:40.480841 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:40.480803 6841 generic.go:356] "Generic (PLEG): container finished" podID="61c9f172-a17d-4531-8626-6bd1433a9a39" containerID="1652a9dab04fee18b97355cbd9745471cfcdba9d1e880d91fe93575c3b5d8d4c" exitCode=0 Oct 31 13:35:40.481169 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:40.481126 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a13836-bb4868df9-9qxgz" event={"ID":"61c9f172-a17d-4531-8626-6bd1433a9a39","Type":"ContainerDied","Data":"1652a9dab04fee18b97355cbd9745471cfcdba9d1e880d91fe93575c3b5d8d4c"} Oct 31 13:35:40.615575 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:40.615542 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:40.619339 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:40.619318 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a13836-bb4868df9-9qxgz" Oct 31 13:35:40.725287 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:40.725211 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61c9f172-a17d-4531-8626-6bd1433a9a39-kube-api-access-9d58h\" (UniqueName: \"kubernetes.io/projected/61c9f172-a17d-4531-8626-6bd1433a9a39-kube-api-access-9d58h\") pod \"61c9f172-a17d-4531-8626-6bd1433a9a39\" (UID: \"61c9f172-a17d-4531-8626-6bd1433a9a39\") " Oct 31 13:35:40.725287 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:40.725248 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61c9f172-a17d-4531-8626-6bd1433a9a39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61c9f172-a17d-4531-8626-6bd1433a9a39-plugins-config-volume\") pod \"61c9f172-a17d-4531-8626-6bd1433a9a39\" (UID: \"61c9f172-a17d-4531-8626-6bd1433a9a39\") " Oct 31 13:35:40.725737 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:40.725704 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61c9f172-a17d-4531-8626-6bd1433a9a39-plugins-config-volume" pod "61c9f172-a17d-4531-8626-6bd1433a9a39" (UID: "61c9f172-a17d-4531-8626-6bd1433a9a39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:35:40.726074 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:40.726053 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61c9f172-a17d-4531-8626-6bd1433a9a39-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:40.727646 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:40.727607 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61c9f172-a17d-4531-8626-6bd1433a9a39-kube-api-access-9d58h" pod "61c9f172-a17d-4531-8626-6bd1433a9a39" (UID: "61c9f172-a17d-4531-8626-6bd1433a9a39"). InnerVolumeSpecName "kube-api-access-9d58h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:35:40.827216 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:40.827177 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d58h\" (UniqueName: \"kubernetes.io/projected/61c9f172-a17d-4531-8626-6bd1433a9a39-kube-api-access-9d58h\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:40.882660 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:40.882627 6841 scope.go:122] "RemoveContainer" containerID="1652a9dab04fee18b97355cbd9745471cfcdba9d1e880d91fe93575c3b5d8d4c" Oct 31 13:35:41.417072 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:41.417035 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:41.464006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:41.463347 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-sxwdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Oct 31 13:35:41.464006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:41.463421 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" podUID="6ac22634-1f31-475f-ae5b-7bb13eb7c6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Oct 31 13:35:41.489673 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:41.489636 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a13836-bb4868df9-9qxgz" event={"ID":"61c9f172-a17d-4531-8626-6bd1433a9a39","Type":"ContainerDied","Data":"a2f6c3e7d3b75eb2914f807cbe3b6c6a752e1c961f33f194465a80c17e4fc554"} Oct 31 13:35:41.490084 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:41.489723 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a13836-bb4868df9-9qxgz" Oct 31 13:35:41.528510 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:41.528476 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a13836-bb4868df9-9qxgz"] Oct 31 13:35:41.537736 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:41.537710 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a13836-bb4868df9-9qxgz"] Oct 31 13:35:41.652905 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:41.652869 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61c9f172-a17d-4531-8626-6bd1433a9a39" path="/var/lib/kubelet/pods/61c9f172-a17d-4531-8626-6bd1433a9a39/volumes" Oct 31 13:35:42.420487 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:42.420455 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:42.475706 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:42.475669 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-sxwdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Oct 31 13:35:42.476084 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:42.476023 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" podUID="6ac22634-1f31-475f-ae5b-7bb13eb7c6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Oct 31 13:35:42.929815 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:42.929773 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:43.361192 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:43.361154 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:43.464131 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:43.463665 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-sxwdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Oct 31 13:35:43.464131 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:43.463726 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" podUID="6ac22634-1f31-475f-ae5b-7bb13eb7c6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Oct 31 13:35:44.467446 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:44.467406 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-sxwdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Oct 31 13:35:44.468061 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:44.468026 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" podUID="6ac22634-1f31-475f-ae5b-7bb13eb7c6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Oct 31 13:35:44.506379 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:44.506329 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:45.479260 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:45.479217 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-sxwdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Oct 31 13:35:45.479919 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:45.479882 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" podUID="6ac22634-1f31-475f-ae5b-7bb13eb7c6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Oct 31 13:35:45.539549 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:45.539510 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:46.325961 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:46.325920 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:46.663042 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:46.662921 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" Oct 31 13:35:46.663809 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:46.663786 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" Oct 31 13:35:46.665072 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:46.665048 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" Oct 31 13:35:47.245623 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:47.245579 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:47.937867 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:47.937829 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:48.722144 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:48.722106 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:49.400457 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:49.400419 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:50.035878 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:35:50.035840 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:35:50.340894 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:50.340860 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:51.089013 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:51.086893 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:51.897648 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:51.897614 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:52.603536 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:52.603499 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:53.330894 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:53.330859 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:54.117345 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:54.117310 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:54.828665 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:54.828626 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:55.611489 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:55.611450 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:56.348268 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:56.348162 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:57.124788 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:57.124736 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:57.845566 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:57.845534 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:58.718537 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:58.718480 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:35:59.438957 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:35:59.438922 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:00.213215 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:00.213178 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:01.233811 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:01.233779 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:01.844955 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:01.844884 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:02.126608 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:02.126525 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:03.102340 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:03.102302 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:03.922010 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:03.919319 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:04.594130 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:04.594095 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:05.325118 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:05.325081 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:06.120960 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:06.120929 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:06.827403 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:06.827365 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:07.619382 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:07.619343 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:08.521004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:08.519234 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:09.314659 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:09.314624 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:09.592972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:09.592879 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:10.147023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:10.146966 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:10.605448 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:10.605414 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:10.865170 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:10.865085 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:11.547365 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:11.547333 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:11.802186 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:11.802106 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:12.192991 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:12.192890 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:12.417037 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:12.417002 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:12.635331 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:12.635226 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:13.266820 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:13.266770 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:13.561090 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:13.560974 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:13.790312 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:13.790280 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:14.246505 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:14.246456 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:15.426797 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:15.426756 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:16.306482 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:16.306442 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:16.515835 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:16.515797 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:17.291080 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:17.291037 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:17.619599 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:17.619508 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:18.007965 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:18.007929 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:18.326497 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:18.326406 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:19.133868 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:19.133828 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:19.918897 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:19.918863 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:20.035656 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:36:20.035623 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:36:20.703761 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:20.703724 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:21.631421 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:21.631379 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:22.387913 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:22.387877 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:23.179820 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:23.179787 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:23.940664 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:23.940578 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:24.317352 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:24.317303 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:24.716590 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:24.716555 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:25.166909 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:25.166819 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:25.407225 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:25.407187 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:25.620713 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:25.620677 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:25.873360 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:25.873254 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:26.616608 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:26.616571 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:26.866855 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:26.866759 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:27.530536 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:27.530501 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:28.440754 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:28.440721 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:29.172761 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:29.172699 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:30.052624 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:30.052578 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:31.036700 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:31.036661 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:31.793448 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:31.793411 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:32.416714 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:32.416671 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:32.618566 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:32.618527 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:33.228961 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:33.228926 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:33.993375 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:33.993338 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:34.756564 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:34.756532 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:35.515302 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:35.515264 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:36.219239 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:36.219171 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:36.907751 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:36.907707 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:37.742564 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:37.742529 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:38.671616 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:38.671579 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:38.905902 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:38.905869 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:39.339506 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:39.339471 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:40.026265 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:40.026222 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:40.234760 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:40.234729 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:40.450892 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:40.450863 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:41.047595 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:41.047562 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:41.701893 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:41.701859 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:42.585500 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:42.585449 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:43.473376 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:43.473345 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:44.213812 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:44.213758 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:44.217389 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:44.217361 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g"] Oct 31 13:36:44.444265 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:44.444229 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:45.096610 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:45.096573 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:45.778112 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:45.778077 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:46.042785 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:46.042690 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:47.296881 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:47.296847 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:48.196923 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:48.196879 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:48.922500 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:48.922421 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:49.676588 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:49.676552 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:50.036857 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:36:50.036819 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:36:50.519860 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:50.519826 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:51.576006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:51.572496 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:52.796962 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:52.796912 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:53.295001 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:53.294574 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:54.069073 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:54.068954 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:55.104868 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:55.104825 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:55.319835 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:55.319798 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x"] Oct 31 13:36:55.450660 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:55.450625 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:55.939398 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:55.939282 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:56.243143 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:56.243106 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:57.042047 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:57.042013 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:57.811197 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:57.811155 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:36:59.242923 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:59.242874 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g" podUID="109fb6e5-38ef-4030-9209-cda454c140ec" containerName="main" containerID="cri-o://ccb6c5a3c4feddf4bbc9fe85326be19eea7846b188f31543b081678461e8346c" gracePeriod=45 Oct 31 13:36:59.378309 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:59.378280 6841 generic.go:356] "Generic (PLEG): container finished" podID="109fb6e5-38ef-4030-9209-cda454c140ec" containerID="ccb6c5a3c4feddf4bbc9fe85326be19eea7846b188f31543b081678461e8346c" exitCode=0 Oct 31 13:36:59.378462 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:59.378343 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g" event={"ID":"109fb6e5-38ef-4030-9209-cda454c140ec","Type":"ContainerDied","Data":"ccb6c5a3c4feddf4bbc9fe85326be19eea7846b188f31543b081678461e8346c"} Oct 31 13:36:59.539448 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:59.539422 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g" Oct 31 13:36:59.654251 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:59.654225 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/109fb6e5-38ef-4030-9209-cda454c140ec-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/109fb6e5-38ef-4030-9209-cda454c140ec-model-cache\") pod \"109fb6e5-38ef-4030-9209-cda454c140ec\" (UID: \"109fb6e5-38ef-4030-9209-cda454c140ec\") " Oct 31 13:36:59.654439 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:59.654307 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/109fb6e5-38ef-4030-9209-cda454c140ec-tls-certs\" (UniqueName: \"kubernetes.io/secret/109fb6e5-38ef-4030-9209-cda454c140ec-tls-certs\") pod \"109fb6e5-38ef-4030-9209-cda454c140ec\" (UID: \"109fb6e5-38ef-4030-9209-cda454c140ec\") " Oct 31 13:36:59.654439 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:59.654345 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/109fb6e5-38ef-4030-9209-cda454c140ec-home\" (UniqueName: \"kubernetes.io/empty-dir/109fb6e5-38ef-4030-9209-cda454c140ec-home\") pod \"109fb6e5-38ef-4030-9209-cda454c140ec\" (UID: \"109fb6e5-38ef-4030-9209-cda454c140ec\") " Oct 31 13:36:59.654439 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:59.654426 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/109fb6e5-38ef-4030-9209-cda454c140ec-kube-api-access-5qbpp\" (UniqueName: \"kubernetes.io/projected/109fb6e5-38ef-4030-9209-cda454c140ec-kube-api-access-5qbpp\") pod \"109fb6e5-38ef-4030-9209-cda454c140ec\" (UID: \"109fb6e5-38ef-4030-9209-cda454c140ec\") " Oct 31 13:36:59.654629 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:59.654485 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/109fb6e5-38ef-4030-9209-cda454c140ec-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/109fb6e5-38ef-4030-9209-cda454c140ec-tmp-dir\") pod \"109fb6e5-38ef-4030-9209-cda454c140ec\" (UID: \"109fb6e5-38ef-4030-9209-cda454c140ec\") " Oct 31 13:36:59.654629 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:59.654480 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/109fb6e5-38ef-4030-9209-cda454c140ec-model-cache" pod "109fb6e5-38ef-4030-9209-cda454c140ec" (UID: "109fb6e5-38ef-4030-9209-cda454c140ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:36:59.654748 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:59.654657 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/109fb6e5-38ef-4030-9209-cda454c140ec-home" pod "109fb6e5-38ef-4030-9209-cda454c140ec" (UID: "109fb6e5-38ef-4030-9209-cda454c140ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:36:59.654748 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:59.654682 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/109fb6e5-38ef-4030-9209-cda454c140ec-dshm\" (UniqueName: \"kubernetes.io/empty-dir/109fb6e5-38ef-4030-9209-cda454c140ec-dshm\") pod \"109fb6e5-38ef-4030-9209-cda454c140ec\" (UID: \"109fb6e5-38ef-4030-9209-cda454c140ec\") " Oct 31 13:36:59.654748 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:59.654731 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/109fb6e5-38ef-4030-9209-cda454c140ec-tmp-dir" pod "109fb6e5-38ef-4030-9209-cda454c140ec" (UID: "109fb6e5-38ef-4030-9209-cda454c140ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:36:59.655320 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:59.655298 6841 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/109fb6e5-38ef-4030-9209-cda454c140ec-model-cache\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:59.655320 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:59.655319 6841 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/109fb6e5-38ef-4030-9209-cda454c140ec-home\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:59.655501 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:59.655329 6841 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/109fb6e5-38ef-4030-9209-cda454c140ec-tmp-dir\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:59.656670 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:59.656639 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/109fb6e5-38ef-4030-9209-cda454c140ec-tls-certs" pod "109fb6e5-38ef-4030-9209-cda454c140ec" (UID: "109fb6e5-38ef-4030-9209-cda454c140ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:36:59.656771 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:59.656675 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/109fb6e5-38ef-4030-9209-cda454c140ec-dshm" pod "109fb6e5-38ef-4030-9209-cda454c140ec" (UID: "109fb6e5-38ef-4030-9209-cda454c140ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:36:59.656771 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:59.656701 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/109fb6e5-38ef-4030-9209-cda454c140ec-kube-api-access-5qbpp" pod "109fb6e5-38ef-4030-9209-cda454c140ec" (UID: "109fb6e5-38ef-4030-9209-cda454c140ec"). InnerVolumeSpecName "kube-api-access-5qbpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:36:59.757479 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:59.757384 6841 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/109fb6e5-38ef-4030-9209-cda454c140ec-tls-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:59.757479 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:59.757408 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qbpp\" (UniqueName: \"kubernetes.io/projected/109fb6e5-38ef-4030-9209-cda454c140ec-kube-api-access-5qbpp\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:59.757479 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:59.757419 6841 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/109fb6e5-38ef-4030-9209-cda454c140ec-dshm\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:59.989777 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:36:59.989746 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:00.387226 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:00.387203 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g" Oct 31 13:37:00.387678 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:00.387246 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g" event={"ID":"109fb6e5-38ef-4030-9209-cda454c140ec","Type":"ContainerDied","Data":"54452efb4994340b976e986882d1ba707c9295d71d65802f405407ba1bae49bc"} Oct 31 13:37:00.387678 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:00.387287 6841 scope.go:122] "RemoveContainer" containerID="ccb6c5a3c4feddf4bbc9fe85326be19eea7846b188f31543b081678461e8346c" Oct 31 13:37:00.427536 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:00.427509 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g"] Oct 31 13:37:00.439743 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:00.439715 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-691f1e/test-model-691f1e-kserve-f8575cb55-dcg2g"] Oct 31 13:37:00.982918 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:00.982881 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:01.644956 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:01.644908 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="109fb6e5-38ef-4030-9209-cda454c140ec" path="/var/lib/kubelet/pods/109fb6e5-38ef-4030-9209-cda454c140ec/volumes" Oct 31 13:37:01.721484 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:01.721442 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:02.574187 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:02.574145 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:02.797003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:02.787526 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:03.080047 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:03.080015 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:03.824853 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:03.824813 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:04.727704 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:04.727673 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:05.507825 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:05.507778 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:06.230849 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:06.230812 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:06.823632 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:06.823593 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:07.177047 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:07.176951 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:08.295798 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:08.295760 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:09.014022 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:09.013974 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:09.061252 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:09.061217 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k"] Oct 31 13:37:09.061457 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:09.061436 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" podUID="68406df2-8453-4762-9398-4019221157aa" containerName="istio-proxy" containerID="cri-o://0caecc427a63417b3ec0852e445fbd548b4baa6ff3b985a71d639814be844a17" gracePeriod=30 Oct 31 13:37:09.827028 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:09.826991 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:10.062248 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:10.062215 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:10.341857 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:10.341815 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x" podUID="1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939" containerName="main" containerID="cri-o://2560e79b2ef03481f4cc62ef0caaff2aa107ae6cad863161993219dccab661c7" gracePeriod=45 Oct 31 13:37:10.382386 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:10.382357 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:10.479303 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:10.479269 6841 generic.go:356] "Generic (PLEG): container finished" podID="1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939" containerID="2560e79b2ef03481f4cc62ef0caaff2aa107ae6cad863161993219dccab661c7" exitCode=0 Oct 31 13:37:10.479471 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:10.479340 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x" event={"ID":"1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939","Type":"ContainerDied","Data":"2560e79b2ef03481f4cc62ef0caaff2aa107ae6cad863161993219dccab661c7"} Oct 31 13:37:10.599594 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:10.599511 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz"] Oct 31 13:37:10.599793 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:10.599758 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" podUID="0e6784e6-2aeb-4b2d-b1f8-97f903337e82" containerName="istio-proxy" containerID="cri-o://58211cd76607bd49c2f389efdbb9a862de0151b95845dea76c651d897c26eb55" gracePeriod=30 Oct 31 13:37:10.664439 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:10.664412 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x" Oct 31 13:37:10.783239 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:10.782549 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:10.909581 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:10.909490 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-model-cache\") pod \"1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939\" (UID: \"1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939\") " Oct 31 13:37:10.910241 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:10.910212 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-kube-api-access-rnh5h\" (UniqueName: \"kubernetes.io/projected/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-kube-api-access-rnh5h\") pod \"1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939\" (UID: \"1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939\") " Oct 31 13:37:10.910456 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:10.910441 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-home\" (UniqueName: \"kubernetes.io/empty-dir/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-home\") pod \"1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939\" (UID: \"1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939\") " Oct 31 13:37:10.910565 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:10.910554 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-tls-certs\") pod \"1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939\" (UID: \"1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939\") " Oct 31 13:37:10.910776 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:10.910761 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-dshm\") pod \"1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939\" (UID: \"1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939\") " Oct 31 13:37:10.911850 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:10.911823 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-tmp-dir\") pod \"1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939\" (UID: \"1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939\") " Oct 31 13:37:10.916080 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:10.916053 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-model-cache" pod "1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939" (UID: "1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:10.916775 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:10.916752 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-tmp-dir" pod "1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939" (UID: "1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:10.917167 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:10.917145 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-home" pod "1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939" (UID: "1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:10.946118 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:10.940848 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-tls-certs" pod "1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939" (UID: "1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:37:10.982740 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:10.982361 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-dshm" pod "1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939" (UID: "1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:10.982740 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:10.982514 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-kube-api-access-rnh5h" pod "1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939" (UID: "1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939"). InnerVolumeSpecName "kube-api-access-rnh5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:11.045144 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:11.036183 6841 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-tmp-dir\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:11.045144 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:11.036216 6841 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-model-cache\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:11.045144 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:11.036231 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnh5h\" (UniqueName: \"kubernetes.io/projected/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-kube-api-access-rnh5h\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:11.045144 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:11.036246 6841 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-home\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:11.045144 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:11.036260 6841 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-tls-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:11.045144 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:11.036273 6841 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939-dshm\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:11.079005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:11.077905 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:11.234389 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:11.234355 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg"] Oct 31 13:37:11.234578 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:11.234556 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" podUID="bbc74bb4-ddf5-4a5f-836d-e206e6ba6020" containerName="istio-proxy" containerID="cri-o://594b02f4d01b830cf8211dafef85ac2010c3e9dc37a8a37645934bc8c8f7d7c1" gracePeriod=30 Oct 31 13:37:11.489795 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:11.489713 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x" Oct 31 13:37:11.489795 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:11.489746 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x" event={"ID":"1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939","Type":"ContainerDied","Data":"1d0b65542c8e38b7e798be4524ac7534ede3cae09ee3378703d1107f8681a49a"} Oct 31 13:37:11.490050 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:11.489799 6841 scope.go:122] "RemoveContainer" containerID="2560e79b2ef03481f4cc62ef0caaff2aa107ae6cad863161993219dccab661c7" Oct 31 13:37:11.529666 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:11.529636 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x"] Oct 31 13:37:11.535183 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:11.535154 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-3cddb3/test-model-3cddb3-kserve-75bf994797-jlw6x"] Oct 31 13:37:11.578355 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:11.578329 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:11.645508 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:11.645477 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939" path="/var/lib/kubelet/pods/1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939/volumes" Oct 31 13:37:11.667939 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:11.667911 6841 patch_prober.go:29] interesting pod/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:37:11.668091 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:11.667960 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" podUID="0e6784e6-2aeb-4b2d-b1f8-97f903337e82" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:37:11.790734 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:11.790643 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:12.339390 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:12.339357 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:12.508113 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:12.504713 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:37:12.515069 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:12.508392 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" podUID="68406df2-8453-4762-9398-4019221157aa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:37:12.564888 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:12.564850 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:13.220866 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:13.220831 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:13.593465 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:13.593384 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:13.806265 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:13.806226 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:14.021610 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.021578 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:14.371025 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.370951 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" Oct 31 13:37:14.427451 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.427418 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68406df2-8453-4762-9398-4019221157aa-kube-api-access-zdvrf\" (UniqueName: \"kubernetes.io/projected/68406df2-8453-4762-9398-4019221157aa-kube-api-access-zdvrf\") pod \"68406df2-8453-4762-9398-4019221157aa\" (UID: \"68406df2-8453-4762-9398-4019221157aa\") " Oct 31 13:37:14.427641 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.427497 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68406df2-8453-4762-9398-4019221157aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68406df2-8453-4762-9398-4019221157aa-istio-podinfo\") pod \"68406df2-8453-4762-9398-4019221157aa\" (UID: \"68406df2-8453-4762-9398-4019221157aa\") " Oct 31 13:37:14.427641 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.427553 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68406df2-8453-4762-9398-4019221157aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68406df2-8453-4762-9398-4019221157aa-istio-data\") pod \"68406df2-8453-4762-9398-4019221157aa\" (UID: \"68406df2-8453-4762-9398-4019221157aa\") " Oct 31 13:37:14.427641 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.427615 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68406df2-8453-4762-9398-4019221157aa-istio-token\" (UniqueName: \"kubernetes.io/projected/68406df2-8453-4762-9398-4019221157aa-istio-token\") pod \"68406df2-8453-4762-9398-4019221157aa\" (UID: \"68406df2-8453-4762-9398-4019221157aa\") " Oct 31 13:37:14.427844 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.427658 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68406df2-8453-4762-9398-4019221157aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68406df2-8453-4762-9398-4019221157aa-istio-envoy\") pod \"68406df2-8453-4762-9398-4019221157aa\" (UID: \"68406df2-8453-4762-9398-4019221157aa\") " Oct 31 13:37:14.427844 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.427824 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68406df2-8453-4762-9398-4019221157aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68406df2-8453-4762-9398-4019221157aa-istiod-ca-cert\") pod \"68406df2-8453-4762-9398-4019221157aa\" (UID: \"68406df2-8453-4762-9398-4019221157aa\") " Oct 31 13:37:14.427966 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.427952 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68406df2-8453-4762-9398-4019221157aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68406df2-8453-4762-9398-4019221157aa-workload-certs\") pod \"68406df2-8453-4762-9398-4019221157aa\" (UID: \"68406df2-8453-4762-9398-4019221157aa\") " Oct 31 13:37:14.428040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.428014 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68406df2-8453-4762-9398-4019221157aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68406df2-8453-4762-9398-4019221157aa-credential-socket\") pod \"68406df2-8453-4762-9398-4019221157aa\" (UID: \"68406df2-8453-4762-9398-4019221157aa\") " Oct 31 13:37:14.428110 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.428065 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68406df2-8453-4762-9398-4019221157aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68406df2-8453-4762-9398-4019221157aa-workload-socket\") pod \"68406df2-8453-4762-9398-4019221157aa\" (UID: \"68406df2-8453-4762-9398-4019221157aa\") " Oct 31 13:37:14.429154 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.428480 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68406df2-8453-4762-9398-4019221157aa-istiod-ca-cert" pod "68406df2-8453-4762-9398-4019221157aa" (UID: "68406df2-8453-4762-9398-4019221157aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:37:14.429154 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.428495 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68406df2-8453-4762-9398-4019221157aa-istio-data" pod "68406df2-8453-4762-9398-4019221157aa" (UID: "68406df2-8453-4762-9398-4019221157aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:14.429154 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.428727 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68406df2-8453-4762-9398-4019221157aa-workload-socket" pod "68406df2-8453-4762-9398-4019221157aa" (UID: "68406df2-8453-4762-9398-4019221157aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:14.429154 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.428867 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68406df2-8453-4762-9398-4019221157aa-workload-certs" pod "68406df2-8453-4762-9398-4019221157aa" (UID: "68406df2-8453-4762-9398-4019221157aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:14.429154 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.429080 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68406df2-8453-4762-9398-4019221157aa-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:14.429154 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.429102 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68406df2-8453-4762-9398-4019221157aa-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:14.429154 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.429117 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68406df2-8453-4762-9398-4019221157aa-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:14.429154 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.429120 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68406df2-8453-4762-9398-4019221157aa-credential-socket" pod "68406df2-8453-4762-9398-4019221157aa" (UID: "68406df2-8453-4762-9398-4019221157aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:14.429154 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.429131 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68406df2-8453-4762-9398-4019221157aa-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:14.430368 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.430341 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68406df2-8453-4762-9398-4019221157aa-kube-api-access-zdvrf" pod "68406df2-8453-4762-9398-4019221157aa" (UID: "68406df2-8453-4762-9398-4019221157aa"). InnerVolumeSpecName "kube-api-access-zdvrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:14.431109 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.431077 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68406df2-8453-4762-9398-4019221157aa-istio-podinfo" pod "68406df2-8453-4762-9398-4019221157aa" (UID: "68406df2-8453-4762-9398-4019221157aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:37:14.431224 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.431204 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68406df2-8453-4762-9398-4019221157aa-istio-token" pod "68406df2-8453-4762-9398-4019221157aa" (UID: "68406df2-8453-4762-9398-4019221157aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:14.431299 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.431278 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68406df2-8453-4762-9398-4019221157aa-istio-envoy" pod "68406df2-8453-4762-9398-4019221157aa" (UID: "68406df2-8453-4762-9398-4019221157aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:14.530377 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.530340 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdvrf\" (UniqueName: \"kubernetes.io/projected/68406df2-8453-4762-9398-4019221157aa-kube-api-access-zdvrf\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:14.530377 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.530372 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68406df2-8453-4762-9398-4019221157aa-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:14.530377 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.530383 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68406df2-8453-4762-9398-4019221157aa-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:14.530633 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.530392 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68406df2-8453-4762-9398-4019221157aa-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:14.530633 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.530401 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68406df2-8453-4762-9398-4019221157aa-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:14.563876 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.563847 6841 generic.go:356] "Generic (PLEG): container finished" podID="68406df2-8453-4762-9398-4019221157aa" containerID="0caecc427a63417b3ec0852e445fbd548b4baa6ff3b985a71d639814be844a17" exitCode=0 Oct 31 13:37:14.564074 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.563899 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" Oct 31 13:37:14.564074 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.563910 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" event={"ID":"68406df2-8453-4762-9398-4019221157aa","Type":"ContainerDied","Data":"0caecc427a63417b3ec0852e445fbd548b4baa6ff3b985a71d639814be844a17"} Oct 31 13:37:14.564074 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.563963 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k" event={"ID":"68406df2-8453-4762-9398-4019221157aa","Type":"ContainerDied","Data":"69e12c96397c6077e2777c163195d92df1a8305d72a5e373f0f47f480d311ad4"} Oct 31 13:37:14.564074 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.564008 6841 scope.go:122] "RemoveContainer" containerID="0caecc427a63417b3ec0852e445fbd548b4baa6ff3b985a71d639814be844a17" Oct 31 13:37:14.585400 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.580606 6841 scope.go:122] "RemoveContainer" containerID="0caecc427a63417b3ec0852e445fbd548b4baa6ff3b985a71d639814be844a17" Oct 31 13:37:14.585400 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:37:14.581488 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0caecc427a63417b3ec0852e445fbd548b4baa6ff3b985a71d639814be844a17\": container with ID starting with 0caecc427a63417b3ec0852e445fbd548b4baa6ff3b985a71d639814be844a17 not found: ID does not exist" containerID="0caecc427a63417b3ec0852e445fbd548b4baa6ff3b985a71d639814be844a17" Oct 31 13:37:14.585400 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.581523 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0caecc427a63417b3ec0852e445fbd548b4baa6ff3b985a71d639814be844a17"} err="failed to get container status \"0caecc427a63417b3ec0852e445fbd548b4baa6ff3b985a71d639814be844a17\": rpc error: code = NotFound desc = could not find container \"0caecc427a63417b3ec0852e445fbd548b4baa6ff3b985a71d639814be844a17\": container with ID starting with 0caecc427a63417b3ec0852e445fbd548b4baa6ff3b985a71d639814be844a17 not found: ID does not exist" Oct 31 13:37:14.598441 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.598414 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k"] Oct 31 13:37:14.602693 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.602666 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-k266k"] Oct 31 13:37:14.722929 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:14.722890 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:15.628227 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:15.628170 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:15.645012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:15.644965 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68406df2-8453-4762-9398-4019221157aa" path="/var/lib/kubelet/pods/68406df2-8453-4762-9398-4019221157aa/volumes" Oct 31 13:37:15.882520 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:15.882464 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" Oct 31 13:37:15.959014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:15.958966 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-istio-data\") pod \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\" (UID: \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\") " Oct 31 13:37:15.959204 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:15.959039 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-kube-api-access-dxcgh\" (UniqueName: \"kubernetes.io/projected/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-kube-api-access-dxcgh\") pod \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\" (UID: \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\") " Oct 31 13:37:15.959279 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:15.959229 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-istio-token\" (UniqueName: \"kubernetes.io/projected/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-istio-token\") pod \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\" (UID: \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\") " Oct 31 13:37:15.959279 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:15.959265 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-workload-certs\") pod \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\" (UID: \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\") " Oct 31 13:37:15.959393 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:15.959303 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-istio-data" pod "0e6784e6-2aeb-4b2d-b1f8-97f903337e82" (UID: "0e6784e6-2aeb-4b2d-b1f8-97f903337e82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:15.959461 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:15.959430 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-istio-envoy\") pod \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\" (UID: \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\") " Oct 31 13:37:15.959570 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:15.959548 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-istio-podinfo\") pod \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\" (UID: \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\") " Oct 31 13:37:15.959667 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:15.959654 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-credential-socket\") pod \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\" (UID: \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\") " Oct 31 13:37:15.961602 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:15.959548 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-workload-certs" pod "0e6784e6-2aeb-4b2d-b1f8-97f903337e82" (UID: "0e6784e6-2aeb-4b2d-b1f8-97f903337e82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:15.961602 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:15.959745 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-istiod-ca-cert\") pod \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\" (UID: \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\") " Oct 31 13:37:15.961602 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:15.959893 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-workload-socket\") pod \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\" (UID: \"0e6784e6-2aeb-4b2d-b1f8-97f903337e82\") " Oct 31 13:37:15.961602 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:15.960530 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-istiod-ca-cert" pod "0e6784e6-2aeb-4b2d-b1f8-97f903337e82" (UID: "0e6784e6-2aeb-4b2d-b1f8-97f903337e82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:37:15.961602 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:15.960736 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-workload-socket" pod "0e6784e6-2aeb-4b2d-b1f8-97f903337e82" (UID: "0e6784e6-2aeb-4b2d-b1f8-97f903337e82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:15.961602 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:15.960963 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:15.961602 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:15.961006 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:15.961602 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:15.961022 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:15.961602 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:15.961036 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:15.961602 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:15.961540 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-credential-socket" pod "0e6784e6-2aeb-4b2d-b1f8-97f903337e82" (UID: "0e6784e6-2aeb-4b2d-b1f8-97f903337e82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:15.962148 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:15.961880 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-istio-token" pod "0e6784e6-2aeb-4b2d-b1f8-97f903337e82" (UID: "0e6784e6-2aeb-4b2d-b1f8-97f903337e82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:15.962148 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:15.962035 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-istio-podinfo" pod "0e6784e6-2aeb-4b2d-b1f8-97f903337e82" (UID: "0e6784e6-2aeb-4b2d-b1f8-97f903337e82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:37:15.962148 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:15.962129 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-kube-api-access-dxcgh" pod "0e6784e6-2aeb-4b2d-b1f8-97f903337e82" (UID: "0e6784e6-2aeb-4b2d-b1f8-97f903337e82"). InnerVolumeSpecName "kube-api-access-dxcgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:15.963225 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:15.963200 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-istio-envoy" pod "0e6784e6-2aeb-4b2d-b1f8-97f903337e82" (UID: "0e6784e6-2aeb-4b2d-b1f8-97f903337e82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:16.062648 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.062608 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxcgh\" (UniqueName: \"kubernetes.io/projected/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-kube-api-access-dxcgh\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:16.062648 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.062640 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:16.062648 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.062651 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:16.062887 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.062659 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:16.062887 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.062669 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6784e6-2aeb-4b2d-b1f8-97f903337e82-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:16.408005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.407698 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:16.549224 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.549196 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" Oct 31 13:37:16.586255 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.586208 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-workload-socket\") pod \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\" (UID: \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\") " Oct 31 13:37:16.586454 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.586273 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-workload-certs\") pod \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\" (UID: \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\") " Oct 31 13:37:16.586454 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.586317 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-istiod-ca-cert\") pod \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\" (UID: \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\") " Oct 31 13:37:16.586454 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.586377 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-kube-api-access-nh874\" (UniqueName: \"kubernetes.io/projected/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-kube-api-access-nh874\") pod \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\" (UID: \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\") " Oct 31 13:37:16.586454 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.586417 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-istio-envoy\") pod \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\" (UID: \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\") " Oct 31 13:37:16.586675 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.586494 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-workload-socket" pod "bbc74bb4-ddf5-4a5f-836d-e206e6ba6020" (UID: "bbc74bb4-ddf5-4a5f-836d-e206e6ba6020"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:16.586675 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.586548 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-credential-socket\") pod \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\" (UID: \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\") " Oct 31 13:37:16.586799 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.586773 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-istiod-ca-cert" pod "bbc74bb4-ddf5-4a5f-836d-e206e6ba6020" (UID: "bbc74bb4-ddf5-4a5f-836d-e206e6ba6020"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:37:16.586952 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.586817 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-istio-data\") pod \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\" (UID: \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\") " Oct 31 13:37:16.586952 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.586862 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-istio-token\" (UniqueName: \"kubernetes.io/projected/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-istio-token\") pod \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\" (UID: \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\") " Oct 31 13:37:16.586952 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.586909 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-credential-socket" pod "bbc74bb4-ddf5-4a5f-836d-e206e6ba6020" (UID: "bbc74bb4-ddf5-4a5f-836d-e206e6ba6020"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:16.586952 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.586920 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-istio-podinfo\") pod \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\" (UID: \"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020\") " Oct 31 13:37:16.587237 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.587091 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-istio-data" pod "bbc74bb4-ddf5-4a5f-836d-e206e6ba6020" (UID: "bbc74bb4-ddf5-4a5f-836d-e206e6ba6020"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:16.587237 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.587111 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-workload-certs" pod "bbc74bb4-ddf5-4a5f-836d-e206e6ba6020" (UID: "bbc74bb4-ddf5-4a5f-836d-e206e6ba6020"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:16.587693 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.587660 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:16.587796 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.587697 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:16.587796 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.587714 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:16.587796 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.587727 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:16.587796 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.587739 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:16.589525 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.589498 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-kube-api-access-nh874" pod "bbc74bb4-ddf5-4a5f-836d-e206e6ba6020" (UID: "bbc74bb4-ddf5-4a5f-836d-e206e6ba6020"). InnerVolumeSpecName "kube-api-access-nh874". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:16.589789 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.589660 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-istio-podinfo" pod "bbc74bb4-ddf5-4a5f-836d-e206e6ba6020" (UID: "bbc74bb4-ddf5-4a5f-836d-e206e6ba6020"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:37:16.589789 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.589737 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-istio-envoy" pod "bbc74bb4-ddf5-4a5f-836d-e206e6ba6020" (UID: "bbc74bb4-ddf5-4a5f-836d-e206e6ba6020"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:16.589789 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.589770 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-istio-token" pod "bbc74bb4-ddf5-4a5f-836d-e206e6ba6020" (UID: "bbc74bb4-ddf5-4a5f-836d-e206e6ba6020"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:16.602271 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.602243 6841 generic.go:356] "Generic (PLEG): container finished" podID="0e6784e6-2aeb-4b2d-b1f8-97f903337e82" containerID="58211cd76607bd49c2f389efdbb9a862de0151b95845dea76c651d897c26eb55" exitCode=0 Oct 31 13:37:16.602389 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.602302 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" Oct 31 13:37:16.602456 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.602408 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" event={"ID":"0e6784e6-2aeb-4b2d-b1f8-97f903337e82","Type":"ContainerDied","Data":"58211cd76607bd49c2f389efdbb9a862de0151b95845dea76c651d897c26eb55"} Oct 31 13:37:16.602517 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.602463 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz" event={"ID":"0e6784e6-2aeb-4b2d-b1f8-97f903337e82","Type":"ContainerDied","Data":"e220e5f6e772d3e6b9653e9a9d07ac61d4a7a895f5c29e9f3a44d04d8dcc5fae"} Oct 31 13:37:16.602517 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.602493 6841 scope.go:122] "RemoveContainer" containerID="58211cd76607bd49c2f389efdbb9a862de0151b95845dea76c651d897c26eb55" Oct 31 13:37:16.612745 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.604853 6841 generic.go:356] "Generic (PLEG): container finished" podID="bbc74bb4-ddf5-4a5f-836d-e206e6ba6020" containerID="594b02f4d01b830cf8211dafef85ac2010c3e9dc37a8a37645934bc8c8f7d7c1" exitCode=0 Oct 31 13:37:16.612745 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.604889 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" event={"ID":"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020","Type":"ContainerDied","Data":"594b02f4d01b830cf8211dafef85ac2010c3e9dc37a8a37645934bc8c8f7d7c1"} Oct 31 13:37:16.612745 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.604906 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" Oct 31 13:37:16.612745 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.604934 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg" event={"ID":"bbc74bb4-ddf5-4a5f-836d-e206e6ba6020","Type":"ContainerDied","Data":"e54a0c518c78e081ca73b9ddf69035c4c88b90413701895fae39709fe849ae17"} Oct 31 13:37:16.619310 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.619292 6841 scope.go:122] "RemoveContainer" containerID="58211cd76607bd49c2f389efdbb9a862de0151b95845dea76c651d897c26eb55" Oct 31 13:37:16.619709 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:37:16.619681 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58211cd76607bd49c2f389efdbb9a862de0151b95845dea76c651d897c26eb55\": container with ID starting with 58211cd76607bd49c2f389efdbb9a862de0151b95845dea76c651d897c26eb55 not found: ID does not exist" containerID="58211cd76607bd49c2f389efdbb9a862de0151b95845dea76c651d897c26eb55" Oct 31 13:37:16.619787 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.619720 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58211cd76607bd49c2f389efdbb9a862de0151b95845dea76c651d897c26eb55"} err="failed to get container status \"58211cd76607bd49c2f389efdbb9a862de0151b95845dea76c651d897c26eb55\": rpc error: code = NotFound desc = could not find container \"58211cd76607bd49c2f389efdbb9a862de0151b95845dea76c651d897c26eb55\": container with ID starting with 58211cd76607bd49c2f389efdbb9a862de0151b95845dea76c651d897c26eb55 not found: ID does not exist" Oct 31 13:37:16.619787 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.619744 6841 scope.go:122] "RemoveContainer" containerID="594b02f4d01b830cf8211dafef85ac2010c3e9dc37a8a37645934bc8c8f7d7c1" Oct 31 13:37:16.644710 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.644685 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz"] Oct 31 13:37:16.651535 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.651507 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-kr6nz"] Oct 31 13:37:16.652953 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.652930 6841 scope.go:122] "RemoveContainer" containerID="594b02f4d01b830cf8211dafef85ac2010c3e9dc37a8a37645934bc8c8f7d7c1" Oct 31 13:37:16.653347 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:37:16.653327 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"594b02f4d01b830cf8211dafef85ac2010c3e9dc37a8a37645934bc8c8f7d7c1\": container with ID starting with 594b02f4d01b830cf8211dafef85ac2010c3e9dc37a8a37645934bc8c8f7d7c1 not found: ID does not exist" containerID="594b02f4d01b830cf8211dafef85ac2010c3e9dc37a8a37645934bc8c8f7d7c1" Oct 31 13:37:16.653432 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.653356 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"594b02f4d01b830cf8211dafef85ac2010c3e9dc37a8a37645934bc8c8f7d7c1"} err="failed to get container status \"594b02f4d01b830cf8211dafef85ac2010c3e9dc37a8a37645934bc8c8f7d7c1\": rpc error: code = NotFound desc = could not find container \"594b02f4d01b830cf8211dafef85ac2010c3e9dc37a8a37645934bc8c8f7d7c1\": container with ID starting with 594b02f4d01b830cf8211dafef85ac2010c3e9dc37a8a37645934bc8c8f7d7c1 not found: ID does not exist" Oct 31 13:37:16.665559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.665511 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg"] Oct 31 13:37:16.670412 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.670385 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-4wglg"] Oct 31 13:37:16.692819 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.692791 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:16.692819 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.692819 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:16.693068 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.692837 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh874\" (UniqueName: \"kubernetes.io/projected/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-kube-api-access-nh874\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:16.693068 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:16.692851 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:17.334132 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:17.334093 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:17.647864 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:17.647773 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e6784e6-2aeb-4b2d-b1f8-97f903337e82" path="/var/lib/kubelet/pods/0e6784e6-2aeb-4b2d-b1f8-97f903337e82/volumes" Oct 31 13:37:17.648731 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:17.648557 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbc74bb4-ddf5-4a5f-836d-e206e6ba6020" path="/var/lib/kubelet/pods/bbc74bb4-ddf5-4a5f-836d-e206e6ba6020/volumes" Oct 31 13:37:18.158239 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:18.158189 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:18.376006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:18.367529 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:18.615019 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:18.614957 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t"] Oct 31 13:37:18.615219 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:18.615175 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" podUID="61c4f511-5495-4b14-ae0f-b019d826ae7d" containerName="istio-proxy" containerID="cri-o://025092551b0a473f1f1640e39ca889d10f706727677fd5a0d827846cf3f7c547" gracePeriod=30 Oct 31 13:37:18.641064 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:18.641006 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:19.367754 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:19.367704 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:19.621537 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:19.621436 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:20.035742 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:37:20.035710 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:37:20.465170 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:20.465139 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:20.554900 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:20.554862 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz"] Oct 31 13:37:20.555118 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:20.555094 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" podUID="4f2bc576-3d6a-44d0-b30a-3eae4f627961" containerName="istio-proxy" containerID="cri-o://dc80ed599d793dd32f95edbd08a7dfff0423e1452a6b917be6d6fa28bca0edc4" gracePeriod=30 Oct 31 13:37:21.406497 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:21.406447 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:22.203570 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:22.203525 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:23.018524 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:23.018482 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:23.218114 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:23.218079 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:23.714789 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:23.714754 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:23.903178 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:23.903111 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" Oct 31 13:37:23.991380 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:23.991332 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c4f511-5495-4b14-ae0f-b019d826ae7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61c4f511-5495-4b14-ae0f-b019d826ae7d-istio-data\") pod \"61c4f511-5495-4b14-ae0f-b019d826ae7d\" (UID: \"61c4f511-5495-4b14-ae0f-b019d826ae7d\") " Oct 31 13:37:23.991566 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:23.991467 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61c4f511-5495-4b14-ae0f-b019d826ae7d-istio-token\" (UniqueName: \"kubernetes.io/projected/61c4f511-5495-4b14-ae0f-b019d826ae7d-istio-token\") pod \"61c4f511-5495-4b14-ae0f-b019d826ae7d\" (UID: \"61c4f511-5495-4b14-ae0f-b019d826ae7d\") " Oct 31 13:37:23.991566 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:23.991516 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61c4f511-5495-4b14-ae0f-b019d826ae7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61c4f511-5495-4b14-ae0f-b019d826ae7d-istiod-ca-cert\") pod \"61c4f511-5495-4b14-ae0f-b019d826ae7d\" (UID: \"61c4f511-5495-4b14-ae0f-b019d826ae7d\") " Oct 31 13:37:23.991722 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:23.991706 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c4f511-5495-4b14-ae0f-b019d826ae7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c4f511-5495-4b14-ae0f-b019d826ae7d-workload-socket\") pod \"61c4f511-5495-4b14-ae0f-b019d826ae7d\" (UID: \"61c4f511-5495-4b14-ae0f-b019d826ae7d\") " Oct 31 13:37:23.991789 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:23.991757 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c4f511-5495-4b14-ae0f-b019d826ae7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c4f511-5495-4b14-ae0f-b019d826ae7d-credential-socket\") pod \"61c4f511-5495-4b14-ae0f-b019d826ae7d\" (UID: \"61c4f511-5495-4b14-ae0f-b019d826ae7d\") " Oct 31 13:37:23.991848 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:23.991793 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61c4f511-5495-4b14-ae0f-b019d826ae7d-kube-api-access-s6fgg\" (UniqueName: \"kubernetes.io/projected/61c4f511-5495-4b14-ae0f-b019d826ae7d-kube-api-access-s6fgg\") pod \"61c4f511-5495-4b14-ae0f-b019d826ae7d\" (UID: \"61c4f511-5495-4b14-ae0f-b019d826ae7d\") " Oct 31 13:37:23.991909 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:23.991865 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c4f511-5495-4b14-ae0f-b019d826ae7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61c4f511-5495-4b14-ae0f-b019d826ae7d-istio-envoy\") pod \"61c4f511-5495-4b14-ae0f-b019d826ae7d\" (UID: \"61c4f511-5495-4b14-ae0f-b019d826ae7d\") " Oct 31 13:37:23.991976 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:23.991931 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c4f511-5495-4b14-ae0f-b019d826ae7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61c4f511-5495-4b14-ae0f-b019d826ae7d-workload-certs\") pod \"61c4f511-5495-4b14-ae0f-b019d826ae7d\" (UID: \"61c4f511-5495-4b14-ae0f-b019d826ae7d\") " Oct 31 13:37:23.991976 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:23.991970 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61c4f511-5495-4b14-ae0f-b019d826ae7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61c4f511-5495-4b14-ae0f-b019d826ae7d-istio-podinfo\") pod \"61c4f511-5495-4b14-ae0f-b019d826ae7d\" (UID: \"61c4f511-5495-4b14-ae0f-b019d826ae7d\") " Oct 31 13:37:23.992119 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:23.992024 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61c4f511-5495-4b14-ae0f-b019d826ae7d-istiod-ca-cert" pod "61c4f511-5495-4b14-ae0f-b019d826ae7d" (UID: "61c4f511-5495-4b14-ae0f-b019d826ae7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:37:23.992119 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:23.992041 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c4f511-5495-4b14-ae0f-b019d826ae7d-istio-data" pod "61c4f511-5495-4b14-ae0f-b019d826ae7d" (UID: "61c4f511-5495-4b14-ae0f-b019d826ae7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:23.992300 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:23.992281 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c4f511-5495-4b14-ae0f-b019d826ae7d-credential-socket" pod "61c4f511-5495-4b14-ae0f-b019d826ae7d" (UID: "61c4f511-5495-4b14-ae0f-b019d826ae7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:23.992770 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:23.992423 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c4f511-5495-4b14-ae0f-b019d826ae7d-workload-certs" pod "61c4f511-5495-4b14-ae0f-b019d826ae7d" (UID: "61c4f511-5495-4b14-ae0f-b019d826ae7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:23.992770 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:23.992740 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c4f511-5495-4b14-ae0f-b019d826ae7d-workload-socket" pod "61c4f511-5495-4b14-ae0f-b019d826ae7d" (UID: "61c4f511-5495-4b14-ae0f-b019d826ae7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:23.993160 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:23.993064 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61c4f511-5495-4b14-ae0f-b019d826ae7d-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:23.993160 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:23.993091 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61c4f511-5495-4b14-ae0f-b019d826ae7d-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:23.993160 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:23.993106 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c4f511-5495-4b14-ae0f-b019d826ae7d-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:23.993160 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:23.993120 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c4f511-5495-4b14-ae0f-b019d826ae7d-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:23.993160 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:23.993135 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61c4f511-5495-4b14-ae0f-b019d826ae7d-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:23.995040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:23.995012 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61c4f511-5495-4b14-ae0f-b019d826ae7d-kube-api-access-s6fgg" pod "61c4f511-5495-4b14-ae0f-b019d826ae7d" (UID: "61c4f511-5495-4b14-ae0f-b019d826ae7d"). InnerVolumeSpecName "kube-api-access-s6fgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:23.995487 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:23.995256 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c4f511-5495-4b14-ae0f-b019d826ae7d-istio-envoy" pod "61c4f511-5495-4b14-ae0f-b019d826ae7d" (UID: "61c4f511-5495-4b14-ae0f-b019d826ae7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:23.995577 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:23.995490 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61c4f511-5495-4b14-ae0f-b019d826ae7d-istio-token" pod "61c4f511-5495-4b14-ae0f-b019d826ae7d" (UID: "61c4f511-5495-4b14-ae0f-b019d826ae7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:23.995772 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:23.995742 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61c4f511-5495-4b14-ae0f-b019d826ae7d-istio-podinfo" pod "61c4f511-5495-4b14-ae0f-b019d826ae7d" (UID: "61c4f511-5495-4b14-ae0f-b019d826ae7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:37:24.094579 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:24.094543 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6fgg\" (UniqueName: \"kubernetes.io/projected/61c4f511-5495-4b14-ae0f-b019d826ae7d-kube-api-access-s6fgg\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:24.094579 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:24.094578 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61c4f511-5495-4b14-ae0f-b019d826ae7d-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:24.094789 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:24.094590 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61c4f511-5495-4b14-ae0f-b019d826ae7d-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:24.094789 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:24.094599 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61c4f511-5495-4b14-ae0f-b019d826ae7d-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:24.417043 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:24.416967 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:24.720243 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:24.720212 6841 generic.go:356] "Generic (PLEG): container finished" podID="61c4f511-5495-4b14-ae0f-b019d826ae7d" containerID="025092551b0a473f1f1640e39ca889d10f706727677fd5a0d827846cf3f7c547" exitCode=0 Oct 31 13:37:24.720424 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:24.720258 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" Oct 31 13:37:24.720424 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:24.720292 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" event={"ID":"61c4f511-5495-4b14-ae0f-b019d826ae7d","Type":"ContainerDied","Data":"025092551b0a473f1f1640e39ca889d10f706727677fd5a0d827846cf3f7c547"} Oct 31 13:37:24.720424 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:24.720348 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t" event={"ID":"61c4f511-5495-4b14-ae0f-b019d826ae7d","Type":"ContainerDied","Data":"f3f6eb6b6390e5279d9b6406bc11718642d48378a098cb62a6d00bbf38c21e64"} Oct 31 13:37:24.720424 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:24.720379 6841 scope.go:122] "RemoveContainer" containerID="025092551b0a473f1f1640e39ca889d10f706727677fd5a0d827846cf3f7c547" Oct 31 13:37:24.732384 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:24.732365 6841 scope.go:122] "RemoveContainer" containerID="025092551b0a473f1f1640e39ca889d10f706727677fd5a0d827846cf3f7c547" Oct 31 13:37:24.732689 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:37:24.732664 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"025092551b0a473f1f1640e39ca889d10f706727677fd5a0d827846cf3f7c547\": container with ID starting with 025092551b0a473f1f1640e39ca889d10f706727677fd5a0d827846cf3f7c547 not found: ID does not exist" containerID="025092551b0a473f1f1640e39ca889d10f706727677fd5a0d827846cf3f7c547" Oct 31 13:37:24.732783 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:24.732694 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"025092551b0a473f1f1640e39ca889d10f706727677fd5a0d827846cf3f7c547"} err="failed to get container status \"025092551b0a473f1f1640e39ca889d10f706727677fd5a0d827846cf3f7c547\": rpc error: code = NotFound desc = could not find container \"025092551b0a473f1f1640e39ca889d10f706727677fd5a0d827846cf3f7c547\": container with ID starting with 025092551b0a473f1f1640e39ca889d10f706727677fd5a0d827846cf3f7c547 not found: ID does not exist" Oct 31 13:37:24.746492 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:24.746469 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t"] Oct 31 13:37:24.750316 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:24.750291 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-b225t"] Oct 31 13:37:25.127292 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:25.127203 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:25.645187 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:25.645160 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61c4f511-5495-4b14-ae0f-b019d826ae7d" path="/var/lib/kubelet/pods/61c4f511-5495-4b14-ae0f-b019d826ae7d/volumes" Oct 31 13:37:25.686013 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:25.685953 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:25.732769 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:25.732736 6841 generic.go:356] "Generic (PLEG): container finished" podID="4f2bc576-3d6a-44d0-b30a-3eae4f627961" containerID="dc80ed599d793dd32f95edbd08a7dfff0423e1452a6b917be6d6fa28bca0edc4" exitCode=0 Oct 31 13:37:25.732885 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:25.732807 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" event={"ID":"4f2bc576-3d6a-44d0-b30a-3eae4f627961","Type":"ContainerDied","Data":"dc80ed599d793dd32f95edbd08a7dfff0423e1452a6b917be6d6fa28bca0edc4"} Oct 31 13:37:25.732948 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:25.732887 6841 scope.go:122] "RemoveContainer" containerID="87e55479dcc685d7818ca1d579dbd3fa589e7cccc2fc22ec0f440007157d5a66" Oct 31 13:37:25.847222 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:25.847194 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:37:25.929031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:25.927557 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f2bc576-3d6a-44d0-b30a-3eae4f627961-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f2bc576-3d6a-44d0-b30a-3eae4f627961-credential-socket\") pod \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\" (UID: \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\") " Oct 31 13:37:25.929031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:25.927645 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f2bc576-3d6a-44d0-b30a-3eae4f627961-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f2bc576-3d6a-44d0-b30a-3eae4f627961-istiod-ca-cert\") pod \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\" (UID: \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\") " Oct 31 13:37:25.929031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:25.927736 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f2bc576-3d6a-44d0-b30a-3eae4f627961-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f2bc576-3d6a-44d0-b30a-3eae4f627961-istio-podinfo\") pod \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\" (UID: \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\") " Oct 31 13:37:25.929031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:25.927797 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f2bc576-3d6a-44d0-b30a-3eae4f627961-kube-api-access-htk49\" (UniqueName: \"kubernetes.io/projected/4f2bc576-3d6a-44d0-b30a-3eae4f627961-kube-api-access-htk49\") pod \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\" (UID: \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\") " Oct 31 13:37:25.929031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:25.927919 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f2bc576-3d6a-44d0-b30a-3eae4f627961-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f2bc576-3d6a-44d0-b30a-3eae4f627961-workload-socket\") pod \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\" (UID: \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\") " Oct 31 13:37:25.929031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:25.927977 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f2bc576-3d6a-44d0-b30a-3eae4f627961-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f2bc576-3d6a-44d0-b30a-3eae4f627961-istio-envoy\") pod \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\" (UID: \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\") " Oct 31 13:37:25.929031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:25.928084 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f2bc576-3d6a-44d0-b30a-3eae4f627961-istio-token\" (UniqueName: \"kubernetes.io/projected/4f2bc576-3d6a-44d0-b30a-3eae4f627961-istio-token\") pod \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\" (UID: \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\") " Oct 31 13:37:25.929031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:25.928128 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f2bc576-3d6a-44d0-b30a-3eae4f627961-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f2bc576-3d6a-44d0-b30a-3eae4f627961-workload-certs\") pod \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\" (UID: \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\") " Oct 31 13:37:25.929031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:25.928180 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f2bc576-3d6a-44d0-b30a-3eae4f627961-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f2bc576-3d6a-44d0-b30a-3eae4f627961-istio-data\") pod \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\" (UID: \"4f2bc576-3d6a-44d0-b30a-3eae4f627961\") " Oct 31 13:37:25.930080 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:25.930049 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f2bc576-3d6a-44d0-b30a-3eae4f627961-istio-data" pod "4f2bc576-3d6a-44d0-b30a-3eae4f627961" (UID: "4f2bc576-3d6a-44d0-b30a-3eae4f627961"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:25.930291 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:25.930269 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f2bc576-3d6a-44d0-b30a-3eae4f627961-credential-socket" pod "4f2bc576-3d6a-44d0-b30a-3eae4f627961" (UID: "4f2bc576-3d6a-44d0-b30a-3eae4f627961"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:25.930582 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:25.930564 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f2bc576-3d6a-44d0-b30a-3eae4f627961-istiod-ca-cert" pod "4f2bc576-3d6a-44d0-b30a-3eae4f627961" (UID: "4f2bc576-3d6a-44d0-b30a-3eae4f627961"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:37:25.931784 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:25.931527 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f2bc576-3d6a-44d0-b30a-3eae4f627961-workload-certs" pod "4f2bc576-3d6a-44d0-b30a-3eae4f627961" (UID: "4f2bc576-3d6a-44d0-b30a-3eae4f627961"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:25.932261 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:25.932236 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f2bc576-3d6a-44d0-b30a-3eae4f627961-workload-socket" pod "4f2bc576-3d6a-44d0-b30a-3eae4f627961" (UID: "4f2bc576-3d6a-44d0-b30a-3eae4f627961"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:25.933807 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:25.933780 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f2bc576-3d6a-44d0-b30a-3eae4f627961-istio-podinfo" pod "4f2bc576-3d6a-44d0-b30a-3eae4f627961" (UID: "4f2bc576-3d6a-44d0-b30a-3eae4f627961"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:37:25.946406 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:25.946191 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:25.956023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:25.955327 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f2bc576-3d6a-44d0-b30a-3eae4f627961-istio-envoy" pod "4f2bc576-3d6a-44d0-b30a-3eae4f627961" (UID: "4f2bc576-3d6a-44d0-b30a-3eae4f627961"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:25.956023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:25.955432 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f2bc576-3d6a-44d0-b30a-3eae4f627961-istio-token" pod "4f2bc576-3d6a-44d0-b30a-3eae4f627961" (UID: "4f2bc576-3d6a-44d0-b30a-3eae4f627961"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:25.957511 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:25.957481 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f2bc576-3d6a-44d0-b30a-3eae4f627961-kube-api-access-htk49" pod "4f2bc576-3d6a-44d0-b30a-3eae4f627961" (UID: "4f2bc576-3d6a-44d0-b30a-3eae4f627961"). InnerVolumeSpecName "kube-api-access-htk49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:26.030516 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:26.030483 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f2bc576-3d6a-44d0-b30a-3eae4f627961-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:26.030516 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:26.030514 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htk49\" (UniqueName: \"kubernetes.io/projected/4f2bc576-3d6a-44d0-b30a-3eae4f627961-kube-api-access-htk49\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:26.030737 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:26.030529 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f2bc576-3d6a-44d0-b30a-3eae4f627961-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:26.030737 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:26.030543 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f2bc576-3d6a-44d0-b30a-3eae4f627961-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:26.030737 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:26.030556 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f2bc576-3d6a-44d0-b30a-3eae4f627961-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:26.030737 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:26.030567 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f2bc576-3d6a-44d0-b30a-3eae4f627961-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:26.030737 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:26.030579 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f2bc576-3d6a-44d0-b30a-3eae4f627961-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:26.030737 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:26.030591 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f2bc576-3d6a-44d0-b30a-3eae4f627961-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:26.030737 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:26.030602 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f2bc576-3d6a-44d0-b30a-3eae4f627961-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:26.627645 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:26.627606 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:26.741667 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:26.741636 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" Oct 31 13:37:26.742133 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:26.741681 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz" event={"ID":"4f2bc576-3d6a-44d0-b30a-3eae4f627961","Type":"ContainerDied","Data":"8dd2c2e63e5f791419d317560898f31fb97987cdb2aa3a63ad03c964df704b21"} Oct 31 13:37:26.742133 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:26.741746 6841 scope.go:122] "RemoveContainer" containerID="dc80ed599d793dd32f95edbd08a7dfff0423e1452a6b917be6d6fa28bca0edc4" Oct 31 13:37:26.768641 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:26.768610 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz"] Oct 31 13:37:26.777018 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:26.776451 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-8b5cz"] Oct 31 13:37:27.385109 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:27.385075 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:27.645038 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:27.644911 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f2bc576-3d6a-44d0-b30a-3eae4f627961" path="/var/lib/kubelet/pods/4f2bc576-3d6a-44d0-b30a-3eae4f627961/volumes" Oct 31 13:37:28.150148 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:28.150110 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:28.396539 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:28.396501 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:29.196878 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:29.196847 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:30.188168 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:30.188137 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:31.254907 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:31.254864 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:32.638157 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:32.637930 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:34.136688 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:34.136654 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:35.378333 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:35.378279 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:36.542595 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:36.542559 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:37.965653 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:37.965614 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:38.758287 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:38.758248 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:40.210431 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:40.210394 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:40.541105 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:40.541027 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:41.455775 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:41.455742 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:42.219834 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:42.219801 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:42.613103 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:42.613025 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht"] Oct 31 13:37:42.730022 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:42.728456 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:43.012944 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:43.012896 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r"] Oct 31 13:37:43.013236 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:43.013183 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r" podUID="283a00d5-0c1d-4e1e-9996-58f08babdb21" containerName="maas-api" containerID="cri-o://273973c94298446c550bb39231b2e1567cc5035bc56ef1d37af21bb9d8b16015" gracePeriod=30 Oct 31 13:37:43.315006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:43.312578 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r" Oct 31 13:37:43.350621 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:43.350530 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:43.420005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:43.418748 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/283a00d5-0c1d-4e1e-9996-58f08babdb21-metrics-certs\" (UniqueName: \"kubernetes.io/secret/283a00d5-0c1d-4e1e-9996-58f08babdb21-metrics-certs\") pod \"283a00d5-0c1d-4e1e-9996-58f08babdb21\" (UID: \"283a00d5-0c1d-4e1e-9996-58f08babdb21\") " Oct 31 13:37:43.420005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:43.418815 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/283a00d5-0c1d-4e1e-9996-58f08babdb21-kube-api-access-q2kz7\" (UniqueName: \"kubernetes.io/projected/283a00d5-0c1d-4e1e-9996-58f08babdb21-kube-api-access-q2kz7\") pod \"283a00d5-0c1d-4e1e-9996-58f08babdb21\" (UID: \"283a00d5-0c1d-4e1e-9996-58f08babdb21\") " Oct 31 13:37:43.420005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:43.418907 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/283a00d5-0c1d-4e1e-9996-58f08babdb21-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/283a00d5-0c1d-4e1e-9996-58f08babdb21-maas-api-tls\") pod \"283a00d5-0c1d-4e1e-9996-58f08babdb21\" (UID: \"283a00d5-0c1d-4e1e-9996-58f08babdb21\") " Oct 31 13:37:43.420005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:43.418946 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/283a00d5-0c1d-4e1e-9996-58f08babdb21-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/283a00d5-0c1d-4e1e-9996-58f08babdb21-openshift-service-ca\") pod \"283a00d5-0c1d-4e1e-9996-58f08babdb21\" (UID: \"283a00d5-0c1d-4e1e-9996-58f08babdb21\") " Oct 31 13:37:43.436011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:43.428768 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/283a00d5-0c1d-4e1e-9996-58f08babdb21-openshift-service-ca" pod "283a00d5-0c1d-4e1e-9996-58f08babdb21" (UID: "283a00d5-0c1d-4e1e-9996-58f08babdb21"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:37:43.452003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:43.450100 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/283a00d5-0c1d-4e1e-9996-58f08babdb21-metrics-certs" pod "283a00d5-0c1d-4e1e-9996-58f08babdb21" (UID: "283a00d5-0c1d-4e1e-9996-58f08babdb21"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:37:43.520593 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:43.520168 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/283a00d5-0c1d-4e1e-9996-58f08babdb21-maas-api-tls" pod "283a00d5-0c1d-4e1e-9996-58f08babdb21" (UID: "283a00d5-0c1d-4e1e-9996-58f08babdb21"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:37:43.520593 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:43.520333 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/283a00d5-0c1d-4e1e-9996-58f08babdb21-kube-api-access-q2kz7" pod "283a00d5-0c1d-4e1e-9996-58f08babdb21" (UID: "283a00d5-0c1d-4e1e-9996-58f08babdb21"). InnerVolumeSpecName "kube-api-access-q2kz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:43.522094 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:43.522002 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/283a00d5-0c1d-4e1e-9996-58f08babdb21-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:43.522094 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:43.522037 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2kz7\" (UniqueName: \"kubernetes.io/projected/283a00d5-0c1d-4e1e-9996-58f08babdb21-kube-api-access-q2kz7\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:43.522094 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:43.522055 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/283a00d5-0c1d-4e1e-9996-58f08babdb21-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:43.522094 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:43.522069 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/283a00d5-0c1d-4e1e-9996-58f08babdb21-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:43.683075 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:43.683034 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:43.904545 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:43.904470 6841 generic.go:356] "Generic (PLEG): container finished" podID="283a00d5-0c1d-4e1e-9996-58f08babdb21" containerID="273973c94298446c550bb39231b2e1567cc5035bc56ef1d37af21bb9d8b16015" exitCode=0 Oct 31 13:37:43.904545 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:43.904518 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r" Oct 31 13:37:43.904744 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:43.904558 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r" event={"ID":"283a00d5-0c1d-4e1e-9996-58f08babdb21","Type":"ContainerDied","Data":"273973c94298446c550bb39231b2e1567cc5035bc56ef1d37af21bb9d8b16015"} Oct 31 13:37:43.904744 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:43.904613 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r" event={"ID":"283a00d5-0c1d-4e1e-9996-58f08babdb21","Type":"ContainerDied","Data":"1f4da1bbd14f5a6fc519520fc5f5a13460e81f40f2656134c6f50669ef83e091"} Oct 31 13:37:43.904744 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:43.904643 6841 scope.go:122] "RemoveContainer" containerID="273973c94298446c550bb39231b2e1567cc5035bc56ef1d37af21bb9d8b16015" Oct 31 13:37:43.915608 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:43.915583 6841 scope.go:122] "RemoveContainer" containerID="273973c94298446c550bb39231b2e1567cc5035bc56ef1d37af21bb9d8b16015" Oct 31 13:37:43.915919 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:37:43.915897 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"273973c94298446c550bb39231b2e1567cc5035bc56ef1d37af21bb9d8b16015\": container with ID starting with 273973c94298446c550bb39231b2e1567cc5035bc56ef1d37af21bb9d8b16015 not found: ID does not exist" containerID="273973c94298446c550bb39231b2e1567cc5035bc56ef1d37af21bb9d8b16015" Oct 31 13:37:43.916064 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:43.915926 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273973c94298446c550bb39231b2e1567cc5035bc56ef1d37af21bb9d8b16015"} err="failed to get container status \"273973c94298446c550bb39231b2e1567cc5035bc56ef1d37af21bb9d8b16015\": rpc error: code = NotFound desc = could not find container \"273973c94298446c550bb39231b2e1567cc5035bc56ef1d37af21bb9d8b16015\": container with ID starting with 273973c94298446c550bb39231b2e1567cc5035bc56ef1d37af21bb9d8b16015 not found: ID does not exist" Oct 31 13:37:43.925707 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:43.925683 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:43.936799 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:43.936778 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r"] Oct 31 13:37:43.941511 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:43.941487 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3cddb3-654765cb66-8d99r"] Oct 31 13:37:44.576577 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:44.576547 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:45.553925 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:45.553888 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:45.643557 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:45.643527 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="283a00d5-0c1d-4e1e-9996-58f08babdb21" path="/var/lib/kubelet/pods/283a00d5-0c1d-4e1e-9996-58f08babdb21/volumes" Oct 31 13:37:46.563489 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:46.563453 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:47.252546 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:47.252487 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:47.654540 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:47.654433 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:48.054389 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:48.054352 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:48.271925 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:48.271888 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8"] Oct 31 13:37:48.272208 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:48.272179 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" podUID="b5c84160-0130-4f19-a32f-1b1b0f038198" containerName="istio-proxy" containerID="cri-o://12a237c2e1ebeadf6e51599ba3e65653a56f07b7b5673a82858e92d18b979baa" gracePeriod=30 Oct 31 13:37:48.424059 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:48.420931 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:48.865353 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:48.865319 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:49.552779 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:49.552750 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:49.786072 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:49.780357 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:50.037321 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:37:50.037283 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:37:50.064878 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:50.064848 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:50.867096 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:50.867059 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:51.421522 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:51.421485 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:52.038508 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:52.038474 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:52.724316 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:52.724281 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:53.583128 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.583031 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:53.612665 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.612640 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" Oct 31 13:37:53.723738 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.723704 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5c84160-0130-4f19-a32f-1b1b0f038198-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5c84160-0130-4f19-a32f-1b1b0f038198-workload-socket\") pod \"b5c84160-0130-4f19-a32f-1b1b0f038198\" (UID: \"b5c84160-0130-4f19-a32f-1b1b0f038198\") " Oct 31 13:37:53.723738 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.723743 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5c84160-0130-4f19-a32f-1b1b0f038198-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5c84160-0130-4f19-a32f-1b1b0f038198-workload-certs\") pod \"b5c84160-0130-4f19-a32f-1b1b0f038198\" (UID: \"b5c84160-0130-4f19-a32f-1b1b0f038198\") " Oct 31 13:37:53.724044 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.723777 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5c84160-0130-4f19-a32f-1b1b0f038198-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5c84160-0130-4f19-a32f-1b1b0f038198-credential-socket\") pod \"b5c84160-0130-4f19-a32f-1b1b0f038198\" (UID: \"b5c84160-0130-4f19-a32f-1b1b0f038198\") " Oct 31 13:37:53.724044 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.723861 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5c84160-0130-4f19-a32f-1b1b0f038198-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5c84160-0130-4f19-a32f-1b1b0f038198-istiod-ca-cert\") pod \"b5c84160-0130-4f19-a32f-1b1b0f038198\" (UID: \"b5c84160-0130-4f19-a32f-1b1b0f038198\") " Oct 31 13:37:53.724044 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.723932 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5c84160-0130-4f19-a32f-1b1b0f038198-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5c84160-0130-4f19-a32f-1b1b0f038198-istio-envoy\") pod \"b5c84160-0130-4f19-a32f-1b1b0f038198\" (UID: \"b5c84160-0130-4f19-a32f-1b1b0f038198\") " Oct 31 13:37:53.724044 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.724000 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5c84160-0130-4f19-a32f-1b1b0f038198-workload-socket" pod "b5c84160-0130-4f19-a32f-1b1b0f038198" (UID: "b5c84160-0130-4f19-a32f-1b1b0f038198"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:53.724044 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.724015 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5c84160-0130-4f19-a32f-1b1b0f038198-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5c84160-0130-4f19-a32f-1b1b0f038198-istio-podinfo\") pod \"b5c84160-0130-4f19-a32f-1b1b0f038198\" (UID: \"b5c84160-0130-4f19-a32f-1b1b0f038198\") " Oct 31 13:37:53.724343 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.724071 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5c84160-0130-4f19-a32f-1b1b0f038198-kube-api-access-4vvsc\" (UniqueName: \"kubernetes.io/projected/b5c84160-0130-4f19-a32f-1b1b0f038198-kube-api-access-4vvsc\") pod \"b5c84160-0130-4f19-a32f-1b1b0f038198\" (UID: \"b5c84160-0130-4f19-a32f-1b1b0f038198\") " Oct 31 13:37:53.724343 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.724111 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5c84160-0130-4f19-a32f-1b1b0f038198-workload-certs" pod "b5c84160-0130-4f19-a32f-1b1b0f038198" (UID: "b5c84160-0130-4f19-a32f-1b1b0f038198"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:53.724343 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.724155 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5c84160-0130-4f19-a32f-1b1b0f038198-credential-socket" pod "b5c84160-0130-4f19-a32f-1b1b0f038198" (UID: "b5c84160-0130-4f19-a32f-1b1b0f038198"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:53.724343 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.724271 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5c84160-0130-4f19-a32f-1b1b0f038198-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5c84160-0130-4f19-a32f-1b1b0f038198-istio-data\") pod \"b5c84160-0130-4f19-a32f-1b1b0f038198\" (UID: \"b5c84160-0130-4f19-a32f-1b1b0f038198\") " Oct 31 13:37:53.724343 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.724313 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5c84160-0130-4f19-a32f-1b1b0f038198-istio-token\" (UniqueName: \"kubernetes.io/projected/b5c84160-0130-4f19-a32f-1b1b0f038198-istio-token\") pod \"b5c84160-0130-4f19-a32f-1b1b0f038198\" (UID: \"b5c84160-0130-4f19-a32f-1b1b0f038198\") " Oct 31 13:37:53.725001 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.724382 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5c84160-0130-4f19-a32f-1b1b0f038198-istiod-ca-cert" pod "b5c84160-0130-4f19-a32f-1b1b0f038198" (UID: "b5c84160-0130-4f19-a32f-1b1b0f038198"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:37:53.725001 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.724502 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5c84160-0130-4f19-a32f-1b1b0f038198-istio-data" pod "b5c84160-0130-4f19-a32f-1b1b0f038198" (UID: "b5c84160-0130-4f19-a32f-1b1b0f038198"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:53.725179 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.725159 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5c84160-0130-4f19-a32f-1b1b0f038198-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:53.725271 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.725185 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5c84160-0130-4f19-a32f-1b1b0f038198-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:53.725271 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.725202 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5c84160-0130-4f19-a32f-1b1b0f038198-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:53.725271 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.725217 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5c84160-0130-4f19-a32f-1b1b0f038198-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:53.725271 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.725232 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5c84160-0130-4f19-a32f-1b1b0f038198-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:53.726763 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.726740 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5c84160-0130-4f19-a32f-1b1b0f038198-istio-token" pod "b5c84160-0130-4f19-a32f-1b1b0f038198" (UID: "b5c84160-0130-4f19-a32f-1b1b0f038198"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:53.727032 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.726999 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5c84160-0130-4f19-a32f-1b1b0f038198-istio-podinfo" pod "b5c84160-0130-4f19-a32f-1b1b0f038198" (UID: "b5c84160-0130-4f19-a32f-1b1b0f038198"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:37:53.727178 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.727156 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5c84160-0130-4f19-a32f-1b1b0f038198-kube-api-access-4vvsc" pod "b5c84160-0130-4f19-a32f-1b1b0f038198" (UID: "b5c84160-0130-4f19-a32f-1b1b0f038198"). InnerVolumeSpecName "kube-api-access-4vvsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:53.727268 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.727216 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5c84160-0130-4f19-a32f-1b1b0f038198-istio-envoy" pod "b5c84160-0130-4f19-a32f-1b1b0f038198" (UID: "b5c84160-0130-4f19-a32f-1b1b0f038198"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:53.750155 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.750118 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng"] Oct 31 13:37:53.751508 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.751481 6841 cpu_manager.go:414] "containerMap: removing container" podUID="b5c84160-0130-4f19-a32f-1b1b0f038198" containerName="istio-proxy" Oct 31 13:37:53.751646 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.751629 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5c84160-0130-4f19-a32f-1b1b0f038198" containerName="istio-proxy" Oct 31 13:37:53.751766 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.751753 6841 cpu_manager.go:414] "containerMap: removing container" podUID="0e6784e6-2aeb-4b2d-b1f8-97f903337e82" containerName="istio-proxy" Oct 31 13:37:53.751858 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.751846 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e6784e6-2aeb-4b2d-b1f8-97f903337e82" containerName="istio-proxy" Oct 31 13:37:53.751961 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.751950 6841 cpu_manager.go:414] "containerMap: removing container" podUID="61c9f172-a17d-4531-8626-6bd1433a9a39" containerName="payload-processing" Oct 31 13:37:53.752084 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.752069 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61c9f172-a17d-4531-8626-6bd1433a9a39" containerName="payload-processing" Oct 31 13:37:53.752195 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.752184 6841 cpu_manager.go:414] "containerMap: removing container" podUID="61c4f511-5495-4b14-ae0f-b019d826ae7d" containerName="istio-proxy" Oct 31 13:37:53.752281 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.752270 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61c4f511-5495-4b14-ae0f-b019d826ae7d" containerName="istio-proxy" Oct 31 13:37:53.752385 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.752375 6841 cpu_manager.go:414] "containerMap: removing container" podUID="1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939" containerName="main" Oct 31 13:37:53.752474 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.752462 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939" containerName="main" Oct 31 13:37:53.752572 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.752561 6841 cpu_manager.go:414] "containerMap: removing container" podUID="283a00d5-0c1d-4e1e-9996-58f08babdb21" containerName="maas-api" Oct 31 13:37:53.752659 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.752648 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="283a00d5-0c1d-4e1e-9996-58f08babdb21" containerName="maas-api" Oct 31 13:37:53.752746 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.752735 6841 cpu_manager.go:414] "containerMap: removing container" podUID="68406df2-8453-4762-9398-4019221157aa" containerName="istio-proxy" Oct 31 13:37:53.752828 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.752817 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68406df2-8453-4762-9398-4019221157aa" containerName="istio-proxy" Oct 31 13:37:53.752976 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.752963 6841 cpu_manager.go:414] "containerMap: removing container" podUID="4f2bc576-3d6a-44d0-b30a-3eae4f627961" containerName="istio-proxy" Oct 31 13:37:53.753108 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.753093 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f2bc576-3d6a-44d0-b30a-3eae4f627961" containerName="istio-proxy" Oct 31 13:37:53.753202 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.753190 6841 cpu_manager.go:414] "containerMap: removing container" podUID="4f2bc576-3d6a-44d0-b30a-3eae4f627961" containerName="istio-proxy" Oct 31 13:37:53.753286 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.753273 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f2bc576-3d6a-44d0-b30a-3eae4f627961" containerName="istio-proxy" Oct 31 13:37:53.753379 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.753368 6841 cpu_manager.go:414] "containerMap: removing container" podUID="bbc74bb4-ddf5-4a5f-836d-e206e6ba6020" containerName="istio-proxy" Oct 31 13:37:53.753462 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.753451 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbc74bb4-ddf5-4a5f-836d-e206e6ba6020" containerName="istio-proxy" Oct 31 13:37:53.753562 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.753552 6841 cpu_manager.go:414] "containerMap: removing container" podUID="109fb6e5-38ef-4030-9209-cda454c140ec" containerName="main" Oct 31 13:37:53.753645 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.753633 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="109fb6e5-38ef-4030-9209-cda454c140ec" containerName="main" Oct 31 13:37:53.753933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.753897 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f2bc576-3d6a-44d0-b30a-3eae4f627961" containerName="istio-proxy" Oct 31 13:37:53.753933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.753924 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f2bc576-3d6a-44d0-b30a-3eae4f627961" containerName="istio-proxy" Oct 31 13:37:53.754263 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.753974 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbc74bb4-ddf5-4a5f-836d-e206e6ba6020" containerName="istio-proxy" Oct 31 13:37:53.754263 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.754002 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="109fb6e5-38ef-4030-9209-cda454c140ec" containerName="main" Oct 31 13:37:53.754263 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.754016 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5c84160-0130-4f19-a32f-1b1b0f038198" containerName="istio-proxy" Oct 31 13:37:53.754263 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.754028 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e6784e6-2aeb-4b2d-b1f8-97f903337e82" containerName="istio-proxy" Oct 31 13:37:53.754263 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.754042 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="61c9f172-a17d-4531-8626-6bd1433a9a39" containerName="payload-processing" Oct 31 13:37:53.754263 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.754080 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="61c4f511-5495-4b14-ae0f-b019d826ae7d" containerName="istio-proxy" Oct 31 13:37:53.754263 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.754093 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f1f43cc-5fd8-4f7b-8bcc-67b32f2ac939" containerName="main" Oct 31 13:37:53.754263 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.754104 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="283a00d5-0c1d-4e1e-9996-58f08babdb21" containerName="maas-api" Oct 31 13:37:53.754263 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.754113 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="68406df2-8453-4762-9398-4019221157aa" containerName="istio-proxy" Oct 31 13:37:53.762315 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.762291 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng"] Oct 31 13:37:53.762411 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.762392 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng" Oct 31 13:37:53.772796 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.772773 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-6c0612\"/\"openshift-service-ca.crt\"" Oct 31 13:37:53.772964 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.772928 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-6c0612\"/\"kube-root-ca.crt\"" Oct 31 13:37:53.774438 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.774416 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-6c0612\"/\"rate-test-model-6c0612-kserve-self-signed-certs\"" Oct 31 13:37:53.774558 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.774506 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-6c0612\"/\"default-dockercfg-f7dfr\"" Oct 31 13:37:53.826954 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.826884 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff57031e-a357-4764-8c12-2463afb93ac7-tmp-dir\") pod \"rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng\" (UID: \"ff57031e-a357-4764-8c12-2463afb93ac7\") " pod="ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng" Oct 31 13:37:53.826954 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.826918 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff57031e-a357-4764-8c12-2463afb93ac7-model-cache\") pod \"rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng\" (UID: \"ff57031e-a357-4764-8c12-2463afb93ac7\") " pod="ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng" Oct 31 13:37:53.827134 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.827075 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5vsj\" (UniqueName: \"kubernetes.io/projected/ff57031e-a357-4764-8c12-2463afb93ac7-kube-api-access-z5vsj\") pod \"rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng\" (UID: \"ff57031e-a357-4764-8c12-2463afb93ac7\") " pod="ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng" Oct 31 13:37:53.827176 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.827130 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff57031e-a357-4764-8c12-2463afb93ac7-home\") pod \"rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng\" (UID: \"ff57031e-a357-4764-8c12-2463afb93ac7\") " pod="ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng" Oct 31 13:37:53.827268 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.827245 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff57031e-a357-4764-8c12-2463afb93ac7-tls-certs\") pod \"rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng\" (UID: \"ff57031e-a357-4764-8c12-2463afb93ac7\") " pod="ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng" Oct 31 13:37:53.827317 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.827304 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff57031e-a357-4764-8c12-2463afb93ac7-dshm\") pod \"rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng\" (UID: \"ff57031e-a357-4764-8c12-2463afb93ac7\") " pod="ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng" Oct 31 13:37:53.827498 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.827482 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5c84160-0130-4f19-a32f-1b1b0f038198-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:53.827531 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.827502 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vvsc\" (UniqueName: \"kubernetes.io/projected/b5c84160-0130-4f19-a32f-1b1b0f038198-kube-api-access-4vvsc\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:53.827531 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.827512 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5c84160-0130-4f19-a32f-1b1b0f038198-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:53.827531 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.827523 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5c84160-0130-4f19-a32f-1b1b0f038198-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:53.900408 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.900355 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:53.936263 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.935539 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff57031e-a357-4764-8c12-2463afb93ac7-home\") pod \"rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng\" (UID: \"ff57031e-a357-4764-8c12-2463afb93ac7\") " pod="ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng" Oct 31 13:37:53.936263 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.935715 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff57031e-a357-4764-8c12-2463afb93ac7-tls-certs\") pod \"rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng\" (UID: \"ff57031e-a357-4764-8c12-2463afb93ac7\") " pod="ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng" Oct 31 13:37:53.936263 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.935781 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff57031e-a357-4764-8c12-2463afb93ac7-dshm\") pod \"rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng\" (UID: \"ff57031e-a357-4764-8c12-2463afb93ac7\") " pod="ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng" Oct 31 13:37:53.936263 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.936097 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff57031e-a357-4764-8c12-2463afb93ac7-tmp-dir\") pod \"rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng\" (UID: \"ff57031e-a357-4764-8c12-2463afb93ac7\") " pod="ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng" Oct 31 13:37:53.936263 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.936140 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff57031e-a357-4764-8c12-2463afb93ac7-model-cache\") pod \"rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng\" (UID: \"ff57031e-a357-4764-8c12-2463afb93ac7\") " pod="ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng" Oct 31 13:37:53.936263 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.936247 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5vsj\" (UniqueName: \"kubernetes.io/projected/ff57031e-a357-4764-8c12-2463afb93ac7-kube-api-access-z5vsj\") pod \"rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng\" (UID: \"ff57031e-a357-4764-8c12-2463afb93ac7\") " pod="ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng" Oct 31 13:37:53.958004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.939824 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff57031e-a357-4764-8c12-2463afb93ac7-tmp-dir\") pod \"rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng\" (UID: \"ff57031e-a357-4764-8c12-2463afb93ac7\") " pod="ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng" Oct 31 13:37:53.977100 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.963340 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5vsj\" (UniqueName: \"kubernetes.io/projected/ff57031e-a357-4764-8c12-2463afb93ac7-kube-api-access-z5vsj\") pod \"rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng\" (UID: \"ff57031e-a357-4764-8c12-2463afb93ac7\") " pod="ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng" Oct 31 13:37:53.977323 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.972463 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff57031e-a357-4764-8c12-2463afb93ac7-model-cache\") pod \"rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng\" (UID: \"ff57031e-a357-4764-8c12-2463afb93ac7\") " pod="ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng" Oct 31 13:37:53.977424 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.972942 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff57031e-a357-4764-8c12-2463afb93ac7-tls-certs\") pod \"rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng\" (UID: \"ff57031e-a357-4764-8c12-2463afb93ac7\") " pod="ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng" Oct 31 13:37:53.977517 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.976295 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff57031e-a357-4764-8c12-2463afb93ac7-dshm\") pod \"rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng\" (UID: \"ff57031e-a357-4764-8c12-2463afb93ac7\") " pod="ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng" Oct 31 13:37:53.991004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:53.989051 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff57031e-a357-4764-8c12-2463afb93ac7-home\") pod \"rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng\" (UID: \"ff57031e-a357-4764-8c12-2463afb93ac7\") " pod="ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng" Oct 31 13:37:54.056042 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:54.055469 6841 generic.go:356] "Generic (PLEG): container finished" podID="b5c84160-0130-4f19-a32f-1b1b0f038198" containerID="12a237c2e1ebeadf6e51599ba3e65653a56f07b7b5673a82858e92d18b979baa" exitCode=0 Oct 31 13:37:54.056042 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:54.055687 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" event={"ID":"b5c84160-0130-4f19-a32f-1b1b0f038198","Type":"ContainerDied","Data":"12a237c2e1ebeadf6e51599ba3e65653a56f07b7b5673a82858e92d18b979baa"} Oct 31 13:37:54.056042 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:54.055772 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" event={"ID":"b5c84160-0130-4f19-a32f-1b1b0f038198","Type":"ContainerDied","Data":"6c216900131950a3c18e163d8f434f6abf65172f16d59d253a7187eb7769f99d"} Oct 31 13:37:54.056042 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:54.055813 6841 scope.go:122] "RemoveContainer" containerID="12a237c2e1ebeadf6e51599ba3e65653a56f07b7b5673a82858e92d18b979baa" Oct 31 13:37:54.058070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:54.056419 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8" Oct 31 13:37:54.081625 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:54.081558 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng" Oct 31 13:37:54.089224 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:54.087770 6841 scope.go:122] "RemoveContainer" containerID="12a237c2e1ebeadf6e51599ba3e65653a56f07b7b5673a82858e92d18b979baa" Oct 31 13:37:54.089224 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:37:54.088362 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12a237c2e1ebeadf6e51599ba3e65653a56f07b7b5673a82858e92d18b979baa\": container with ID starting with 12a237c2e1ebeadf6e51599ba3e65653a56f07b7b5673a82858e92d18b979baa not found: ID does not exist" containerID="12a237c2e1ebeadf6e51599ba3e65653a56f07b7b5673a82858e92d18b979baa" Oct 31 13:37:54.089224 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:54.088391 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12a237c2e1ebeadf6e51599ba3e65653a56f07b7b5673a82858e92d18b979baa"} err="failed to get container status \"12a237c2e1ebeadf6e51599ba3e65653a56f07b7b5673a82858e92d18b979baa\": rpc error: code = NotFound desc = could not find container \"12a237c2e1ebeadf6e51599ba3e65653a56f07b7b5673a82858e92d18b979baa\": container with ID starting with 12a237c2e1ebeadf6e51599ba3e65653a56f07b7b5673a82858e92d18b979baa not found: ID does not exist" Oct 31 13:37:54.095135 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:54.095106 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8"] Oct 31 13:37:54.104101 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:54.104075 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-pdzm8"] Oct 31 13:37:54.378539 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:54.378511 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng"] Oct 31 13:37:54.381084 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:37:54.381049 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff57031e_a357_4764_8c12_2463afb93ac7.slice/crio-118b03d22ec290f51f980b5ef852b7cdc8fa1b758e5b6463087da3fda6caf7ff WatchSource:0}: Error finding container 118b03d22ec290f51f980b5ef852b7cdc8fa1b758e5b6463087da3fda6caf7ff: Status 404 returned error can't find the container with id 118b03d22ec290f51f980b5ef852b7cdc8fa1b758e5b6463087da3fda6caf7ff Oct 31 13:37:54.740011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:54.739960 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:55.066021 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:55.065917 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng" event={"ID":"ff57031e-a357-4764-8c12-2463afb93ac7","Type":"ContainerStarted","Data":"cf5789507845241b9a6da678bd863fb88b2779e679494fdb6bbeb09d7d5dc461"} Oct 31 13:37:55.066021 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:55.065962 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng" event={"ID":"ff57031e-a357-4764-8c12-2463afb93ac7","Type":"ContainerStarted","Data":"118b03d22ec290f51f980b5ef852b7cdc8fa1b758e5b6463087da3fda6caf7ff"} Oct 31 13:37:55.082134 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:55.082107 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng" Oct 31 13:37:55.096718 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:55.096672 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng" podStartSLOduration=2.096659042 podStartE2EDuration="2.096659042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:37:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:37:55.094331373 +0000 UTC m=+3921.577454274" watchObservedRunningTime="2026-10-31 13:37:55.096659042 +0000 UTC m=+3921.579781978" Oct 31 13:37:55.558369 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:55.558333 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:55.645839 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:55.645804 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5c84160-0130-4f19-a32f-1b1b0f038198" path="/var/lib/kubelet/pods/b5c84160-0130-4f19-a32f-1b1b0f038198/volumes" Oct 31 13:37:55.829261 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:55.829178 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:56.193651 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:56.193622 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:56.814386 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:56.814346 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:57.635545 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:57.635506 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht" podUID="b413db83-3583-447b-ab12-b6a73a4769be" containerName="main" containerID="cri-o://bf9f4700bd80a86e85c6358bd9a2d882a9949f67f0e29b79c6af11a949917009" gracePeriod=45 Oct 31 13:37:57.637368 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:57.637328 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:57.688524 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:57.688490 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht" podUID="b413db83-3583-447b-ab12-b6a73a4769be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.224:8000/ready\": dial tcp 10.134.0.224:8000: connect: connection refused" Oct 31 13:37:57.931740 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:57.931714 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht" Oct 31 13:37:58.019282 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:58.019250 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b413db83-3583-447b-ab12-b6a73a4769be-kube-api-access-4ghzz\" (UniqueName: \"kubernetes.io/projected/b413db83-3583-447b-ab12-b6a73a4769be-kube-api-access-4ghzz\") pod \"b413db83-3583-447b-ab12-b6a73a4769be\" (UID: \"b413db83-3583-447b-ab12-b6a73a4769be\") " Oct 31 13:37:58.019454 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:58.019328 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b413db83-3583-447b-ab12-b6a73a4769be-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b413db83-3583-447b-ab12-b6a73a4769be-tmp-dir\") pod \"b413db83-3583-447b-ab12-b6a73a4769be\" (UID: \"b413db83-3583-447b-ab12-b6a73a4769be\") " Oct 31 13:37:58.019454 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:58.019399 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b413db83-3583-447b-ab12-b6a73a4769be-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b413db83-3583-447b-ab12-b6a73a4769be-dshm\") pod \"b413db83-3583-447b-ab12-b6a73a4769be\" (UID: \"b413db83-3583-447b-ab12-b6a73a4769be\") " Oct 31 13:37:58.019454 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:58.019429 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b413db83-3583-447b-ab12-b6a73a4769be-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b413db83-3583-447b-ab12-b6a73a4769be-model-cache\") pod \"b413db83-3583-447b-ab12-b6a73a4769be\" (UID: \"b413db83-3583-447b-ab12-b6a73a4769be\") " Oct 31 13:37:58.019652 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:58.019627 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b413db83-3583-447b-ab12-b6a73a4769be-tmp-dir" pod "b413db83-3583-447b-ab12-b6a73a4769be" (UID: "b413db83-3583-447b-ab12-b6a73a4769be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:58.019652 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:58.019645 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b413db83-3583-447b-ab12-b6a73a4769be-home\" (UniqueName: \"kubernetes.io/empty-dir/b413db83-3583-447b-ab12-b6a73a4769be-home\") pod \"b413db83-3583-447b-ab12-b6a73a4769be\" (UID: \"b413db83-3583-447b-ab12-b6a73a4769be\") " Oct 31 13:37:58.019796 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:58.019690 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b413db83-3583-447b-ab12-b6a73a4769be-tls-certs\" (UniqueName: \"kubernetes.io/secret/b413db83-3583-447b-ab12-b6a73a4769be-tls-certs\") pod \"b413db83-3583-447b-ab12-b6a73a4769be\" (UID: \"b413db83-3583-447b-ab12-b6a73a4769be\") " Oct 31 13:37:58.019796 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:58.019778 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b413db83-3583-447b-ab12-b6a73a4769be-model-cache" pod "b413db83-3583-447b-ab12-b6a73a4769be" (UID: "b413db83-3583-447b-ab12-b6a73a4769be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:58.019941 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:58.019916 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b413db83-3583-447b-ab12-b6a73a4769be-home" pod "b413db83-3583-447b-ab12-b6a73a4769be" (UID: "b413db83-3583-447b-ab12-b6a73a4769be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:58.020371 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:58.020338 6841 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b413db83-3583-447b-ab12-b6a73a4769be-home\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:58.020371 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:58.020367 6841 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b413db83-3583-447b-ab12-b6a73a4769be-tmp-dir\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:58.020577 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:58.020382 6841 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b413db83-3583-447b-ab12-b6a73a4769be-model-cache\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:58.021839 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:58.021816 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b413db83-3583-447b-ab12-b6a73a4769be-kube-api-access-4ghzz" pod "b413db83-3583-447b-ab12-b6a73a4769be" (UID: "b413db83-3583-447b-ab12-b6a73a4769be"). InnerVolumeSpecName "kube-api-access-4ghzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:58.021927 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:58.021852 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b413db83-3583-447b-ab12-b6a73a4769be-tls-certs" pod "b413db83-3583-447b-ab12-b6a73a4769be" (UID: "b413db83-3583-447b-ab12-b6a73a4769be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:37:58.021927 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:58.021868 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b413db83-3583-447b-ab12-b6a73a4769be-dshm" pod "b413db83-3583-447b-ab12-b6a73a4769be" (UID: "b413db83-3583-447b-ab12-b6a73a4769be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:58.090548 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:58.090514 6841 generic.go:356] "Generic (PLEG): container finished" podID="b413db83-3583-447b-ab12-b6a73a4769be" containerID="bf9f4700bd80a86e85c6358bd9a2d882a9949f67f0e29b79c6af11a949917009" exitCode=0 Oct 31 13:37:58.090693 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:58.090563 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht" Oct 31 13:37:58.090693 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:58.090596 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht" event={"ID":"b413db83-3583-447b-ab12-b6a73a4769be","Type":"ContainerDied","Data":"bf9f4700bd80a86e85c6358bd9a2d882a9949f67f0e29b79c6af11a949917009"} Oct 31 13:37:58.090693 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:58.090646 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht" event={"ID":"b413db83-3583-447b-ab12-b6a73a4769be","Type":"ContainerDied","Data":"1e44e4d3e70e498f819a2b807592e8a4e9aa4d33c52e1301e66150ccd30e3377"} Oct 31 13:37:58.090693 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:58.090665 6841 scope.go:122] "RemoveContainer" containerID="bf9f4700bd80a86e85c6358bd9a2d882a9949f67f0e29b79c6af11a949917009" Oct 31 13:37:58.101710 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:58.101691 6841 scope.go:122] "RemoveContainer" containerID="bf9f4700bd80a86e85c6358bd9a2d882a9949f67f0e29b79c6af11a949917009" Oct 31 13:37:58.102032 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:37:58.102013 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf9f4700bd80a86e85c6358bd9a2d882a9949f67f0e29b79c6af11a949917009\": container with ID starting with bf9f4700bd80a86e85c6358bd9a2d882a9949f67f0e29b79c6af11a949917009 not found: ID does not exist" containerID="bf9f4700bd80a86e85c6358bd9a2d882a9949f67f0e29b79c6af11a949917009" Oct 31 13:37:58.102106 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:58.102041 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf9f4700bd80a86e85c6358bd9a2d882a9949f67f0e29b79c6af11a949917009"} err="failed to get container status \"bf9f4700bd80a86e85c6358bd9a2d882a9949f67f0e29b79c6af11a949917009\": rpc error: code = NotFound desc = could not find container \"bf9f4700bd80a86e85c6358bd9a2d882a9949f67f0e29b79c6af11a949917009\": container with ID starting with bf9f4700bd80a86e85c6358bd9a2d882a9949f67f0e29b79c6af11a949917009 not found: ID does not exist" Oct 31 13:37:58.121498 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:58.121476 6841 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b413db83-3583-447b-ab12-b6a73a4769be-dshm\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:58.121498 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:58.121498 6841 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b413db83-3583-447b-ab12-b6a73a4769be-tls-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:58.121647 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:58.121509 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ghzz\" (UniqueName: \"kubernetes.io/projected/b413db83-3583-447b-ab12-b6a73a4769be-kube-api-access-4ghzz\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:58.155553 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:58.155487 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht"] Oct 31 13:37:58.159035 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:58.159015 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-50b4aa/test-model-50b4aa-kserve-5b55694797-bvhht"] Oct 31 13:37:58.834793 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:58.834749 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:59.584863 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:59.584825 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:37:59.644016 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:59.643969 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b413db83-3583-447b-ab12-b6a73a4769be" path="/var/lib/kubelet/pods/b413db83-3583-447b-ab12-b6a73a4769be/volumes" Oct 31 13:37:59.834721 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:37:59.834686 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:00.425685 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:00.425647 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:01.181875 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:01.181834 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:02.003586 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:02.003551 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:02.610340 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:02.610305 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:02.975014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:02.974951 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:03.736637 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:03.736593 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:04.082844 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:04.082760 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng" Oct 31 13:38:04.095851 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:04.095820 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng" Oct 31 13:38:04.166810 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:04.166782 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng" Oct 31 13:38:04.402603 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:04.402506 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g"] Oct 31 13:38:04.403323 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:04.403303 6841 cpu_manager.go:414] "containerMap: removing container" podUID="b413db83-3583-447b-ab12-b6a73a4769be" containerName="main" Oct 31 13:38:04.403323 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:04.403321 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b413db83-3583-447b-ab12-b6a73a4769be" containerName="main" Oct 31 13:38:04.403511 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:04.403455 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="b413db83-3583-447b-ab12-b6a73a4769be" containerName="main" Oct 31 13:38:04.407247 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:04.407225 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g" Oct 31 13:38:04.411717 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:04.411693 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-45bcad\"/\"kube-root-ca.crt\"" Oct 31 13:38:04.412088 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:04.412063 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-45bcad\"/\"openshift-service-ca.crt\"" Oct 31 13:38:04.412924 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:04.412900 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-45bcad\"/\"default-dockercfg-kh96p\"" Oct 31 13:38:04.419977 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:04.419960 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-45bcad\"/\"sub-test-model-45bcad-kserve-self-signed-certs\"" Oct 31 13:38:04.430328 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:04.430301 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g"] Oct 31 13:38:04.527949 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:04.527909 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c017a13d-2772-4379-80e6-6ce0d37af2c5-home\") pod \"sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g\" (UID: \"c017a13d-2772-4379-80e6-6ce0d37af2c5\") " pod="ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g" Oct 31 13:38:04.528138 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:04.528043 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c017a13d-2772-4379-80e6-6ce0d37af2c5-tmp-dir\") pod \"sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g\" (UID: \"c017a13d-2772-4379-80e6-6ce0d37af2c5\") " pod="ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g" Oct 31 13:38:04.528227 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:04.528211 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c017a13d-2772-4379-80e6-6ce0d37af2c5-tls-certs\") pod \"sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g\" (UID: \"c017a13d-2772-4379-80e6-6ce0d37af2c5\") " pod="ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g" Oct 31 13:38:04.528267 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:04.528254 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c017a13d-2772-4379-80e6-6ce0d37af2c5-model-cache\") pod \"sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g\" (UID: \"c017a13d-2772-4379-80e6-6ce0d37af2c5\") " pod="ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g" Oct 31 13:38:04.528305 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:04.528283 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9gbm\" (UniqueName: \"kubernetes.io/projected/c017a13d-2772-4379-80e6-6ce0d37af2c5-kube-api-access-s9gbm\") pod \"sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g\" (UID: \"c017a13d-2772-4379-80e6-6ce0d37af2c5\") " pod="ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g" Oct 31 13:38:04.528444 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:04.528423 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c017a13d-2772-4379-80e6-6ce0d37af2c5-dshm\") pod \"sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g\" (UID: \"c017a13d-2772-4379-80e6-6ce0d37af2c5\") " pod="ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g" Oct 31 13:38:04.562035 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:04.562000 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:04.631279 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:04.631230 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c017a13d-2772-4379-80e6-6ce0d37af2c5-home\") pod \"sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g\" (UID: \"c017a13d-2772-4379-80e6-6ce0d37af2c5\") " pod="ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g" Oct 31 13:38:04.631625 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:04.631598 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c017a13d-2772-4379-80e6-6ce0d37af2c5-tmp-dir\") pod \"sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g\" (UID: \"c017a13d-2772-4379-80e6-6ce0d37af2c5\") " pod="ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g" Oct 31 13:38:04.632067 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:04.632047 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c017a13d-2772-4379-80e6-6ce0d37af2c5-tls-certs\") pod \"sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g\" (UID: \"c017a13d-2772-4379-80e6-6ce0d37af2c5\") " pod="ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g" Oct 31 13:38:04.632225 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:04.632210 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c017a13d-2772-4379-80e6-6ce0d37af2c5-model-cache\") pod \"sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g\" (UID: \"c017a13d-2772-4379-80e6-6ce0d37af2c5\") " pod="ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g" Oct 31 13:38:04.632343 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:04.632326 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9gbm\" (UniqueName: \"kubernetes.io/projected/c017a13d-2772-4379-80e6-6ce0d37af2c5-kube-api-access-s9gbm\") pod \"sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g\" (UID: \"c017a13d-2772-4379-80e6-6ce0d37af2c5\") " pod="ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g" Oct 31 13:38:04.632519 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:04.632505 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c017a13d-2772-4379-80e6-6ce0d37af2c5-dshm\") pod \"sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g\" (UID: \"c017a13d-2772-4379-80e6-6ce0d37af2c5\") " pod="ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g" Oct 31 13:38:04.653059 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:04.639222 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c017a13d-2772-4379-80e6-6ce0d37af2c5-home\") pod \"sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g\" (UID: \"c017a13d-2772-4379-80e6-6ce0d37af2c5\") " pod="ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g" Oct 31 13:38:04.653059 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:04.639453 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c017a13d-2772-4379-80e6-6ce0d37af2c5-tmp-dir\") pod \"sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g\" (UID: \"c017a13d-2772-4379-80e6-6ce0d37af2c5\") " pod="ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g" Oct 31 13:38:04.653059 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:04.639662 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c017a13d-2772-4379-80e6-6ce0d37af2c5-model-cache\") pod \"sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g\" (UID: \"c017a13d-2772-4379-80e6-6ce0d37af2c5\") " pod="ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g" Oct 31 13:38:04.678017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:04.675466 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c017a13d-2772-4379-80e6-6ce0d37af2c5-dshm\") pod \"sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g\" (UID: \"c017a13d-2772-4379-80e6-6ce0d37af2c5\") " pod="ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g" Oct 31 13:38:04.678017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:04.676667 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c017a13d-2772-4379-80e6-6ce0d37af2c5-tls-certs\") pod \"sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g\" (UID: \"c017a13d-2772-4379-80e6-6ce0d37af2c5\") " pod="ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g" Oct 31 13:38:04.678262 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:04.678202 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9gbm\" (UniqueName: \"kubernetes.io/projected/c017a13d-2772-4379-80e6-6ce0d37af2c5-kube-api-access-s9gbm\") pod \"sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g\" (UID: \"c017a13d-2772-4379-80e6-6ce0d37af2c5\") " pod="ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g" Oct 31 13:38:04.724246 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:04.722943 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g" Oct 31 13:38:04.785847 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:04.781862 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:05.019922 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:38:05.019881 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc017a13d_2772_4379_80e6_6ce0d37af2c5.slice/crio-73e5c992bae9755f9107ba9d8f9e1afead07b33623b166be7f57d42bb015ed72 WatchSource:0}: Error finding container 73e5c992bae9755f9107ba9d8f9e1afead07b33623b166be7f57d42bb015ed72: Status 404 returned error can't find the container with id 73e5c992bae9755f9107ba9d8f9e1afead07b33623b166be7f57d42bb015ed72 Oct 31 13:38:05.020250 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:05.020229 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g"] Oct 31 13:38:05.164854 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:05.164814 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g" event={"ID":"c017a13d-2772-4379-80e6-6ce0d37af2c5","Type":"ContainerStarted","Data":"7f954cf752d58d0a71b24b8f7a69fa208026242bd56aeae58fb6ce2d1575ed71"} Oct 31 13:38:05.165094 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:05.164863 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g" event={"ID":"c017a13d-2772-4379-80e6-6ce0d37af2c5","Type":"ContainerStarted","Data":"73e5c992bae9755f9107ba9d8f9e1afead07b33623b166be7f57d42bb015ed72"} Oct 31 13:38:05.187952 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:05.187901 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g" podStartSLOduration=1.187887832 podStartE2EDuration="1.187887832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:38:05.183423015 +0000 UTC m=+3931.666545916" watchObservedRunningTime="2026-10-31 13:38:05.187887832 +0000 UTC m=+3931.671010732" Oct 31 13:38:05.432810 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:05.432712 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:05.723170 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:05.723138 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g" Oct 31 13:38:06.422312 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:06.422277 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:07.361165 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:07.361131 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:08.262827 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:08.262798 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:09.095528 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:09.095492 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:09.510597 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:09.510565 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:10.585648 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:10.585613 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:11.841036 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:11.840997 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:12.836080 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:12.836041 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:14.083863 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:14.083815 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:14.723925 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:14.723873 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g" Oct 31 13:38:14.736090 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:14.736061 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g" Oct 31 13:38:14.925104 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:14.925069 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:15.286737 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:15.286707 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g" Oct 31 13:38:15.824822 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:15.824783 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:16.139858 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:16.139775 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh"] Oct 31 13:38:16.140041 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:16.140000 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" podUID="a8f62f2e-7128-4109-af26-c1d6955b83bf" containerName="istio-proxy" containerID="cri-o://80d722ca5500ec6b228191657f45924adc6a807c1bc0f77d0d3754736a97bbaa" gracePeriod=30 Oct 31 13:38:16.716544 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:16.716509 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:16.994425 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:16.994345 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:17.350417 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:17.350346 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:17.888008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:17.887659 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:18.157090 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:18.156966 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:18.692573 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:18.692539 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:18.969504 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:18.969413 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:19.192489 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:19.192436 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:19.687953 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:19.687920 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8"] Oct 31 13:38:19.688158 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:19.688136 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" podUID="a38e68ad-a968-4534-8135-8bb0721098f4" containerName="istio-proxy" containerID="cri-o://6d1a10ae7b1289038f925c85c13bb20d1d5cce3f1a0b16174aa4190367a80a64" gracePeriod=30 Oct 31 13:38:19.743808 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:19.743773 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:19.958611 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:19.958579 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:20.037640 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:38:20.037607 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:38:20.090639 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:20.090600 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:38:20.090797 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:20.090669 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" podUID="a8f62f2e-7128-4109-af26-c1d6955b83bf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:38:20.834575 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:20.834535 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:21.354225 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:21.354190 6841 generic.go:356] "Generic (PLEG): container finished" podID="a8f62f2e-7128-4109-af26-c1d6955b83bf" containerID="80d722ca5500ec6b228191657f45924adc6a807c1bc0f77d0d3754736a97bbaa" exitCode=0 Oct 31 13:38:21.354698 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:21.354268 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" event={"ID":"a8f62f2e-7128-4109-af26-c1d6955b83bf","Type":"ContainerDied","Data":"80d722ca5500ec6b228191657f45924adc6a807c1bc0f77d0d3754736a97bbaa"} Oct 31 13:38:21.354698 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:21.354343 6841 scope.go:122] "RemoveContainer" containerID="d90cb5a9db1c02f1985e9a1dd8da4877aaa655169c80036ffea2598c4bd755e8" Oct 31 13:38:21.423603 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:21.423545 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:38:21.509056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:21.509027 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8f62f2e-7128-4109-af26-c1d6955b83bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f62f2e-7128-4109-af26-c1d6955b83bf-workload-socket\") pod \"a8f62f2e-7128-4109-af26-c1d6955b83bf\" (UID: \"a8f62f2e-7128-4109-af26-c1d6955b83bf\") " Oct 31 13:38:21.509214 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:21.509090 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8f62f2e-7128-4109-af26-c1d6955b83bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8f62f2e-7128-4109-af26-c1d6955b83bf-istiod-ca-cert\") pod \"a8f62f2e-7128-4109-af26-c1d6955b83bf\" (UID: \"a8f62f2e-7128-4109-af26-c1d6955b83bf\") " Oct 31 13:38:21.509214 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:21.509206 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8f62f2e-7128-4109-af26-c1d6955b83bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8f62f2e-7128-4109-af26-c1d6955b83bf-workload-certs\") pod \"a8f62f2e-7128-4109-af26-c1d6955b83bf\" (UID: \"a8f62f2e-7128-4109-af26-c1d6955b83bf\") " Oct 31 13:38:21.509308 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:21.509230 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8f62f2e-7128-4109-af26-c1d6955b83bf-kube-api-access-xhpvl\" (UniqueName: \"kubernetes.io/projected/a8f62f2e-7128-4109-af26-c1d6955b83bf-kube-api-access-xhpvl\") pod \"a8f62f2e-7128-4109-af26-c1d6955b83bf\" (UID: \"a8f62f2e-7128-4109-af26-c1d6955b83bf\") " Oct 31 13:38:21.509308 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:21.509296 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8f62f2e-7128-4109-af26-c1d6955b83bf-istio-token\" (UniqueName: \"kubernetes.io/projected/a8f62f2e-7128-4109-af26-c1d6955b83bf-istio-token\") pod \"a8f62f2e-7128-4109-af26-c1d6955b83bf\" (UID: \"a8f62f2e-7128-4109-af26-c1d6955b83bf\") " Oct 31 13:38:21.509391 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:21.509313 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f62f2e-7128-4109-af26-c1d6955b83bf-workload-socket" pod "a8f62f2e-7128-4109-af26-c1d6955b83bf" (UID: "a8f62f2e-7128-4109-af26-c1d6955b83bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:38:21.509391 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:21.509366 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8f62f2e-7128-4109-af26-c1d6955b83bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f62f2e-7128-4109-af26-c1d6955b83bf-credential-socket\") pod \"a8f62f2e-7128-4109-af26-c1d6955b83bf\" (UID: \"a8f62f2e-7128-4109-af26-c1d6955b83bf\") " Oct 31 13:38:21.509490 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:21.509403 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8f62f2e-7128-4109-af26-c1d6955b83bf-istiod-ca-cert" pod "a8f62f2e-7128-4109-af26-c1d6955b83bf" (UID: "a8f62f2e-7128-4109-af26-c1d6955b83bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:38:21.509490 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:21.509454 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f62f2e-7128-4109-af26-c1d6955b83bf-workload-certs" pod "a8f62f2e-7128-4109-af26-c1d6955b83bf" (UID: "a8f62f2e-7128-4109-af26-c1d6955b83bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:38:21.509490 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:21.509458 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8f62f2e-7128-4109-af26-c1d6955b83bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8f62f2e-7128-4109-af26-c1d6955b83bf-istio-envoy\") pod \"a8f62f2e-7128-4109-af26-c1d6955b83bf\" (UID: \"a8f62f2e-7128-4109-af26-c1d6955b83bf\") " Oct 31 13:38:21.509608 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:21.509563 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8f62f2e-7128-4109-af26-c1d6955b83bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8f62f2e-7128-4109-af26-c1d6955b83bf-istio-data\") pod \"a8f62f2e-7128-4109-af26-c1d6955b83bf\" (UID: \"a8f62f2e-7128-4109-af26-c1d6955b83bf\") " Oct 31 13:38:21.509693 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:21.509661 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8f62f2e-7128-4109-af26-c1d6955b83bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8f62f2e-7128-4109-af26-c1d6955b83bf-istio-podinfo\") pod \"a8f62f2e-7128-4109-af26-c1d6955b83bf\" (UID: \"a8f62f2e-7128-4109-af26-c1d6955b83bf\") " Oct 31 13:38:21.509770 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:21.509688 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f62f2e-7128-4109-af26-c1d6955b83bf-credential-socket" pod "a8f62f2e-7128-4109-af26-c1d6955b83bf" (UID: "a8f62f2e-7128-4109-af26-c1d6955b83bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:38:21.509862 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:21.509838 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f62f2e-7128-4109-af26-c1d6955b83bf-istio-data" pod "a8f62f2e-7128-4109-af26-c1d6955b83bf" (UID: "a8f62f2e-7128-4109-af26-c1d6955b83bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:38:21.510385 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:21.510363 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f62f2e-7128-4109-af26-c1d6955b83bf-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:21.510455 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:21.510392 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8f62f2e-7128-4109-af26-c1d6955b83bf-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:21.510455 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:21.510407 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f62f2e-7128-4109-af26-c1d6955b83bf-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:21.510455 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:21.510424 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8f62f2e-7128-4109-af26-c1d6955b83bf-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:21.510455 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:21.510440 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8f62f2e-7128-4109-af26-c1d6955b83bf-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:21.512029 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:21.512002 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f62f2e-7128-4109-af26-c1d6955b83bf-istio-envoy" pod "a8f62f2e-7128-4109-af26-c1d6955b83bf" (UID: "a8f62f2e-7128-4109-af26-c1d6955b83bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:38:21.512542 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:21.512024 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8f62f2e-7128-4109-af26-c1d6955b83bf-kube-api-access-xhpvl" pod "a8f62f2e-7128-4109-af26-c1d6955b83bf" (UID: "a8f62f2e-7128-4109-af26-c1d6955b83bf"). InnerVolumeSpecName "kube-api-access-xhpvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:38:21.512542 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:21.512492 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8f62f2e-7128-4109-af26-c1d6955b83bf-istio-podinfo" pod "a8f62f2e-7128-4109-af26-c1d6955b83bf" (UID: "a8f62f2e-7128-4109-af26-c1d6955b83bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:38:21.513678 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:21.513653 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8f62f2e-7128-4109-af26-c1d6955b83bf-istio-token" pod "a8f62f2e-7128-4109-af26-c1d6955b83bf" (UID: "a8f62f2e-7128-4109-af26-c1d6955b83bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:38:21.611907 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:21.611866 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhpvl\" (UniqueName: \"kubernetes.io/projected/a8f62f2e-7128-4109-af26-c1d6955b83bf-kube-api-access-xhpvl\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:21.611907 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:21.611896 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8f62f2e-7128-4109-af26-c1d6955b83bf-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:21.611907 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:21.611905 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8f62f2e-7128-4109-af26-c1d6955b83bf-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:21.612203 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:21.611923 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8f62f2e-7128-4109-af26-c1d6955b83bf-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:21.782731 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:21.782690 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:22.362725 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:22.362691 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" Oct 31 13:38:22.363223 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:22.362730 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh" event={"ID":"a8f62f2e-7128-4109-af26-c1d6955b83bf","Type":"ContainerDied","Data":"9ec31381a20a79c08f38f19cde663cfdce95eb6bea80a1e9c8981e8d35963537"} Oct 31 13:38:22.363223 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:22.362791 6841 scope.go:122] "RemoveContainer" containerID="80d722ca5500ec6b228191657f45924adc6a807c1bc0f77d0d3754736a97bbaa" Oct 31 13:38:22.393131 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:22.393097 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh"] Oct 31 13:38:22.398973 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:22.398942 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-knzzh"] Oct 31 13:38:22.666905 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:22.666816 6841 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:38:22.666905 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:22.666885 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" podUID="a38e68ad-a968-4534-8135-8bb0721098f4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:38:22.703665 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:22.703633 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:23.475590 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:23.475549 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:23.644733 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:23.644700 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8f62f2e-7128-4109-af26-c1d6955b83bf" path="/var/lib/kubelet/pods/a8f62f2e-7128-4109-af26-c1d6955b83bf/volumes" Oct 31 13:38:24.420179 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:24.420143 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:24.979548 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:24.979480 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" Oct 31 13:38:25.071146 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.071112 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a38e68ad-a968-4534-8135-8bb0721098f4-kube-api-access-6fdrm\" (UniqueName: \"kubernetes.io/projected/a38e68ad-a968-4534-8135-8bb0721098f4-kube-api-access-6fdrm\") pod \"a38e68ad-a968-4534-8135-8bb0721098f4\" (UID: \"a38e68ad-a968-4534-8135-8bb0721098f4\") " Oct 31 13:38:25.071354 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.071212 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a38e68ad-a968-4534-8135-8bb0721098f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a38e68ad-a968-4534-8135-8bb0721098f4-istiod-ca-cert\") pod \"a38e68ad-a968-4534-8135-8bb0721098f4\" (UID: \"a38e68ad-a968-4534-8135-8bb0721098f4\") " Oct 31 13:38:25.071354 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.071322 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a38e68ad-a968-4534-8135-8bb0721098f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38e68ad-a968-4534-8135-8bb0721098f4-workload-socket\") pod \"a38e68ad-a968-4534-8135-8bb0721098f4\" (UID: \"a38e68ad-a968-4534-8135-8bb0721098f4\") " Oct 31 13:38:25.071457 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.071405 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a38e68ad-a968-4534-8135-8bb0721098f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38e68ad-a968-4534-8135-8bb0721098f4-credential-socket\") pod \"a38e68ad-a968-4534-8135-8bb0721098f4\" (UID: \"a38e68ad-a968-4534-8135-8bb0721098f4\") " Oct 31 13:38:25.071457 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.071446 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a38e68ad-a968-4534-8135-8bb0721098f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a38e68ad-a968-4534-8135-8bb0721098f4-istio-podinfo\") pod \"a38e68ad-a968-4534-8135-8bb0721098f4\" (UID: \"a38e68ad-a968-4534-8135-8bb0721098f4\") " Oct 31 13:38:25.071579 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.071518 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a38e68ad-a968-4534-8135-8bb0721098f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a38e68ad-a968-4534-8135-8bb0721098f4-istio-envoy\") pod \"a38e68ad-a968-4534-8135-8bb0721098f4\" (UID: \"a38e68ad-a968-4534-8135-8bb0721098f4\") " Oct 31 13:38:25.071645 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.071584 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a38e68ad-a968-4534-8135-8bb0721098f4-istiod-ca-cert" pod "a38e68ad-a968-4534-8135-8bb0721098f4" (UID: "a38e68ad-a968-4534-8135-8bb0721098f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:38:25.071720 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.071585 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a38e68ad-a968-4534-8135-8bb0721098f4-workload-socket" pod "a38e68ad-a968-4534-8135-8bb0721098f4" (UID: "a38e68ad-a968-4534-8135-8bb0721098f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:38:25.071720 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.071627 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a38e68ad-a968-4534-8135-8bb0721098f4-istio-token\" (UniqueName: \"kubernetes.io/projected/a38e68ad-a968-4534-8135-8bb0721098f4-istio-token\") pod \"a38e68ad-a968-4534-8135-8bb0721098f4\" (UID: \"a38e68ad-a968-4534-8135-8bb0721098f4\") " Oct 31 13:38:25.071933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.071666 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a38e68ad-a968-4534-8135-8bb0721098f4-credential-socket" pod "a38e68ad-a968-4534-8135-8bb0721098f4" (UID: "a38e68ad-a968-4534-8135-8bb0721098f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:38:25.071933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.071824 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a38e68ad-a968-4534-8135-8bb0721098f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a38e68ad-a968-4534-8135-8bb0721098f4-workload-certs\") pod \"a38e68ad-a968-4534-8135-8bb0721098f4\" (UID: \"a38e68ad-a968-4534-8135-8bb0721098f4\") " Oct 31 13:38:25.071933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.071892 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a38e68ad-a968-4534-8135-8bb0721098f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a38e68ad-a968-4534-8135-8bb0721098f4-istio-data\") pod \"a38e68ad-a968-4534-8135-8bb0721098f4\" (UID: \"a38e68ad-a968-4534-8135-8bb0721098f4\") " Oct 31 13:38:25.072162 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.072091 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a38e68ad-a968-4534-8135-8bb0721098f4-workload-certs" pod "a38e68ad-a968-4534-8135-8bb0721098f4" (UID: "a38e68ad-a968-4534-8135-8bb0721098f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:38:25.072797 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.072772 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a38e68ad-a968-4534-8135-8bb0721098f4-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:25.072882 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.072804 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a38e68ad-a968-4534-8135-8bb0721098f4-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:25.072882 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.072821 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38e68ad-a968-4534-8135-8bb0721098f4-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:25.072882 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.072832 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a38e68ad-a968-4534-8135-8bb0721098f4-istio-data" pod "a38e68ad-a968-4534-8135-8bb0721098f4" (UID: "a38e68ad-a968-4534-8135-8bb0721098f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:38:25.072882 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.072837 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38e68ad-a968-4534-8135-8bb0721098f4-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:25.074021 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.073963 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a38e68ad-a968-4534-8135-8bb0721098f4-istio-envoy" pod "a38e68ad-a968-4534-8135-8bb0721098f4" (UID: "a38e68ad-a968-4534-8135-8bb0721098f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:38:25.074096 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.073973 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a38e68ad-a968-4534-8135-8bb0721098f4-istio-token" pod "a38e68ad-a968-4534-8135-8bb0721098f4" (UID: "a38e68ad-a968-4534-8135-8bb0721098f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:38:25.074096 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.074009 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a38e68ad-a968-4534-8135-8bb0721098f4-kube-api-access-6fdrm" pod "a38e68ad-a968-4534-8135-8bb0721098f4" (UID: "a38e68ad-a968-4534-8135-8bb0721098f4"). InnerVolumeSpecName "kube-api-access-6fdrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:38:25.074096 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.074084 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a38e68ad-a968-4534-8135-8bb0721098f4-istio-podinfo" pod "a38e68ad-a968-4534-8135-8bb0721098f4" (UID: "a38e68ad-a968-4534-8135-8bb0721098f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:38:25.184070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.180746 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a38e68ad-a968-4534-8135-8bb0721098f4-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:25.184070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.180785 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a38e68ad-a968-4534-8135-8bb0721098f4-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:25.184070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.180806 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fdrm\" (UniqueName: \"kubernetes.io/projected/a38e68ad-a968-4534-8135-8bb0721098f4-kube-api-access-6fdrm\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:25.184070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.180826 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a38e68ad-a968-4534-8135-8bb0721098f4-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:25.184070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.180840 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a38e68ad-a968-4534-8135-8bb0721098f4-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:25.227365 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.227331 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:25.388321 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.388234 6841 generic.go:356] "Generic (PLEG): container finished" podID="a38e68ad-a968-4534-8135-8bb0721098f4" containerID="6d1a10ae7b1289038f925c85c13bb20d1d5cce3f1a0b16174aa4190367a80a64" exitCode=0 Oct 31 13:38:25.388321 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.388281 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" Oct 31 13:38:25.388321 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.388312 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" event={"ID":"a38e68ad-a968-4534-8135-8bb0721098f4","Type":"ContainerDied","Data":"6d1a10ae7b1289038f925c85c13bb20d1d5cce3f1a0b16174aa4190367a80a64"} Oct 31 13:38:25.388590 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.388363 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8" event={"ID":"a38e68ad-a968-4534-8135-8bb0721098f4","Type":"ContainerDied","Data":"8e9d11a40212592c448a37f3b95bc1e50f05376f37a88826fa3778c1b1e99d96"} Oct 31 13:38:25.388590 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.388384 6841 scope.go:122] "RemoveContainer" containerID="6d1a10ae7b1289038f925c85c13bb20d1d5cce3f1a0b16174aa4190367a80a64" Oct 31 13:38:25.401477 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.401450 6841 scope.go:122] "RemoveContainer" containerID="6d1a10ae7b1289038f925c85c13bb20d1d5cce3f1a0b16174aa4190367a80a64" Oct 31 13:38:25.401781 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:38:25.401759 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d1a10ae7b1289038f925c85c13bb20d1d5cce3f1a0b16174aa4190367a80a64\": container with ID starting with 6d1a10ae7b1289038f925c85c13bb20d1d5cce3f1a0b16174aa4190367a80a64 not found: ID does not exist" containerID="6d1a10ae7b1289038f925c85c13bb20d1d5cce3f1a0b16174aa4190367a80a64" Oct 31 13:38:25.401835 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.401791 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d1a10ae7b1289038f925c85c13bb20d1d5cce3f1a0b16174aa4190367a80a64"} err="failed to get container status \"6d1a10ae7b1289038f925c85c13bb20d1d5cce3f1a0b16174aa4190367a80a64\": rpc error: code = NotFound desc = could not find container \"6d1a10ae7b1289038f925c85c13bb20d1d5cce3f1a0b16174aa4190367a80a64\": container with ID starting with 6d1a10ae7b1289038f925c85c13bb20d1d5cce3f1a0b16174aa4190367a80a64 not found: ID does not exist" Oct 31 13:38:25.416618 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.416586 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8"] Oct 31 13:38:25.424531 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.424502 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-5g6z8"] Oct 31 13:38:25.648966 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.648873 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a38e68ad-a968-4534-8135-8bb0721098f4" path="/var/lib/kubelet/pods/a38e68ad-a968-4534-8135-8bb0721098f4/volumes" Oct 31 13:38:25.836136 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:25.836100 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:26.081042 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:26.080972 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:26.308437 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:26.308401 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:26.566689 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:26.566655 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:27.676969 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:27.676932 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:28.056444 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:28.056403 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:28.446281 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:28.446245 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:29.254599 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:29.254568 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:29.936952 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:29.936915 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:30.280516 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.280483 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh"] Oct 31 13:38:30.281260 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.281243 6841 cpu_manager.go:414] "containerMap: removing container" podUID="a8f62f2e-7128-4109-af26-c1d6955b83bf" containerName="istio-proxy" Oct 31 13:38:30.281260 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.281260 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8f62f2e-7128-4109-af26-c1d6955b83bf" containerName="istio-proxy" Oct 31 13:38:30.281366 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.281279 6841 cpu_manager.go:414] "containerMap: removing container" podUID="a8f62f2e-7128-4109-af26-c1d6955b83bf" containerName="istio-proxy" Oct 31 13:38:30.281366 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.281285 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8f62f2e-7128-4109-af26-c1d6955b83bf" containerName="istio-proxy" Oct 31 13:38:30.281366 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.281304 6841 cpu_manager.go:414] "containerMap: removing container" podUID="a38e68ad-a968-4534-8135-8bb0721098f4" containerName="istio-proxy" Oct 31 13:38:30.281366 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.281310 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a38e68ad-a968-4534-8135-8bb0721098f4" containerName="istio-proxy" Oct 31 13:38:30.281517 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.281420 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8f62f2e-7128-4109-af26-c1d6955b83bf" containerName="istio-proxy" Oct 31 13:38:30.281517 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.281433 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8f62f2e-7128-4109-af26-c1d6955b83bf" containerName="istio-proxy" Oct 31 13:38:30.281517 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.281444 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="a38e68ad-a968-4534-8135-8bb0721098f4" containerName="istio-proxy" Oct 31 13:38:30.286755 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.286733 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh" Oct 31 13:38:30.295036 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.294968 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-569075\"/\"kube-root-ca.crt\"" Oct 31 13:38:30.295198 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.295173 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-569075\"/\"openshift-service-ca.crt\"" Oct 31 13:38:30.295309 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.295233 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-569075\"/\"default-dockercfg-hb9tc\"" Oct 31 13:38:30.295388 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.295327 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-569075\"/\"sub-test-model-569075-kserve-self-signed-certs\"" Oct 31 13:38:30.302904 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.302881 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh"] Oct 31 13:38:30.375768 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.375731 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf8ks\" (UniqueName: \"kubernetes.io/projected/12a97fe1-6ffc-4359-a6f7-063d6c065e31-kube-api-access-bf8ks\") pod \"sub-test-model-569075-kserve-696f786fbf-f56sh\" (UID: \"12a97fe1-6ffc-4359-a6f7-063d6c065e31\") " pod="ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh" Oct 31 13:38:30.375925 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.375799 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12a97fe1-6ffc-4359-a6f7-063d6c065e31-model-cache\") pod \"sub-test-model-569075-kserve-696f786fbf-f56sh\" (UID: \"12a97fe1-6ffc-4359-a6f7-063d6c065e31\") " pod="ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh" Oct 31 13:38:30.375925 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.375904 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12a97fe1-6ffc-4359-a6f7-063d6c065e31-tls-certs\") pod \"sub-test-model-569075-kserve-696f786fbf-f56sh\" (UID: \"12a97fe1-6ffc-4359-a6f7-063d6c065e31\") " pod="ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh" Oct 31 13:38:30.376083 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.376064 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12a97fe1-6ffc-4359-a6f7-063d6c065e31-home\") pod \"sub-test-model-569075-kserve-696f786fbf-f56sh\" (UID: \"12a97fe1-6ffc-4359-a6f7-063d6c065e31\") " pod="ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh" Oct 31 13:38:30.376121 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.376113 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12a97fe1-6ffc-4359-a6f7-063d6c065e31-tmp-dir\") pod \"sub-test-model-569075-kserve-696f786fbf-f56sh\" (UID: \"12a97fe1-6ffc-4359-a6f7-063d6c065e31\") " pod="ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh" Oct 31 13:38:30.376299 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.376286 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12a97fe1-6ffc-4359-a6f7-063d6c065e31-dshm\") pod \"sub-test-model-569075-kserve-696f786fbf-f56sh\" (UID: \"12a97fe1-6ffc-4359-a6f7-063d6c065e31\") " pod="ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh" Oct 31 13:38:30.413429 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.413397 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:30.495576 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.491474 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12a97fe1-6ffc-4359-a6f7-063d6c065e31-dshm\") pod \"sub-test-model-569075-kserve-696f786fbf-f56sh\" (UID: \"12a97fe1-6ffc-4359-a6f7-063d6c065e31\") " pod="ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh" Oct 31 13:38:30.495576 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.491573 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf8ks\" (UniqueName: \"kubernetes.io/projected/12a97fe1-6ffc-4359-a6f7-063d6c065e31-kube-api-access-bf8ks\") pod \"sub-test-model-569075-kserve-696f786fbf-f56sh\" (UID: \"12a97fe1-6ffc-4359-a6f7-063d6c065e31\") " pod="ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh" Oct 31 13:38:30.495576 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.491640 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12a97fe1-6ffc-4359-a6f7-063d6c065e31-model-cache\") pod \"sub-test-model-569075-kserve-696f786fbf-f56sh\" (UID: \"12a97fe1-6ffc-4359-a6f7-063d6c065e31\") " pod="ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh" Oct 31 13:38:30.495576 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.491685 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12a97fe1-6ffc-4359-a6f7-063d6c065e31-tls-certs\") pod \"sub-test-model-569075-kserve-696f786fbf-f56sh\" (UID: \"12a97fe1-6ffc-4359-a6f7-063d6c065e31\") " pod="ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh" Oct 31 13:38:30.495576 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.491799 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12a97fe1-6ffc-4359-a6f7-063d6c065e31-home\") pod \"sub-test-model-569075-kserve-696f786fbf-f56sh\" (UID: \"12a97fe1-6ffc-4359-a6f7-063d6c065e31\") " pod="ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh" Oct 31 13:38:30.495576 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.491855 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12a97fe1-6ffc-4359-a6f7-063d6c065e31-tmp-dir\") pod \"sub-test-model-569075-kserve-696f786fbf-f56sh\" (UID: \"12a97fe1-6ffc-4359-a6f7-063d6c065e31\") " pod="ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh" Oct 31 13:38:30.495576 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.492332 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12a97fe1-6ffc-4359-a6f7-063d6c065e31-tmp-dir\") pod \"sub-test-model-569075-kserve-696f786fbf-f56sh\" (UID: \"12a97fe1-6ffc-4359-a6f7-063d6c065e31\") " pod="ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh" Oct 31 13:38:30.495576 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.492668 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12a97fe1-6ffc-4359-a6f7-063d6c065e31-home\") pod \"sub-test-model-569075-kserve-696f786fbf-f56sh\" (UID: \"12a97fe1-6ffc-4359-a6f7-063d6c065e31\") " pod="ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh" Oct 31 13:38:30.495576 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.493232 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12a97fe1-6ffc-4359-a6f7-063d6c065e31-model-cache\") pod \"sub-test-model-569075-kserve-696f786fbf-f56sh\" (UID: \"12a97fe1-6ffc-4359-a6f7-063d6c065e31\") " pod="ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh" Oct 31 13:38:30.530052 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.511854 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12a97fe1-6ffc-4359-a6f7-063d6c065e31-tls-certs\") pod \"sub-test-model-569075-kserve-696f786fbf-f56sh\" (UID: \"12a97fe1-6ffc-4359-a6f7-063d6c065e31\") " pod="ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh" Oct 31 13:38:30.541581 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.541270 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12a97fe1-6ffc-4359-a6f7-063d6c065e31-dshm\") pod \"sub-test-model-569075-kserve-696f786fbf-f56sh\" (UID: \"12a97fe1-6ffc-4359-a6f7-063d6c065e31\") " pod="ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh" Oct 31 13:38:30.545999 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.542511 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf8ks\" (UniqueName: \"kubernetes.io/projected/12a97fe1-6ffc-4359-a6f7-063d6c065e31-kube-api-access-bf8ks\") pod \"sub-test-model-569075-kserve-696f786fbf-f56sh\" (UID: \"12a97fe1-6ffc-4359-a6f7-063d6c065e31\") " pod="ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh" Oct 31 13:38:30.601774 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.601742 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh" Oct 31 13:38:30.773868 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.773829 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:30.892184 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:30.892153 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh"] Oct 31 13:38:30.893040 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:38:30.892974 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12a97fe1_6ffc_4359_a6f7_063d6c065e31.slice/crio-3e56823bbd2532440c68cf78f2be63ac990ad61934bcf43dbbd6ca7da7d55712 WatchSource:0}: Error finding container 3e56823bbd2532440c68cf78f2be63ac990ad61934bcf43dbbd6ca7da7d55712: Status 404 returned error can't find the container with id 3e56823bbd2532440c68cf78f2be63ac990ad61934bcf43dbbd6ca7da7d55712 Oct 31 13:38:31.443538 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:31.443501 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh" event={"ID":"12a97fe1-6ffc-4359-a6f7-063d6c065e31","Type":"ContainerStarted","Data":"c582e173b95d80405b5cd34379f3172b4a89d3f2b5dfc5c07e418f2d1325ab14"} Oct 31 13:38:31.443952 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:31.443547 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh" event={"ID":"12a97fe1-6ffc-4359-a6f7-063d6c065e31","Type":"ContainerStarted","Data":"3e56823bbd2532440c68cf78f2be63ac990ad61934bcf43dbbd6ca7da7d55712"} Oct 31 13:38:31.468089 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:31.468038 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh" podStartSLOduration=1.468025364 podStartE2EDuration="1.468025364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:38:31.463373055 +0000 UTC m=+3957.946495955" watchObservedRunningTime="2026-10-31 13:38:31.468025364 +0000 UTC m=+3957.951148265" Oct 31 13:38:31.604039 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:31.603470 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh" Oct 31 13:38:31.638411 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:31.638320 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:32.596339 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:32.596307 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:33.004874 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:33.004835 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:33.411268 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:33.411185 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:34.190204 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:34.190173 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:35.026265 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:35.026228 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:35.816380 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:35.816339 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:36.591397 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:36.591361 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:37.726568 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:37.726531 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:38.435448 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:38.435411 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:39.357627 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:39.357590 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:40.064139 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:40.064083 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:40.358818 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:40.358708 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:40.602762 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:40.602729 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh" Oct 31 13:38:40.615130 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:40.615060 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh" Oct 31 13:38:40.679492 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:40.679457 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:41.171458 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:41.171423 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:41.551603 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:41.551573 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh" Oct 31 13:38:42.001732 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:42.001695 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:42.858861 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:42.858824 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:43.513970 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:43.513934 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:44.226533 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:44.226497 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:44.717359 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:44.717324 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:45.035547 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:45.035463 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:45.799595 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:45.799512 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:46.500273 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:46.500238 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:47.247686 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:47.247625 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:48.557304 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:48.557242 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:49.481760 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:49.481719 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:50.038445 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:38:50.038405 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:38:50.615077 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:50.614964 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:51.810649 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:51.810605 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:52.166344 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:52.166250 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:52.454550 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:52.454516 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:52.757189 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:52.757108 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q"] Oct 31 13:38:52.757355 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:52.757318 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" podUID="c7a4d6d4-06ba-4385-94cc-66f96e07232a" containerName="istio-proxy" containerID="cri-o://480da5b0e6d30ecb1fad6b5158c5143fa7950fd60a6dcdd6e8800d19391918c5" gracePeriod=30 Oct 31 13:38:52.929006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:52.913726 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:53.145817 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:53.145734 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:53.421964 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:53.421877 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:54.248844 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:54.248806 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:55.124781 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:55.124733 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:55.398693 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:55.398619 6841 patch_prober.go:29] interesting pod/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:38:55.399104 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:55.398694 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" podUID="c7a4d6d4-06ba-4385-94cc-66f96e07232a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:38:55.786957 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:55.786917 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:56.234009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:56.233950 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:56.495062 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:56.493642 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:57.100434 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:57.100346 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:57.878967 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:57.878933 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:58.038701 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.038635 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:38:58.089699 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.089668 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7a4d6d4-06ba-4385-94cc-66f96e07232a-kube-api-access-b2tj4\" (UniqueName: \"kubernetes.io/projected/c7a4d6d4-06ba-4385-94cc-66f96e07232a-kube-api-access-b2tj4\") pod \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\" (UID: \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\") " Oct 31 13:38:58.089848 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.089709 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a4d6d4-06ba-4385-94cc-66f96e07232a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a4d6d4-06ba-4385-94cc-66f96e07232a-credential-socket\") pod \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\" (UID: \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\") " Oct 31 13:38:58.089848 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.089788 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7a4d6d4-06ba-4385-94cc-66f96e07232a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a4d6d4-06ba-4385-94cc-66f96e07232a-istio-podinfo\") pod \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\" (UID: \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\") " Oct 31 13:38:58.090001 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.089858 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a4d6d4-06ba-4385-94cc-66f96e07232a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a4d6d4-06ba-4385-94cc-66f96e07232a-workload-socket\") pod \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\" (UID: \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\") " Oct 31 13:38:58.090001 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.089889 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a4d6d4-06ba-4385-94cc-66f96e07232a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a4d6d4-06ba-4385-94cc-66f96e07232a-istio-envoy\") pod \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\" (UID: \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\") " Oct 31 13:38:58.090169 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.090119 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a4d6d4-06ba-4385-94cc-66f96e07232a-credential-socket" pod "c7a4d6d4-06ba-4385-94cc-66f96e07232a" (UID: "c7a4d6d4-06ba-4385-94cc-66f96e07232a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:38:58.090252 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.090218 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a4d6d4-06ba-4385-94cc-66f96e07232a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a4d6d4-06ba-4385-94cc-66f96e07232a-workload-certs\") pod \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\" (UID: \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\") " Oct 31 13:38:58.090385 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.090304 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7a4d6d4-06ba-4385-94cc-66f96e07232a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a4d6d4-06ba-4385-94cc-66f96e07232a-istiod-ca-cert\") pod \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\" (UID: \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\") " Oct 31 13:38:58.090445 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.090429 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7a4d6d4-06ba-4385-94cc-66f96e07232a-istio-token\" (UniqueName: \"kubernetes.io/projected/c7a4d6d4-06ba-4385-94cc-66f96e07232a-istio-token\") pod \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\" (UID: \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\") " Oct 31 13:38:58.090833 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.090527 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a4d6d4-06ba-4385-94cc-66f96e07232a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a4d6d4-06ba-4385-94cc-66f96e07232a-istio-data\") pod \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\" (UID: \"c7a4d6d4-06ba-4385-94cc-66f96e07232a\") " Oct 31 13:38:58.090833 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.090725 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7a4d6d4-06ba-4385-94cc-66f96e07232a-istiod-ca-cert" pod "c7a4d6d4-06ba-4385-94cc-66f96e07232a" (UID: "c7a4d6d4-06ba-4385-94cc-66f96e07232a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:38:58.090833 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.090797 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a4d6d4-06ba-4385-94cc-66f96e07232a-workload-certs" pod "c7a4d6d4-06ba-4385-94cc-66f96e07232a" (UID: "c7a4d6d4-06ba-4385-94cc-66f96e07232a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:38:58.091177 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.091061 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a4d6d4-06ba-4385-94cc-66f96e07232a-istio-data" pod "c7a4d6d4-06ba-4385-94cc-66f96e07232a" (UID: "c7a4d6d4-06ba-4385-94cc-66f96e07232a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:38:58.091177 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.091140 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a4d6d4-06ba-4385-94cc-66f96e07232a-workload-socket" pod "c7a4d6d4-06ba-4385-94cc-66f96e07232a" (UID: "c7a4d6d4-06ba-4385-94cc-66f96e07232a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:38:58.091451 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.091430 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a4d6d4-06ba-4385-94cc-66f96e07232a-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:58.091530 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.091457 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a4d6d4-06ba-4385-94cc-66f96e07232a-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:58.091530 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.091471 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a4d6d4-06ba-4385-94cc-66f96e07232a-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:58.091530 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.091487 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a4d6d4-06ba-4385-94cc-66f96e07232a-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:58.091530 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.091500 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a4d6d4-06ba-4385-94cc-66f96e07232a-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:58.092902 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.092493 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7a4d6d4-06ba-4385-94cc-66f96e07232a-istio-podinfo" pod "c7a4d6d4-06ba-4385-94cc-66f96e07232a" (UID: "c7a4d6d4-06ba-4385-94cc-66f96e07232a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:38:58.092902 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.092566 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a4d6d4-06ba-4385-94cc-66f96e07232a-istio-envoy" pod "c7a4d6d4-06ba-4385-94cc-66f96e07232a" (UID: "c7a4d6d4-06ba-4385-94cc-66f96e07232a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:38:58.092902 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.092674 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7a4d6d4-06ba-4385-94cc-66f96e07232a-kube-api-access-b2tj4" pod "c7a4d6d4-06ba-4385-94cc-66f96e07232a" (UID: "c7a4d6d4-06ba-4385-94cc-66f96e07232a"). InnerVolumeSpecName "kube-api-access-b2tj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:38:58.093327 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.093297 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7a4d6d4-06ba-4385-94cc-66f96e07232a-istio-token" pod "c7a4d6d4-06ba-4385-94cc-66f96e07232a" (UID: "c7a4d6d4-06ba-4385-94cc-66f96e07232a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:38:58.193470 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.193439 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a4d6d4-06ba-4385-94cc-66f96e07232a-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:58.193470 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.193466 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a4d6d4-06ba-4385-94cc-66f96e07232a-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:58.193470 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.193476 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7a4d6d4-06ba-4385-94cc-66f96e07232a-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:58.193707 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.193485 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2tj4\" (UniqueName: \"kubernetes.io/projected/c7a4d6d4-06ba-4385-94cc-66f96e07232a-kube-api-access-b2tj4\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:58.729421 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.729383 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:58.785201 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.785166 6841 generic.go:356] "Generic (PLEG): container finished" podID="c7a4d6d4-06ba-4385-94cc-66f96e07232a" containerID="480da5b0e6d30ecb1fad6b5158c5143fa7950fd60a6dcdd6e8800d19391918c5" exitCode=0 Oct 31 13:38:58.785348 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.785220 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" Oct 31 13:38:58.785348 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.785246 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" event={"ID":"c7a4d6d4-06ba-4385-94cc-66f96e07232a","Type":"ContainerDied","Data":"480da5b0e6d30ecb1fad6b5158c5143fa7950fd60a6dcdd6e8800d19391918c5"} Oct 31 13:38:58.785348 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.785308 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q" event={"ID":"c7a4d6d4-06ba-4385-94cc-66f96e07232a","Type":"ContainerDied","Data":"7d018b28b7ca5189fc61b3640f309dd128568d465fed1474727787925c2a6e4c"} Oct 31 13:38:58.785348 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.785340 6841 scope.go:122] "RemoveContainer" containerID="480da5b0e6d30ecb1fad6b5158c5143fa7950fd60a6dcdd6e8800d19391918c5" Oct 31 13:38:58.801508 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.800881 6841 scope.go:122] "RemoveContainer" containerID="116ceac509139ce93124e720a1f41afb0822806e9fb44e48bb37f30562e8232a" Oct 31 13:38:58.812444 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.812414 6841 scope.go:122] "RemoveContainer" containerID="480da5b0e6d30ecb1fad6b5158c5143fa7950fd60a6dcdd6e8800d19391918c5" Oct 31 13:38:58.812741 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:38:58.812713 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"480da5b0e6d30ecb1fad6b5158c5143fa7950fd60a6dcdd6e8800d19391918c5\": container with ID starting with 480da5b0e6d30ecb1fad6b5158c5143fa7950fd60a6dcdd6e8800d19391918c5 not found: ID does not exist" containerID="480da5b0e6d30ecb1fad6b5158c5143fa7950fd60a6dcdd6e8800d19391918c5" Oct 31 13:38:58.812845 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.812752 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"480da5b0e6d30ecb1fad6b5158c5143fa7950fd60a6dcdd6e8800d19391918c5"} err="failed to get container status \"480da5b0e6d30ecb1fad6b5158c5143fa7950fd60a6dcdd6e8800d19391918c5\": rpc error: code = NotFound desc = could not find container \"480da5b0e6d30ecb1fad6b5158c5143fa7950fd60a6dcdd6e8800d19391918c5\": container with ID starting with 480da5b0e6d30ecb1fad6b5158c5143fa7950fd60a6dcdd6e8800d19391918c5 not found: ID does not exist" Oct 31 13:38:58.812845 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.812778 6841 scope.go:122] "RemoveContainer" containerID="116ceac509139ce93124e720a1f41afb0822806e9fb44e48bb37f30562e8232a" Oct 31 13:38:58.813062 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:38:58.813039 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"116ceac509139ce93124e720a1f41afb0822806e9fb44e48bb37f30562e8232a\": container with ID starting with 116ceac509139ce93124e720a1f41afb0822806e9fb44e48bb37f30562e8232a not found: ID does not exist" containerID="116ceac509139ce93124e720a1f41afb0822806e9fb44e48bb37f30562e8232a" Oct 31 13:38:58.813121 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.813071 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"116ceac509139ce93124e720a1f41afb0822806e9fb44e48bb37f30562e8232a"} err="failed to get container status \"116ceac509139ce93124e720a1f41afb0822806e9fb44e48bb37f30562e8232a\": rpc error: code = NotFound desc = could not find container \"116ceac509139ce93124e720a1f41afb0822806e9fb44e48bb37f30562e8232a\": container with ID starting with 116ceac509139ce93124e720a1f41afb0822806e9fb44e48bb37f30562e8232a not found: ID does not exist" Oct 31 13:38:58.816088 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.816067 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q"] Oct 31 13:38:58.821509 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:58.821488 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-7n27q"] Oct 31 13:38:59.539053 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:59.539012 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:38:59.651380 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:38:59.651197 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7a4d6d4-06ba-4385-94cc-66f96e07232a" path="/var/lib/kubelet/pods/c7a4d6d4-06ba-4385-94cc-66f96e07232a/volumes" Oct 31 13:39:00.470862 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:00.470824 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:00.774198 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:00.774116 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:01.034956 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:01.034875 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:01.474006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:01.473591 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:01.848521 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:01.848417 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:02.155317 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:02.155231 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:02.402803 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:02.402767 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:02.951908 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:02.951875 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:04.013427 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:04.013392 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:04.827358 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:04.827323 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:05.654143 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:05.654113 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:06.611477 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:06.611433 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:07.563667 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:07.563630 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:08.503976 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:08.503941 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:09.248261 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:09.248228 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:09.515903 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:09.515823 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:09.741320 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:09.741286 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:10.219919 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:10.219872 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:11.135300 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:11.135257 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:12.039825 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:12.039764 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:12.826940 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:12.826905 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:13.694966 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:13.694929 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:14.458043 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:14.458009 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:15.385696 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:15.385648 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:16.024808 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:16.024753 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:16.508683 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:16.508652 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:16.774148 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:16.774061 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:17.126971 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:17.126875 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:17.738472 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:17.738435 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:18.554099 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:18.554053 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:19.096428 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:19.096393 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:19.295257 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:19.295224 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:20.036463 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:39:20.036430 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:39:20.185662 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:20.185619 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:21.131089 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:21.131037 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:21.957889 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:21.957850 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:22.864263 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:22.864222 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:23.920946 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:23.920900 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:24.698726 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:24.698697 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:25.388474 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:25.388433 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:25.685788 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:25.685701 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:26.011316 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:26.011278 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42"] Oct 31 13:39:26.011816 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:26.011776 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42" podUID="8437acf6-79cd-4e8e-b0d5-9e81f3602748" containerName="maas-api" containerID="cri-o://4e45b5dcb76377df37506916d7d8295e58df389826c2030d8f3cf9b7cfe9e1d3" gracePeriod=30 Oct 31 13:39:26.162576 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:26.162507 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:26.449351 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:26.449320 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42" Oct 31 13:39:26.601880 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:26.600022 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8437acf6-79cd-4e8e-b0d5-9e81f3602748-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8437acf6-79cd-4e8e-b0d5-9e81f3602748-maas-api-tls\") pod \"8437acf6-79cd-4e8e-b0d5-9e81f3602748\" (UID: \"8437acf6-79cd-4e8e-b0d5-9e81f3602748\") " Oct 31 13:39:26.601880 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:26.600180 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8437acf6-79cd-4e8e-b0d5-9e81f3602748-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8437acf6-79cd-4e8e-b0d5-9e81f3602748-metrics-certs\") pod \"8437acf6-79cd-4e8e-b0d5-9e81f3602748\" (UID: \"8437acf6-79cd-4e8e-b0d5-9e81f3602748\") " Oct 31 13:39:26.601880 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:26.600365 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8437acf6-79cd-4e8e-b0d5-9e81f3602748-kube-api-access-qp98s\" (UniqueName: \"kubernetes.io/projected/8437acf6-79cd-4e8e-b0d5-9e81f3602748-kube-api-access-qp98s\") pod \"8437acf6-79cd-4e8e-b0d5-9e81f3602748\" (UID: \"8437acf6-79cd-4e8e-b0d5-9e81f3602748\") " Oct 31 13:39:26.601880 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:26.600470 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8437acf6-79cd-4e8e-b0d5-9e81f3602748-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8437acf6-79cd-4e8e-b0d5-9e81f3602748-openshift-service-ca\") pod \"8437acf6-79cd-4e8e-b0d5-9e81f3602748\" (UID: \"8437acf6-79cd-4e8e-b0d5-9e81f3602748\") " Oct 31 13:39:26.603945 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:26.603898 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8437acf6-79cd-4e8e-b0d5-9e81f3602748-openshift-service-ca" pod "8437acf6-79cd-4e8e-b0d5-9e81f3602748" (UID: "8437acf6-79cd-4e8e-b0d5-9e81f3602748"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:39:26.608736 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:26.608695 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8437acf6-79cd-4e8e-b0d5-9e81f3602748-kube-api-access-qp98s" pod "8437acf6-79cd-4e8e-b0d5-9e81f3602748" (UID: "8437acf6-79cd-4e8e-b0d5-9e81f3602748"). InnerVolumeSpecName "kube-api-access-qp98s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:39:26.610115 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:26.610083 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8437acf6-79cd-4e8e-b0d5-9e81f3602748-metrics-certs" pod "8437acf6-79cd-4e8e-b0d5-9e81f3602748" (UID: "8437acf6-79cd-4e8e-b0d5-9e81f3602748"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:39:26.610282 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:26.610260 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8437acf6-79cd-4e8e-b0d5-9e81f3602748-maas-api-tls" pod "8437acf6-79cd-4e8e-b0d5-9e81f3602748" (UID: "8437acf6-79cd-4e8e-b0d5-9e81f3602748"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:39:26.646573 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:26.646483 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:26.704100 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:26.704068 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8437acf6-79cd-4e8e-b0d5-9e81f3602748-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:39:26.704100 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:26.704095 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp98s\" (UniqueName: \"kubernetes.io/projected/8437acf6-79cd-4e8e-b0d5-9e81f3602748-kube-api-access-qp98s\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:39:26.704100 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:26.704108 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8437acf6-79cd-4e8e-b0d5-9e81f3602748-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:39:26.704349 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:26.704117 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8437acf6-79cd-4e8e-b0d5-9e81f3602748-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:39:27.057877 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:27.057844 6841 generic.go:356] "Generic (PLEG): container finished" podID="8437acf6-79cd-4e8e-b0d5-9e81f3602748" containerID="4e45b5dcb76377df37506916d7d8295e58df389826c2030d8f3cf9b7cfe9e1d3" exitCode=0 Oct 31 13:39:27.058100 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:27.057899 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42" Oct 31 13:39:27.058100 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:27.057930 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42" event={"ID":"8437acf6-79cd-4e8e-b0d5-9e81f3602748","Type":"ContainerDied","Data":"4e45b5dcb76377df37506916d7d8295e58df389826c2030d8f3cf9b7cfe9e1d3"} Oct 31 13:39:27.058100 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:27.058017 6841 scope.go:122] "RemoveContainer" containerID="4e45b5dcb76377df37506916d7d8295e58df389826c2030d8f3cf9b7cfe9e1d3" Oct 31 13:39:27.058100 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:27.058014 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42" event={"ID":"8437acf6-79cd-4e8e-b0d5-9e81f3602748","Type":"ContainerDied","Data":"b5ad6d33468e2b6aa96f0fc9044f2a2603816a748ab9fe6afad615ae6c12f021"} Oct 31 13:39:27.069163 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:27.069144 6841 scope.go:122] "RemoveContainer" containerID="4e45b5dcb76377df37506916d7d8295e58df389826c2030d8f3cf9b7cfe9e1d3" Oct 31 13:39:27.069444 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:39:27.069423 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e45b5dcb76377df37506916d7d8295e58df389826c2030d8f3cf9b7cfe9e1d3\": container with ID starting with 4e45b5dcb76377df37506916d7d8295e58df389826c2030d8f3cf9b7cfe9e1d3 not found: ID does not exist" containerID="4e45b5dcb76377df37506916d7d8295e58df389826c2030d8f3cf9b7cfe9e1d3" Oct 31 13:39:27.069512 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:27.069457 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e45b5dcb76377df37506916d7d8295e58df389826c2030d8f3cf9b7cfe9e1d3"} err="failed to get container status \"4e45b5dcb76377df37506916d7d8295e58df389826c2030d8f3cf9b7cfe9e1d3\": rpc error: code = NotFound desc = could not find container \"4e45b5dcb76377df37506916d7d8295e58df389826c2030d8f3cf9b7cfe9e1d3\": container with ID starting with 4e45b5dcb76377df37506916d7d8295e58df389826c2030d8f3cf9b7cfe9e1d3 not found: ID does not exist" Oct 31 13:39:27.088556 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:27.088533 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42"] Oct 31 13:39:27.093651 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:27.093624 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ef744-654d6788f-ssp42"] Oct 31 13:39:27.468122 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:27.468086 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:27.643953 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:27.643921 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8437acf6-79cd-4e8e-b0d5-9e81f3602748" path="/var/lib/kubelet/pods/8437acf6-79cd-4e8e-b0d5-9e81f3602748/volumes" Oct 31 13:39:28.750575 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:28.750547 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:29.812551 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:29.812511 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:30.638412 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:30.638372 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:31.403484 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:31.403451 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:32.147134 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:32.147098 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:32.962390 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:32.962353 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:33.225472 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:33.225383 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:33.851365 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:33.851328 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:34.168550 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:34.168457 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:34.451844 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:34.451799 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:35.058799 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:35.058762 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:35.259402 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:35.259367 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:35.924401 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:35.924362 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:36.649285 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:36.649252 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:37.336320 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:37.336283 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:38.210728 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:38.210684 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:39.253970 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:39.253933 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:40.109459 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:40.109428 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:40.743748 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:40.743712 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:41.050800 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:41.050749 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:41.765264 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:41.765230 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:42.006007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:42.003533 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:42.621652 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:42.621615 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:43.152377 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:43.152340 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:43.480257 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:43.480222 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:44.304708 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:44.304668 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:45.121136 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:45.121098 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:45.820836 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:45.820803 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:46.574231 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:46.574195 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:47.326943 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:47.326910 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:48.118431 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:48.118395 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:48.816629 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:48.816590 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:49.647445 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:49.647411 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:50.035860 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:39:50.035825 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:39:50.195608 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:50.195581 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:50.521254 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:50.521197 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:51.217739 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:51.217662 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:51.937839 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:51.937801 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:52.835327 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:52.835289 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:53.829169 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:53.829134 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:54.389459 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:54.389422 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm"] Oct 31 13:39:54.390337 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:54.390291 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm" podUID="d2a5911a-8b22-49ae-b976-dc61217c3dc2" containerName="maas-api" containerID="cri-o://b8797fff3993c20d9facd6a09da766a51df4129a6c97e90d0aee028814d9896d" gracePeriod=30 Oct 31 13:39:54.613158 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:54.603545 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:54.863772 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:54.863748 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm" Oct 31 13:39:55.014388 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:55.014351 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:55.037018 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:55.035322 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2a5911a-8b22-49ae-b976-dc61217c3dc2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2a5911a-8b22-49ae-b976-dc61217c3dc2-metrics-certs\") pod \"d2a5911a-8b22-49ae-b976-dc61217c3dc2\" (UID: \"d2a5911a-8b22-49ae-b976-dc61217c3dc2\") " Oct 31 13:39:55.037018 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:55.035372 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2a5911a-8b22-49ae-b976-dc61217c3dc2-kube-api-access-ljf45\" (UniqueName: \"kubernetes.io/projected/d2a5911a-8b22-49ae-b976-dc61217c3dc2-kube-api-access-ljf45\") pod \"d2a5911a-8b22-49ae-b976-dc61217c3dc2\" (UID: \"d2a5911a-8b22-49ae-b976-dc61217c3dc2\") " Oct 31 13:39:55.041163 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:55.038365 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2a5911a-8b22-49ae-b976-dc61217c3dc2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2a5911a-8b22-49ae-b976-dc61217c3dc2-maas-api-tls\") pod \"d2a5911a-8b22-49ae-b976-dc61217c3dc2\" (UID: \"d2a5911a-8b22-49ae-b976-dc61217c3dc2\") " Oct 31 13:39:55.041163 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:55.038559 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2a5911a-8b22-49ae-b976-dc61217c3dc2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2a5911a-8b22-49ae-b976-dc61217c3dc2-openshift-service-ca\") pod \"d2a5911a-8b22-49ae-b976-dc61217c3dc2\" (UID: \"d2a5911a-8b22-49ae-b976-dc61217c3dc2\") " Oct 31 13:39:55.041163 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:55.040256 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2a5911a-8b22-49ae-b976-dc61217c3dc2-openshift-service-ca" pod "d2a5911a-8b22-49ae-b976-dc61217c3dc2" (UID: "d2a5911a-8b22-49ae-b976-dc61217c3dc2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:39:55.048098 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:55.046949 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2a5911a-8b22-49ae-b976-dc61217c3dc2-maas-api-tls" pod "d2a5911a-8b22-49ae-b976-dc61217c3dc2" (UID: "d2a5911a-8b22-49ae-b976-dc61217c3dc2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:39:55.054192 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:55.054155 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2a5911a-8b22-49ae-b976-dc61217c3dc2-metrics-certs" pod "d2a5911a-8b22-49ae-b976-dc61217c3dc2" (UID: "d2a5911a-8b22-49ae-b976-dc61217c3dc2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:39:55.054579 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:55.054536 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a5911a-8b22-49ae-b976-dc61217c3dc2-kube-api-access-ljf45" pod "d2a5911a-8b22-49ae-b976-dc61217c3dc2" (UID: "d2a5911a-8b22-49ae-b976-dc61217c3dc2"). InnerVolumeSpecName "kube-api-access-ljf45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:39:55.140651 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:55.140566 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2a5911a-8b22-49ae-b976-dc61217c3dc2-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:39:55.140651 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:55.140599 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2a5911a-8b22-49ae-b976-dc61217c3dc2-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:39:55.140651 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:55.140612 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljf45\" (UniqueName: \"kubernetes.io/projected/d2a5911a-8b22-49ae-b976-dc61217c3dc2-kube-api-access-ljf45\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:39:55.140651 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:55.140624 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2a5911a-8b22-49ae-b976-dc61217c3dc2-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:39:55.374342 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:55.374310 6841 generic.go:356] "Generic (PLEG): container finished" podID="d2a5911a-8b22-49ae-b976-dc61217c3dc2" containerID="b8797fff3993c20d9facd6a09da766a51df4129a6c97e90d0aee028814d9896d" exitCode=0 Oct 31 13:39:55.374570 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:55.374368 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm" event={"ID":"d2a5911a-8b22-49ae-b976-dc61217c3dc2","Type":"ContainerDied","Data":"b8797fff3993c20d9facd6a09da766a51df4129a6c97e90d0aee028814d9896d"} Oct 31 13:39:55.374570 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:55.374372 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm" Oct 31 13:39:55.374570 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:55.374407 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm" event={"ID":"d2a5911a-8b22-49ae-b976-dc61217c3dc2","Type":"ContainerDied","Data":"f5501312a3422bb6dd7375f874130d9939fb1dc5a37f4fede040bd3a85c191fb"} Oct 31 13:39:55.374570 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:55.374427 6841 scope.go:122] "RemoveContainer" containerID="b8797fff3993c20d9facd6a09da766a51df4129a6c97e90d0aee028814d9896d" Oct 31 13:39:55.385215 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:55.385196 6841 scope.go:122] "RemoveContainer" containerID="b8797fff3993c20d9facd6a09da766a51df4129a6c97e90d0aee028814d9896d" Oct 31 13:39:55.385502 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:39:55.385483 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8797fff3993c20d9facd6a09da766a51df4129a6c97e90d0aee028814d9896d\": container with ID starting with b8797fff3993c20d9facd6a09da766a51df4129a6c97e90d0aee028814d9896d not found: ID does not exist" containerID="b8797fff3993c20d9facd6a09da766a51df4129a6c97e90d0aee028814d9896d" Oct 31 13:39:55.385587 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:55.385508 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8797fff3993c20d9facd6a09da766a51df4129a6c97e90d0aee028814d9896d"} err="failed to get container status \"b8797fff3993c20d9facd6a09da766a51df4129a6c97e90d0aee028814d9896d\": rpc error: code = NotFound desc = could not find container \"b8797fff3993c20d9facd6a09da766a51df4129a6c97e90d0aee028814d9896d\": container with ID starting with b8797fff3993c20d9facd6a09da766a51df4129a6c97e90d0aee028814d9896d not found: ID does not exist" Oct 31 13:39:55.399585 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:55.399526 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm"] Oct 31 13:39:55.405436 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:55.405403 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-29e3a0-556c7fd94c-mqfdm"] Oct 31 13:39:55.643428 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:55.643397 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2a5911a-8b22-49ae-b976-dc61217c3dc2" path="/var/lib/kubelet/pods/d2a5911a-8b22-49ae-b976-dc61217c3dc2/volumes" Oct 31 13:39:55.921500 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:55.921454 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:56.732661 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:56.732618 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:56.963843 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:56.963804 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:57.171037 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:57.170942 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:57.520975 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:57.520943 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:57.777849 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:57.777717 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:57.806916 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:57.806882 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj"] Oct 31 13:39:57.807199 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:57.807169 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" podUID="a8494c61-00b4-41e6-b8ed-f4c2081f8944" containerName="istio-proxy" containerID="cri-o://a781f06ced084f339b1e8309d5f678b5d425c791a0aecfb2e04476bbcfca8eee" gracePeriod=30 Oct 31 13:39:57.988146 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:57.988114 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:58.752688 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:58.752651 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:58.986385 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:58.986354 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:39:59.721391 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:39:59.721353 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:00.719160 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:00.719123 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:01.491115 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:01.491084 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:02.254535 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:02.254497 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:03.030671 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.030613 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:03.108907 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.108875 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:40:03.179252 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.179186 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8494c61-00b4-41e6-b8ed-f4c2081f8944-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8494c61-00b4-41e6-b8ed-f4c2081f8944-istio-podinfo\") pod \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\" (UID: \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\") " Oct 31 13:40:03.179451 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.179290 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8494c61-00b4-41e6-b8ed-f4c2081f8944-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8494c61-00b4-41e6-b8ed-f4c2081f8944-credential-socket\") pod \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\" (UID: \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\") " Oct 31 13:40:03.179451 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.179406 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8494c61-00b4-41e6-b8ed-f4c2081f8944-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8494c61-00b4-41e6-b8ed-f4c2081f8944-istio-data\") pod \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\" (UID: \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\") " Oct 31 13:40:03.179581 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.179449 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8494c61-00b4-41e6-b8ed-f4c2081f8944-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8494c61-00b4-41e6-b8ed-f4c2081f8944-istiod-ca-cert\") pod \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\" (UID: \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\") " Oct 31 13:40:03.179581 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.179485 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8494c61-00b4-41e6-b8ed-f4c2081f8944-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8494c61-00b4-41e6-b8ed-f4c2081f8944-workload-certs\") pod \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\" (UID: \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\") " Oct 31 13:40:03.179704 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.179621 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8494c61-00b4-41e6-b8ed-f4c2081f8944-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8494c61-00b4-41e6-b8ed-f4c2081f8944-workload-socket\") pod \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\" (UID: \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\") " Oct 31 13:40:03.179769 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.179736 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8494c61-00b4-41e6-b8ed-f4c2081f8944-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8494c61-00b4-41e6-b8ed-f4c2081f8944-istio-envoy\") pod \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\" (UID: \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\") " Oct 31 13:40:03.179862 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.179833 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8494c61-00b4-41e6-b8ed-f4c2081f8944-istio-token\" (UniqueName: \"kubernetes.io/projected/a8494c61-00b4-41e6-b8ed-f4c2081f8944-istio-token\") pod \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\" (UID: \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\") " Oct 31 13:40:03.179934 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.179875 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8494c61-00b4-41e6-b8ed-f4c2081f8944-kube-api-access-zpt7t\" (UniqueName: \"kubernetes.io/projected/a8494c61-00b4-41e6-b8ed-f4c2081f8944-kube-api-access-zpt7t\") pod \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\" (UID: \"a8494c61-00b4-41e6-b8ed-f4c2081f8944\") " Oct 31 13:40:03.180294 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.180224 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8494c61-00b4-41e6-b8ed-f4c2081f8944-workload-socket" pod "a8494c61-00b4-41e6-b8ed-f4c2081f8944" (UID: "a8494c61-00b4-41e6-b8ed-f4c2081f8944"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:40:03.180294 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.180256 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8494c61-00b4-41e6-b8ed-f4c2081f8944-istiod-ca-cert" pod "a8494c61-00b4-41e6-b8ed-f4c2081f8944" (UID: "a8494c61-00b4-41e6-b8ed-f4c2081f8944"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:40:03.180501 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.180347 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8494c61-00b4-41e6-b8ed-f4c2081f8944-workload-certs" pod "a8494c61-00b4-41e6-b8ed-f4c2081f8944" (UID: "a8494c61-00b4-41e6-b8ed-f4c2081f8944"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:40:03.180501 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.180428 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8494c61-00b4-41e6-b8ed-f4c2081f8944-credential-socket" pod "a8494c61-00b4-41e6-b8ed-f4c2081f8944" (UID: "a8494c61-00b4-41e6-b8ed-f4c2081f8944"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:40:03.180672 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.180602 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8494c61-00b4-41e6-b8ed-f4c2081f8944-istio-data" pod "a8494c61-00b4-41e6-b8ed-f4c2081f8944" (UID: "a8494c61-00b4-41e6-b8ed-f4c2081f8944"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:40:03.180765 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.180695 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8494c61-00b4-41e6-b8ed-f4c2081f8944-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:03.180828 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.180772 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8494c61-00b4-41e6-b8ed-f4c2081f8944-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:03.180828 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.180787 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8494c61-00b4-41e6-b8ed-f4c2081f8944-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:03.180828 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.180801 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8494c61-00b4-41e6-b8ed-f4c2081f8944-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:03.184076 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.184049 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8494c61-00b4-41e6-b8ed-f4c2081f8944-istio-envoy" pod "a8494c61-00b4-41e6-b8ed-f4c2081f8944" (UID: "a8494c61-00b4-41e6-b8ed-f4c2081f8944"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:40:03.184217 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.184089 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8494c61-00b4-41e6-b8ed-f4c2081f8944-istio-token" pod "a8494c61-00b4-41e6-b8ed-f4c2081f8944" (UID: "a8494c61-00b4-41e6-b8ed-f4c2081f8944"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:40:03.184217 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.184137 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8494c61-00b4-41e6-b8ed-f4c2081f8944-istio-podinfo" pod "a8494c61-00b4-41e6-b8ed-f4c2081f8944" (UID: "a8494c61-00b4-41e6-b8ed-f4c2081f8944"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:40:03.184343 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.184312 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8494c61-00b4-41e6-b8ed-f4c2081f8944-kube-api-access-zpt7t" pod "a8494c61-00b4-41e6-b8ed-f4c2081f8944" (UID: "a8494c61-00b4-41e6-b8ed-f4c2081f8944"). InnerVolumeSpecName "kube-api-access-zpt7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:40:03.282525 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.282489 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8494c61-00b4-41e6-b8ed-f4c2081f8944-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:03.282525 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.282520 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8494c61-00b4-41e6-b8ed-f4c2081f8944-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:03.282525 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.282533 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8494c61-00b4-41e6-b8ed-f4c2081f8944-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:03.283041 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.282547 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpt7t\" (UniqueName: \"kubernetes.io/projected/a8494c61-00b4-41e6-b8ed-f4c2081f8944-kube-api-access-zpt7t\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:03.283041 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.282559 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8494c61-00b4-41e6-b8ed-f4c2081f8944-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:03.457487 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.457457 6841 generic.go:356] "Generic (PLEG): container finished" podID="a8494c61-00b4-41e6-b8ed-f4c2081f8944" containerID="a781f06ced084f339b1e8309d5f678b5d425c791a0aecfb2e04476bbcfca8eee" exitCode=0 Oct 31 13:40:03.457673 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.457504 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" Oct 31 13:40:03.457673 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.457534 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" event={"ID":"a8494c61-00b4-41e6-b8ed-f4c2081f8944","Type":"ContainerDied","Data":"a781f06ced084f339b1e8309d5f678b5d425c791a0aecfb2e04476bbcfca8eee"} Oct 31 13:40:03.457673 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.457588 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj" event={"ID":"a8494c61-00b4-41e6-b8ed-f4c2081f8944","Type":"ContainerDied","Data":"594736f1574700a57eed04c45159054a679e3d3f8e43f502ab37c31c56853708"} Oct 31 13:40:03.457673 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.457611 6841 scope.go:122] "RemoveContainer" containerID="a781f06ced084f339b1e8309d5f678b5d425c791a0aecfb2e04476bbcfca8eee" Oct 31 13:40:03.469711 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.469686 6841 scope.go:122] "RemoveContainer" containerID="3019597717a794c14594995799ffde8fde8055320e4d2d73d5c079cb751c1deb" Oct 31 13:40:03.480481 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.480463 6841 scope.go:122] "RemoveContainer" containerID="a781f06ced084f339b1e8309d5f678b5d425c791a0aecfb2e04476bbcfca8eee" Oct 31 13:40:03.480763 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:40:03.480743 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a781f06ced084f339b1e8309d5f678b5d425c791a0aecfb2e04476bbcfca8eee\": container with ID starting with a781f06ced084f339b1e8309d5f678b5d425c791a0aecfb2e04476bbcfca8eee not found: ID does not exist" containerID="a781f06ced084f339b1e8309d5f678b5d425c791a0aecfb2e04476bbcfca8eee" Oct 31 13:40:03.480843 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.480770 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a781f06ced084f339b1e8309d5f678b5d425c791a0aecfb2e04476bbcfca8eee"} err="failed to get container status \"a781f06ced084f339b1e8309d5f678b5d425c791a0aecfb2e04476bbcfca8eee\": rpc error: code = NotFound desc = could not find container \"a781f06ced084f339b1e8309d5f678b5d425c791a0aecfb2e04476bbcfca8eee\": container with ID starting with a781f06ced084f339b1e8309d5f678b5d425c791a0aecfb2e04476bbcfca8eee not found: ID does not exist" Oct 31 13:40:03.480843 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.480788 6841 scope.go:122] "RemoveContainer" containerID="3019597717a794c14594995799ffde8fde8055320e4d2d73d5c079cb751c1deb" Oct 31 13:40:03.481113 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:40:03.481081 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3019597717a794c14594995799ffde8fde8055320e4d2d73d5c079cb751c1deb\": container with ID starting with 3019597717a794c14594995799ffde8fde8055320e4d2d73d5c079cb751c1deb not found: ID does not exist" containerID="3019597717a794c14594995799ffde8fde8055320e4d2d73d5c079cb751c1deb" Oct 31 13:40:03.481197 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.481112 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3019597717a794c14594995799ffde8fde8055320e4d2d73d5c079cb751c1deb"} err="failed to get container status \"3019597717a794c14594995799ffde8fde8055320e4d2d73d5c079cb751c1deb\": rpc error: code = NotFound desc = could not find container \"3019597717a794c14594995799ffde8fde8055320e4d2d73d5c079cb751c1deb\": container with ID starting with 3019597717a794c14594995799ffde8fde8055320e4d2d73d5c079cb751c1deb not found: ID does not exist" Oct 31 13:40:03.489512 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.489491 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj"] Oct 31 13:40:03.495350 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.495323 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-mvchj"] Oct 31 13:40:03.643527 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.643494 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8494c61-00b4-41e6-b8ed-f4c2081f8944" path="/var/lib/kubelet/pods/a8494c61-00b4-41e6-b8ed-f4c2081f8944/volumes" Oct 31 13:40:03.917939 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:03.917855 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:04.704482 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:04.704450 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:05.517393 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:05.517331 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:05.647822 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:05.647792 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-7cdec6-5c4d8b7565-4lv4n"] Oct 31 13:40:05.801791 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:05.801696 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:06.026187 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:06.026068 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:06.374688 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:06.374655 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:06.611101 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:06.611066 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:07.664157 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:07.664124 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:08.395726 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:08.395693 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:09.121550 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:09.121514 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:09.823498 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:09.823449 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:10.571435 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:10.571393 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:10.649063 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:10.649017 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-7cdec6-5c4d8b7565-4lv4n" podUID="eee4ba24-3680-41c1-adbf-c7432d2d20fc" containerName="payload-processing" containerID="cri-o://2595865c9ab28f267007edee0ac245cceab442b6c01d6f18b33f6a378406bf67" gracePeriod=25 Oct 31 13:40:11.321398 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:11.321359 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:11.676742 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:11.676645 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c"] Oct 31 13:40:11.677199 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:11.676910 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c" podUID="682058fc-61a5-4c39-a42f-ada4dc83a0cc" containerName="maas-api" containerID="cri-o://cf01ccae404ac6996c33cc779525af4ba63f384def38d06074cf763bcfb3f44c" gracePeriod=30 Oct 31 13:40:11.829000 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:11.828958 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:12.106318 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:12.106290 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c" Oct 31 13:40:12.128607 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:12.128424 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/682058fc-61a5-4c39-a42f-ada4dc83a0cc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/682058fc-61a5-4c39-a42f-ada4dc83a0cc-maas-api-tls\") pod \"682058fc-61a5-4c39-a42f-ada4dc83a0cc\" (UID: \"682058fc-61a5-4c39-a42f-ada4dc83a0cc\") " Oct 31 13:40:12.128607 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:12.128568 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/682058fc-61a5-4c39-a42f-ada4dc83a0cc-kube-api-access-5hsvq\" (UniqueName: \"kubernetes.io/projected/682058fc-61a5-4c39-a42f-ada4dc83a0cc-kube-api-access-5hsvq\") pod \"682058fc-61a5-4c39-a42f-ada4dc83a0cc\" (UID: \"682058fc-61a5-4c39-a42f-ada4dc83a0cc\") " Oct 31 13:40:12.128886 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:12.128729 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/682058fc-61a5-4c39-a42f-ada4dc83a0cc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/682058fc-61a5-4c39-a42f-ada4dc83a0cc-openshift-service-ca\") pod \"682058fc-61a5-4c39-a42f-ada4dc83a0cc\" (UID: \"682058fc-61a5-4c39-a42f-ada4dc83a0cc\") " Oct 31 13:40:12.128886 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:12.128788 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/682058fc-61a5-4c39-a42f-ada4dc83a0cc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/682058fc-61a5-4c39-a42f-ada4dc83a0cc-metrics-certs\") pod \"682058fc-61a5-4c39-a42f-ada4dc83a0cc\" (UID: \"682058fc-61a5-4c39-a42f-ada4dc83a0cc\") " Oct 31 13:40:12.129194 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:12.129167 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/682058fc-61a5-4c39-a42f-ada4dc83a0cc-openshift-service-ca" pod "682058fc-61a5-4c39-a42f-ada4dc83a0cc" (UID: "682058fc-61a5-4c39-a42f-ada4dc83a0cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:40:12.129414 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:12.129391 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/682058fc-61a5-4c39-a42f-ada4dc83a0cc-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:12.130936 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:12.130896 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/682058fc-61a5-4c39-a42f-ada4dc83a0cc-maas-api-tls" pod "682058fc-61a5-4c39-a42f-ada4dc83a0cc" (UID: "682058fc-61a5-4c39-a42f-ada4dc83a0cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:40:12.131399 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:12.131373 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/682058fc-61a5-4c39-a42f-ada4dc83a0cc-kube-api-access-5hsvq" pod "682058fc-61a5-4c39-a42f-ada4dc83a0cc" (UID: "682058fc-61a5-4c39-a42f-ada4dc83a0cc"). InnerVolumeSpecName "kube-api-access-5hsvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:40:12.132567 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:12.132531 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/682058fc-61a5-4c39-a42f-ada4dc83a0cc-metrics-certs" pod "682058fc-61a5-4c39-a42f-ada4dc83a0cc" (UID: "682058fc-61a5-4c39-a42f-ada4dc83a0cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:40:12.230953 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:12.230916 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/682058fc-61a5-4c39-a42f-ada4dc83a0cc-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:12.230953 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:12.230948 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hsvq\" (UniqueName: \"kubernetes.io/projected/682058fc-61a5-4c39-a42f-ada4dc83a0cc-kube-api-access-5hsvq\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:12.230953 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:12.230959 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/682058fc-61a5-4c39-a42f-ada4dc83a0cc-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:12.331053 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:12.330933 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:12.533419 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:12.533383 6841 generic.go:356] "Generic (PLEG): container finished" podID="682058fc-61a5-4c39-a42f-ada4dc83a0cc" containerID="cf01ccae404ac6996c33cc779525af4ba63f384def38d06074cf763bcfb3f44c" exitCode=0 Oct 31 13:40:12.533590 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:12.533440 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c" Oct 31 13:40:12.533590 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:12.533442 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c" event={"ID":"682058fc-61a5-4c39-a42f-ada4dc83a0cc","Type":"ContainerDied","Data":"cf01ccae404ac6996c33cc779525af4ba63f384def38d06074cf763bcfb3f44c"} Oct 31 13:40:12.533590 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:12.533567 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c" event={"ID":"682058fc-61a5-4c39-a42f-ada4dc83a0cc","Type":"ContainerDied","Data":"cf1c1bd7982b858dd9ce4c06cd40cc7681c3c340a0a19fca45a7b85ceb086520"} Oct 31 13:40:12.533590 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:12.533589 6841 scope.go:122] "RemoveContainer" containerID="cf01ccae404ac6996c33cc779525af4ba63f384def38d06074cf763bcfb3f44c" Oct 31 13:40:12.545428 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:12.545409 6841 scope.go:122] "RemoveContainer" containerID="cf01ccae404ac6996c33cc779525af4ba63f384def38d06074cf763bcfb3f44c" Oct 31 13:40:12.545746 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:40:12.545727 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf01ccae404ac6996c33cc779525af4ba63f384def38d06074cf763bcfb3f44c\": container with ID starting with cf01ccae404ac6996c33cc779525af4ba63f384def38d06074cf763bcfb3f44c not found: ID does not exist" containerID="cf01ccae404ac6996c33cc779525af4ba63f384def38d06074cf763bcfb3f44c" Oct 31 13:40:12.545806 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:12.545757 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf01ccae404ac6996c33cc779525af4ba63f384def38d06074cf763bcfb3f44c"} err="failed to get container status \"cf01ccae404ac6996c33cc779525af4ba63f384def38d06074cf763bcfb3f44c\": rpc error: code = NotFound desc = could not find container \"cf01ccae404ac6996c33cc779525af4ba63f384def38d06074cf763bcfb3f44c\": container with ID starting with cf01ccae404ac6996c33cc779525af4ba63f384def38d06074cf763bcfb3f44c not found: ID does not exist" Oct 31 13:40:12.560992 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:12.560949 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c"] Oct 31 13:40:12.565430 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:12.565406 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-7cdec6-7f776647d8-hx25c"] Oct 31 13:40:13.152594 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:13.152553 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:13.385775 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:13.385740 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:13.643893 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:13.643857 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="682058fc-61a5-4c39-a42f-ada4dc83a0cc" path="/var/lib/kubelet/pods/682058fc-61a5-4c39-a42f-ada4dc83a0cc/volumes" Oct 31 13:40:13.720733 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:13.720698 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:14.127007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:14.118654 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:14.337475 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:14.337436 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:14.695212 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:14.695179 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-7cdec6-5c4d8b7565-4lv4n" podUID="eee4ba24-3680-41c1-adbf-c7432d2d20fc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:40:14.893025 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:14.892960 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:15.200935 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:15.200902 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:16.044719 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:16.044631 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:16.918457 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:16.918423 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:17.617596 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:17.617534 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:18.402109 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:18.402069 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:19.318099 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:19.318065 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:20.036796 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:40:20.036763 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:40:20.039832 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:20.039805 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:20.773804 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:20.773765 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:21.564993 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:21.564954 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:21.800460 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:21.800428 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:22.023114 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:22.023069 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:22.699502 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:22.699464 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:22.909654 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:22.909618 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:23.459591 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:23.459558 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:24.212353 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:24.212316 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:24.695331 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:24.695296 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-7cdec6-5c4d8b7565-4lv4n" podUID="eee4ba24-3680-41c1-adbf-c7432d2d20fc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:40:25.123686 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:25.123600 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:25.714615 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:25.714568 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:26.626506 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:26.626469 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:27.432925 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:27.432884 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:28.200823 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:28.200777 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:28.990178 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:28.990124 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:29.229382 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:29.229342 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:29.700877 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:29.700842 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:30.644596 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:30.644561 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:30.913593 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:30.913534 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-7cdec6-5c4d8b7565-4lv4n" Oct 31 13:40:31.050212 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:31.050181 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eee4ba24-3680-41c1-adbf-c7432d2d20fc-kube-api-access-wjlwq\" (UniqueName: \"kubernetes.io/projected/eee4ba24-3680-41c1-adbf-c7432d2d20fc-kube-api-access-wjlwq\") pod \"eee4ba24-3680-41c1-adbf-c7432d2d20fc\" (UID: \"eee4ba24-3680-41c1-adbf-c7432d2d20fc\") " Oct 31 13:40:31.050383 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:31.050305 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eee4ba24-3680-41c1-adbf-c7432d2d20fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eee4ba24-3680-41c1-adbf-c7432d2d20fc-plugins-config-volume\") pod \"eee4ba24-3680-41c1-adbf-c7432d2d20fc\" (UID: \"eee4ba24-3680-41c1-adbf-c7432d2d20fc\") " Oct 31 13:40:31.050687 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:31.050664 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eee4ba24-3680-41c1-adbf-c7432d2d20fc-plugins-config-volume" pod "eee4ba24-3680-41c1-adbf-c7432d2d20fc" (UID: "eee4ba24-3680-41c1-adbf-c7432d2d20fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:40:31.052236 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:31.052217 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eee4ba24-3680-41c1-adbf-c7432d2d20fc-kube-api-access-wjlwq" pod "eee4ba24-3680-41c1-adbf-c7432d2d20fc" (UID: "eee4ba24-3680-41c1-adbf-c7432d2d20fc"). InnerVolumeSpecName "kube-api-access-wjlwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:40:31.152480 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:31.152447 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjlwq\" (UniqueName: \"kubernetes.io/projected/eee4ba24-3680-41c1-adbf-c7432d2d20fc-kube-api-access-wjlwq\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:31.152480 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:31.152477 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eee4ba24-3680-41c1-adbf-c7432d2d20fc-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:31.629023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:31.628966 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:31.729383 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:31.729349 6841 generic.go:356] "Generic (PLEG): container finished" podID="eee4ba24-3680-41c1-adbf-c7432d2d20fc" containerID="2595865c9ab28f267007edee0ac245cceab442b6c01d6f18b33f6a378406bf67" exitCode=0 Oct 31 13:40:31.729803 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:31.729407 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-7cdec6-5c4d8b7565-4lv4n" Oct 31 13:40:31.729803 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:31.729433 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-7cdec6-5c4d8b7565-4lv4n" event={"ID":"eee4ba24-3680-41c1-adbf-c7432d2d20fc","Type":"ContainerDied","Data":"2595865c9ab28f267007edee0ac245cceab442b6c01d6f18b33f6a378406bf67"} Oct 31 13:40:31.729803 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:31.729484 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-7cdec6-5c4d8b7565-4lv4n" event={"ID":"eee4ba24-3680-41c1-adbf-c7432d2d20fc","Type":"ContainerDied","Data":"6d77a123fb2b16b93b2640844954f10a0f0ee41c5c3a6ed69455e9cf6cf4e932"} Oct 31 13:40:31.729803 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:31.729514 6841 scope.go:122] "RemoveContainer" containerID="2595865c9ab28f267007edee0ac245cceab442b6c01d6f18b33f6a378406bf67" Oct 31 13:40:31.740012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:31.739965 6841 scope.go:122] "RemoveContainer" containerID="2595865c9ab28f267007edee0ac245cceab442b6c01d6f18b33f6a378406bf67" Oct 31 13:40:31.740276 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:40:31.740248 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2595865c9ab28f267007edee0ac245cceab442b6c01d6f18b33f6a378406bf67\": container with ID starting with 2595865c9ab28f267007edee0ac245cceab442b6c01d6f18b33f6a378406bf67 not found: ID does not exist" containerID="2595865c9ab28f267007edee0ac245cceab442b6c01d6f18b33f6a378406bf67" Oct 31 13:40:31.740344 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:31.740278 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2595865c9ab28f267007edee0ac245cceab442b6c01d6f18b33f6a378406bf67"} err="failed to get container status \"2595865c9ab28f267007edee0ac245cceab442b6c01d6f18b33f6a378406bf67\": rpc error: code = NotFound desc = could not find container \"2595865c9ab28f267007edee0ac245cceab442b6c01d6f18b33f6a378406bf67\": container with ID starting with 2595865c9ab28f267007edee0ac245cceab442b6c01d6f18b33f6a378406bf67 not found: ID does not exist" Oct 31 13:40:31.754509 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:31.754485 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-7cdec6-5c4d8b7565-4lv4n"] Oct 31 13:40:31.760803 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:31.760780 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-7cdec6-5c4d8b7565-4lv4n"] Oct 31 13:40:32.655473 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:32.655433 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:33.641544 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:33.641509 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:33.644458 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:33.644434 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eee4ba24-3680-41c1-adbf-c7432d2d20fc" path="/var/lib/kubelet/pods/eee4ba24-3680-41c1-adbf-c7432d2d20fc/volumes" Oct 31 13:40:34.501115 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:34.501080 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:35.298262 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:35.298204 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:36.106343 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:36.106308 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:36.928741 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:36.928704 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:37.907525 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:37.907492 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:38.968716 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:38.968658 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:39.735859 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:39.735810 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:40.514608 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:40.514572 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:40.797435 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:40.797311 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:40.948017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:40.947963 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9"] Oct 31 13:40:40.948284 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:40.948253 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9" podUID="3a4747b7-ff5a-4c37-b3f9-5db97fce5d12" containerName="maas-api" containerID="cri-o://92fe698505839cf89b84aea619f132e229e1109998ea790c9b94669e298f0445" gracePeriod=30 Oct 31 13:40:41.089486 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:41.089461 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:41.362515 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:41.362490 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9" Oct 31 13:40:41.437061 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:41.437005 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:41.532974 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:41.531171 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a4747b7-ff5a-4c37-b3f9-5db97fce5d12-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a4747b7-ff5a-4c37-b3f9-5db97fce5d12-metrics-certs\") pod \"3a4747b7-ff5a-4c37-b3f9-5db97fce5d12\" (UID: \"3a4747b7-ff5a-4c37-b3f9-5db97fce5d12\") " Oct 31 13:40:41.532974 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:41.531316 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a4747b7-ff5a-4c37-b3f9-5db97fce5d12-kube-api-access-6679b\" (UniqueName: \"kubernetes.io/projected/3a4747b7-ff5a-4c37-b3f9-5db97fce5d12-kube-api-access-6679b\") pod \"3a4747b7-ff5a-4c37-b3f9-5db97fce5d12\" (UID: \"3a4747b7-ff5a-4c37-b3f9-5db97fce5d12\") " Oct 31 13:40:41.532974 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:41.531428 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a4747b7-ff5a-4c37-b3f9-5db97fce5d12-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a4747b7-ff5a-4c37-b3f9-5db97fce5d12-openshift-service-ca\") pod \"3a4747b7-ff5a-4c37-b3f9-5db97fce5d12\" (UID: \"3a4747b7-ff5a-4c37-b3f9-5db97fce5d12\") " Oct 31 13:40:41.532974 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:41.531467 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a4747b7-ff5a-4c37-b3f9-5db97fce5d12-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a4747b7-ff5a-4c37-b3f9-5db97fce5d12-maas-api-tls\") pod \"3a4747b7-ff5a-4c37-b3f9-5db97fce5d12\" (UID: \"3a4747b7-ff5a-4c37-b3f9-5db97fce5d12\") " Oct 31 13:40:41.538405 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:41.538374 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a4747b7-ff5a-4c37-b3f9-5db97fce5d12-metrics-certs" pod "3a4747b7-ff5a-4c37-b3f9-5db97fce5d12" (UID: "3a4747b7-ff5a-4c37-b3f9-5db97fce5d12"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:40:41.538877 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:41.538834 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a4747b7-ff5a-4c37-b3f9-5db97fce5d12-maas-api-tls" pod "3a4747b7-ff5a-4c37-b3f9-5db97fce5d12" (UID: "3a4747b7-ff5a-4c37-b3f9-5db97fce5d12"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:40:41.539062 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:41.538868 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a4747b7-ff5a-4c37-b3f9-5db97fce5d12-openshift-service-ca" pod "3a4747b7-ff5a-4c37-b3f9-5db97fce5d12" (UID: "3a4747b7-ff5a-4c37-b3f9-5db97fce5d12"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:40:41.542865 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:41.542836 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a4747b7-ff5a-4c37-b3f9-5db97fce5d12-kube-api-access-6679b" pod "3a4747b7-ff5a-4c37-b3f9-5db97fce5d12" (UID: "3a4747b7-ff5a-4c37-b3f9-5db97fce5d12"). InnerVolumeSpecName "kube-api-access-6679b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:40:41.634317 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:41.634250 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a4747b7-ff5a-4c37-b3f9-5db97fce5d12-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:41.634317 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:41.634281 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6679b\" (UniqueName: \"kubernetes.io/projected/3a4747b7-ff5a-4c37-b3f9-5db97fce5d12-kube-api-access-6679b\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:41.634317 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:41.634292 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a4747b7-ff5a-4c37-b3f9-5db97fce5d12-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:41.634317 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:41.634301 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a4747b7-ff5a-4c37-b3f9-5db97fce5d12-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:41.803951 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:41.803913 6841 generic.go:356] "Generic (PLEG): container finished" podID="3a4747b7-ff5a-4c37-b3f9-5db97fce5d12" containerID="92fe698505839cf89b84aea619f132e229e1109998ea790c9b94669e298f0445" exitCode=0 Oct 31 13:40:41.804131 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:41.804011 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9" Oct 31 13:40:41.804131 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:41.804008 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9" event={"ID":"3a4747b7-ff5a-4c37-b3f9-5db97fce5d12","Type":"ContainerDied","Data":"92fe698505839cf89b84aea619f132e229e1109998ea790c9b94669e298f0445"} Oct 31 13:40:41.804245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:41.804131 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9" event={"ID":"3a4747b7-ff5a-4c37-b3f9-5db97fce5d12","Type":"ContainerDied","Data":"13e45277c82cc2eb4d8c14f60cb2cbb4b085a864c7eb264cd69a88eb6bbb6281"} Oct 31 13:40:41.804245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:41.804158 6841 scope.go:122] "RemoveContainer" containerID="92fe698505839cf89b84aea619f132e229e1109998ea790c9b94669e298f0445" Oct 31 13:40:41.815346 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:41.815321 6841 scope.go:122] "RemoveContainer" containerID="92fe698505839cf89b84aea619f132e229e1109998ea790c9b94669e298f0445" Oct 31 13:40:41.815605 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:40:41.815587 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92fe698505839cf89b84aea619f132e229e1109998ea790c9b94669e298f0445\": container with ID starting with 92fe698505839cf89b84aea619f132e229e1109998ea790c9b94669e298f0445 not found: ID does not exist" containerID="92fe698505839cf89b84aea619f132e229e1109998ea790c9b94669e298f0445" Oct 31 13:40:41.815659 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:41.815614 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92fe698505839cf89b84aea619f132e229e1109998ea790c9b94669e298f0445"} err="failed to get container status \"92fe698505839cf89b84aea619f132e229e1109998ea790c9b94669e298f0445\": rpc error: code = NotFound desc = could not find container \"92fe698505839cf89b84aea619f132e229e1109998ea790c9b94669e298f0445\": container with ID starting with 92fe698505839cf89b84aea619f132e229e1109998ea790c9b94669e298f0445 not found: ID does not exist" Oct 31 13:40:41.824762 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:41.824739 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9"] Oct 31 13:40:41.830039 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:41.830015 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-377f60-646d468796-4d4l9"] Oct 31 13:40:42.250034 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:42.249963 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:43.037850 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:43.037814 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:43.643449 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:43.643417 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a4747b7-ff5a-4c37-b3f9-5db97fce5d12" path="/var/lib/kubelet/pods/3a4747b7-ff5a-4c37-b3f9-5db97fce5d12/volumes" Oct 31 13:40:43.844680 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:43.844648 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:44.589563 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:44.589529 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:45.308173 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:45.308137 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:46.021672 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:46.021634 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:46.962572 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:46.962538 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:47.376881 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:47.376790 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:47.700293 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:47.700262 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:48.096461 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:48.096372 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:48.315902 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:48.315840 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:48.644777 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:48.644743 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:49.138858 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:49.138826 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:49.353653 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:49.353616 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:50.039270 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:40:50.039232 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:40:50.047540 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:50.047512 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:51.002702 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:51.002674 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:51.708320 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:51.708286 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:52.403844 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:52.403810 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:53.220010 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:53.219960 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:53.374284 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:53.374253 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc"] Oct 31 13:40:53.630550 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:53.630467 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:54.234562 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:54.234526 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:54.458832 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:54.458790 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:55.021302 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:55.021266 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:55.839682 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:55.839649 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:56.387192 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:56.387147 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:56.593856 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:56.593818 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:57.076219 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:57.076187 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:57.310930 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:57.310900 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:58.091865 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:58.091835 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:58.796454 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:58.796408 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:40:59.654971 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:40:59.654936 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:00.095908 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:00.095867 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-107955-86f6b68d45-ztxhj"] Oct 31 13:41:00.282014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:00.281936 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:00.507506 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:00.507473 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:00.742351 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:00.742313 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:01.605542 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:01.605499 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:02.068685 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:02.068648 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:02.378392 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:02.378282 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:03.344055 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:03.344017 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:04.151710 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:04.151675 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:04.965062 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:04.965025 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:05.097062 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:05.097018 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-107955-86f6b68d45-ztxhj" podUID="481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe" containerName="payload-processing" containerID="cri-o://940d9c7a7e36edbdf013cfaf00a94be25cb3828c044f69f676a2736bb67af8d4" gracePeriod=25 Oct 31 13:41:05.734354 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:05.734315 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:06.517638 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:06.517608 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:06.837441 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:06.837359 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-107955-86f6b68d45-ztxhj" podUID="481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:41:07.358557 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:07.358522 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:08.209797 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:08.209757 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:08.396465 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:08.396431 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" podUID="da83bd57-52af-48d3-bd76-42e36c18112f" containerName="main" containerID="cri-o://3d1752ddc92abbbb6cf16987d1888bc6c054f2064bfa4cafc8bca8492b1c2f50" gracePeriod=45 Oct 31 13:41:08.465925 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:08.465853 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" podUID="da83bd57-52af-48d3-bd76-42e36c18112f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": dial tcp 10.134.0.73:8000: connect: connection refused" Oct 31 13:41:08.679150 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:08.679124 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" Oct 31 13:41:08.825943 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:08.825873 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da83bd57-52af-48d3-bd76-42e36c18112f-tls-certs\" (UniqueName: \"kubernetes.io/secret/da83bd57-52af-48d3-bd76-42e36c18112f-tls-certs\") pod \"da83bd57-52af-48d3-bd76-42e36c18112f\" (UID: \"da83bd57-52af-48d3-bd76-42e36c18112f\") " Oct 31 13:41:08.825943 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:08.825931 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da83bd57-52af-48d3-bd76-42e36c18112f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/da83bd57-52af-48d3-bd76-42e36c18112f-dshm\") pod \"da83bd57-52af-48d3-bd76-42e36c18112f\" (UID: \"da83bd57-52af-48d3-bd76-42e36c18112f\") " Oct 31 13:41:08.826269 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:08.825999 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da83bd57-52af-48d3-bd76-42e36c18112f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da83bd57-52af-48d3-bd76-42e36c18112f-model-cache\") pod \"da83bd57-52af-48d3-bd76-42e36c18112f\" (UID: \"da83bd57-52af-48d3-bd76-42e36c18112f\") " Oct 31 13:41:08.826269 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:08.826116 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da83bd57-52af-48d3-bd76-42e36c18112f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da83bd57-52af-48d3-bd76-42e36c18112f-tmp-dir\") pod \"da83bd57-52af-48d3-bd76-42e36c18112f\" (UID: \"da83bd57-52af-48d3-bd76-42e36c18112f\") " Oct 31 13:41:08.826269 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:08.826135 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da83bd57-52af-48d3-bd76-42e36c18112f-kube-api-access-lhhb7\" (UniqueName: \"kubernetes.io/projected/da83bd57-52af-48d3-bd76-42e36c18112f-kube-api-access-lhhb7\") pod \"da83bd57-52af-48d3-bd76-42e36c18112f\" (UID: \"da83bd57-52af-48d3-bd76-42e36c18112f\") " Oct 31 13:41:08.826269 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:08.826224 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da83bd57-52af-48d3-bd76-42e36c18112f-home\" (UniqueName: \"kubernetes.io/empty-dir/da83bd57-52af-48d3-bd76-42e36c18112f-home\") pod \"da83bd57-52af-48d3-bd76-42e36c18112f\" (UID: \"da83bd57-52af-48d3-bd76-42e36c18112f\") " Oct 31 13:41:08.826520 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:08.826336 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da83bd57-52af-48d3-bd76-42e36c18112f-model-cache" pod "da83bd57-52af-48d3-bd76-42e36c18112f" (UID: "da83bd57-52af-48d3-bd76-42e36c18112f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:08.826649 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:08.826623 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da83bd57-52af-48d3-bd76-42e36c18112f-home" pod "da83bd57-52af-48d3-bd76-42e36c18112f" (UID: "da83bd57-52af-48d3-bd76-42e36c18112f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:08.826728 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:08.826708 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da83bd57-52af-48d3-bd76-42e36c18112f-tmp-dir" pod "da83bd57-52af-48d3-bd76-42e36c18112f" (UID: "da83bd57-52af-48d3-bd76-42e36c18112f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:08.827024 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:08.827003 6841 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da83bd57-52af-48d3-bd76-42e36c18112f-tmp-dir\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:08.827125 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:08.827028 6841 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da83bd57-52af-48d3-bd76-42e36c18112f-home\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:08.827125 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:08.827043 6841 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da83bd57-52af-48d3-bd76-42e36c18112f-model-cache\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:08.828768 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:08.828744 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da83bd57-52af-48d3-bd76-42e36c18112f-dshm" pod "da83bd57-52af-48d3-bd76-42e36c18112f" (UID: "da83bd57-52af-48d3-bd76-42e36c18112f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:08.829227 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:08.829206 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da83bd57-52af-48d3-bd76-42e36c18112f-tls-certs" pod "da83bd57-52af-48d3-bd76-42e36c18112f" (UID: "da83bd57-52af-48d3-bd76-42e36c18112f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:41:08.831766 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:08.831745 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da83bd57-52af-48d3-bd76-42e36c18112f-kube-api-access-lhhb7" pod "da83bd57-52af-48d3-bd76-42e36c18112f" (UID: "da83bd57-52af-48d3-bd76-42e36c18112f"). InnerVolumeSpecName "kube-api-access-lhhb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:41:08.928427 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:08.928393 6841 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da83bd57-52af-48d3-bd76-42e36c18112f-tls-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:08.928427 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:08.928424 6841 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da83bd57-52af-48d3-bd76-42e36c18112f-dshm\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:08.928628 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:08.928436 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhhb7\" (UniqueName: \"kubernetes.io/projected/da83bd57-52af-48d3-bd76-42e36c18112f-kube-api-access-lhhb7\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:09.002696 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:09.002664 6841 generic.go:356] "Generic (PLEG): container finished" podID="da83bd57-52af-48d3-bd76-42e36c18112f" containerID="3d1752ddc92abbbb6cf16987d1888bc6c054f2064bfa4cafc8bca8492b1c2f50" exitCode=0 Oct 31 13:41:09.002883 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:09.002710 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" Oct 31 13:41:09.002883 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:09.002748 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" event={"ID":"da83bd57-52af-48d3-bd76-42e36c18112f","Type":"ContainerDied","Data":"3d1752ddc92abbbb6cf16987d1888bc6c054f2064bfa4cafc8bca8492b1c2f50"} Oct 31 13:41:09.002883 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:09.002792 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc" event={"ID":"da83bd57-52af-48d3-bd76-42e36c18112f","Type":"ContainerDied","Data":"f93e3c5751f498c1b8f8db8243e0018ac5bb2154d043448ca3d965ccd5e6f69d"} Oct 31 13:41:09.002883 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:09.002812 6841 scope.go:122] "RemoveContainer" containerID="3d1752ddc92abbbb6cf16987d1888bc6c054f2064bfa4cafc8bca8492b1c2f50" Oct 31 13:41:09.011931 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:09.011245 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:09.014937 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:09.014916 6841 scope.go:122] "RemoveContainer" containerID="3d1752ddc92abbbb6cf16987d1888bc6c054f2064bfa4cafc8bca8492b1c2f50" Oct 31 13:41:09.015205 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:41:09.015186 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d1752ddc92abbbb6cf16987d1888bc6c054f2064bfa4cafc8bca8492b1c2f50\": container with ID starting with 3d1752ddc92abbbb6cf16987d1888bc6c054f2064bfa4cafc8bca8492b1c2f50 not found: ID does not exist" containerID="3d1752ddc92abbbb6cf16987d1888bc6c054f2064bfa4cafc8bca8492b1c2f50" Oct 31 13:41:09.015278 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:09.015218 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d1752ddc92abbbb6cf16987d1888bc6c054f2064bfa4cafc8bca8492b1c2f50"} err="failed to get container status \"3d1752ddc92abbbb6cf16987d1888bc6c054f2064bfa4cafc8bca8492b1c2f50\": rpc error: code = NotFound desc = could not find container \"3d1752ddc92abbbb6cf16987d1888bc6c054f2064bfa4cafc8bca8492b1c2f50\": container with ID starting with 3d1752ddc92abbbb6cf16987d1888bc6c054f2064bfa4cafc8bca8492b1c2f50 not found: ID does not exist" Oct 31 13:41:09.029099 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:09.029077 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc"] Oct 31 13:41:09.035977 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:09.035941 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-107955/distinct-2-w0-107955-kserve-64487cdd8f-2xrhc"] Oct 31 13:41:09.643378 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:09.643345 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da83bd57-52af-48d3-bd76-42e36c18112f" path="/var/lib/kubelet/pods/da83bd57-52af-48d3-bd76-42e36c18112f/volumes" Oct 31 13:41:09.792623 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:09.792590 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:10.000509 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:10.000478 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:10.696682 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:10.696644 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:11.620099 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:11.620058 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:12.349590 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:12.349545 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:13.297961 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:13.297929 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:14.110649 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:14.110613 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:15.034673 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:15.034641 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:15.890515 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:15.890482 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:16.513450 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:16.513404 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:16.745420 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:16.745384 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:16.837574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:16.837496 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-107955-86f6b68d45-ztxhj" podUID="481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:41:17.180109 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:17.180015 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-107955-fc749446-559d9"] Oct 31 13:41:17.180597 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:17.180261 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-107955-fc749446-559d9" podUID="c0573f69-b5ed-475b-b579-4c084744071d" containerName="maas-api" containerID="cri-o://422b62b920308dffe52358f6b3789f68b12b68c364efa998fc42205a60435118" gracePeriod=30 Oct 31 13:41:17.368656 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:17.368490 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:17.599999 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:17.599952 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-107955-fc749446-559d9" Oct 31 13:41:17.638494 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:17.638447 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:17.699992 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:17.699955 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0573f69-b5ed-475b-b579-4c084744071d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0573f69-b5ed-475b-b579-4c084744071d-openshift-service-ca\") pod \"c0573f69-b5ed-475b-b579-4c084744071d\" (UID: \"c0573f69-b5ed-475b-b579-4c084744071d\") " Oct 31 13:41:17.700389 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:17.700359 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0573f69-b5ed-475b-b579-4c084744071d-openshift-service-ca" pod "c0573f69-b5ed-475b-b579-4c084744071d" (UID: "c0573f69-b5ed-475b-b579-4c084744071d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:41:17.801108 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:17.801020 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0573f69-b5ed-475b-b579-4c084744071d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0573f69-b5ed-475b-b579-4c084744071d-maas-api-tls\") pod \"c0573f69-b5ed-475b-b579-4c084744071d\" (UID: \"c0573f69-b5ed-475b-b579-4c084744071d\") " Oct 31 13:41:17.801296 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:17.801274 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0573f69-b5ed-475b-b579-4c084744071d-kube-api-access-p9ln7\" (UniqueName: \"kubernetes.io/projected/c0573f69-b5ed-475b-b579-4c084744071d-kube-api-access-p9ln7\") pod \"c0573f69-b5ed-475b-b579-4c084744071d\" (UID: \"c0573f69-b5ed-475b-b579-4c084744071d\") " Oct 31 13:41:17.801350 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:17.801329 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0573f69-b5ed-475b-b579-4c084744071d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0573f69-b5ed-475b-b579-4c084744071d-metrics-certs\") pod \"c0573f69-b5ed-475b-b579-4c084744071d\" (UID: \"c0573f69-b5ed-475b-b579-4c084744071d\") " Oct 31 13:41:17.801915 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:17.801874 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0573f69-b5ed-475b-b579-4c084744071d-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:17.803362 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:17.803329 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0573f69-b5ed-475b-b579-4c084744071d-maas-api-tls" pod "c0573f69-b5ed-475b-b579-4c084744071d" (UID: "c0573f69-b5ed-475b-b579-4c084744071d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:41:17.803790 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:17.803771 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0573f69-b5ed-475b-b579-4c084744071d-metrics-certs" pod "c0573f69-b5ed-475b-b579-4c084744071d" (UID: "c0573f69-b5ed-475b-b579-4c084744071d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:41:17.803848 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:17.803824 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0573f69-b5ed-475b-b579-4c084744071d-kube-api-access-p9ln7" pod "c0573f69-b5ed-475b-b579-4c084744071d" (UID: "c0573f69-b5ed-475b-b579-4c084744071d"). InnerVolumeSpecName "kube-api-access-p9ln7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:41:17.903238 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:17.903204 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9ln7\" (UniqueName: \"kubernetes.io/projected/c0573f69-b5ed-475b-b579-4c084744071d-kube-api-access-p9ln7\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:17.903238 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:17.903235 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0573f69-b5ed-475b-b579-4c084744071d-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:17.903238 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:17.903247 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0573f69-b5ed-475b-b579-4c084744071d-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:18.089471 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:18.089365 6841 generic.go:356] "Generic (PLEG): container finished" podID="c0573f69-b5ed-475b-b579-4c084744071d" containerID="422b62b920308dffe52358f6b3789f68b12b68c364efa998fc42205a60435118" exitCode=0 Oct 31 13:41:18.089471 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:18.089442 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-107955-fc749446-559d9" Oct 31 13:41:18.089471 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:18.089443 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-107955-fc749446-559d9" event={"ID":"c0573f69-b5ed-475b-b579-4c084744071d","Type":"ContainerDied","Data":"422b62b920308dffe52358f6b3789f68b12b68c364efa998fc42205a60435118"} Oct 31 13:41:18.089756 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:18.089502 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-107955-fc749446-559d9" event={"ID":"c0573f69-b5ed-475b-b579-4c084744071d","Type":"ContainerDied","Data":"5146abf42ca284e3a03d1b086826171ce51d2a730429ceaff05bbf3429a968a1"} Oct 31 13:41:18.089756 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:18.089527 6841 scope.go:122] "RemoveContainer" containerID="422b62b920308dffe52358f6b3789f68b12b68c364efa998fc42205a60435118" Oct 31 13:41:18.100764 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:18.100725 6841 scope.go:122] "RemoveContainer" containerID="422b62b920308dffe52358f6b3789f68b12b68c364efa998fc42205a60435118" Oct 31 13:41:18.101113 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:41:18.101082 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"422b62b920308dffe52358f6b3789f68b12b68c364efa998fc42205a60435118\": container with ID starting with 422b62b920308dffe52358f6b3789f68b12b68c364efa998fc42205a60435118 not found: ID does not exist" containerID="422b62b920308dffe52358f6b3789f68b12b68c364efa998fc42205a60435118" Oct 31 13:41:18.101220 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:18.101121 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"422b62b920308dffe52358f6b3789f68b12b68c364efa998fc42205a60435118"} err="failed to get container status \"422b62b920308dffe52358f6b3789f68b12b68c364efa998fc42205a60435118\": rpc error: code = NotFound desc = could not find container \"422b62b920308dffe52358f6b3789f68b12b68c364efa998fc42205a60435118\": container with ID starting with 422b62b920308dffe52358f6b3789f68b12b68c364efa998fc42205a60435118 not found: ID does not exist" Oct 31 13:41:18.148299 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:18.148269 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-107955-fc749446-559d9"] Oct 31 13:41:18.159743 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:18.159711 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-107955-fc749446-559d9"] Oct 31 13:41:18.477347 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:18.477315 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:19.228531 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:19.228497 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:19.643321 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:19.643242 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0573f69-b5ed-475b-b579-4c084744071d" path="/var/lib/kubelet/pods/c0573f69-b5ed-475b-b579-4c084744071d/volumes" Oct 31 13:41:19.923742 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:19.923651 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:20.035446 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:41:20.035415 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:41:20.636446 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:20.636409 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:21.309115 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:21.309076 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:22.053596 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:22.053559 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:22.852777 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:22.852746 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:23.324681 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:23.324645 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:23.542716 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:23.542679 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:23.965998 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:23.965953 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:24.325786 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:24.325693 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:24.844472 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:24.844431 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:25.367482 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:25.367419 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-107955-86f6b68d45-ztxhj" Oct 31 13:41:25.518606 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:25.518565 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe-kube-api-access-cm6zb\" (UniqueName: \"kubernetes.io/projected/481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe-kube-api-access-cm6zb\") pod \"481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe\" (UID: \"481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe\") " Oct 31 13:41:25.518793 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:25.518778 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe-plugins-config-volume\") pod \"481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe\" (UID: \"481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe\") " Oct 31 13:41:25.519264 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:25.519235 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe-plugins-config-volume" pod "481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe" (UID: "481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:41:25.519390 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:25.519321 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:25.520687 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:25.520659 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe-kube-api-access-cm6zb" pod "481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe" (UID: "481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe"). InnerVolumeSpecName "kube-api-access-cm6zb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:41:25.621253 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:25.621177 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm6zb\" (UniqueName: \"kubernetes.io/projected/481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe-kube-api-access-cm6zb\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:25.710738 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:25.710708 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:26.156077 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:26.156038 6841 generic.go:356] "Generic (PLEG): container finished" podID="481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe" containerID="940d9c7a7e36edbdf013cfaf00a94be25cb3828c044f69f676a2736bb67af8d4" exitCode=0 Oct 31 13:41:26.156289 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:26.156085 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-107955-86f6b68d45-ztxhj" Oct 31 13:41:26.156289 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:26.156113 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-107955-86f6b68d45-ztxhj" event={"ID":"481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe","Type":"ContainerDied","Data":"940d9c7a7e36edbdf013cfaf00a94be25cb3828c044f69f676a2736bb67af8d4"} Oct 31 13:41:26.156289 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:26.156177 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-107955-86f6b68d45-ztxhj" event={"ID":"481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe","Type":"ContainerDied","Data":"08eec4465150d73f872aad4fcbb0869cad9f0e413b290082ba07ea55944d9023"} Oct 31 13:41:26.156289 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:26.156206 6841 scope.go:122] "RemoveContainer" containerID="940d9c7a7e36edbdf013cfaf00a94be25cb3828c044f69f676a2736bb67af8d4" Oct 31 13:41:26.166904 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:26.166887 6841 scope.go:122] "RemoveContainer" containerID="940d9c7a7e36edbdf013cfaf00a94be25cb3828c044f69f676a2736bb67af8d4" Oct 31 13:41:26.167231 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:41:26.167209 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"940d9c7a7e36edbdf013cfaf00a94be25cb3828c044f69f676a2736bb67af8d4\": container with ID starting with 940d9c7a7e36edbdf013cfaf00a94be25cb3828c044f69f676a2736bb67af8d4 not found: ID does not exist" containerID="940d9c7a7e36edbdf013cfaf00a94be25cb3828c044f69f676a2736bb67af8d4" Oct 31 13:41:26.167319 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:26.167244 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"940d9c7a7e36edbdf013cfaf00a94be25cb3828c044f69f676a2736bb67af8d4"} err="failed to get container status \"940d9c7a7e36edbdf013cfaf00a94be25cb3828c044f69f676a2736bb67af8d4\": rpc error: code = NotFound desc = could not find container \"940d9c7a7e36edbdf013cfaf00a94be25cb3828c044f69f676a2736bb67af8d4\": container with ID starting with 940d9c7a7e36edbdf013cfaf00a94be25cb3828c044f69f676a2736bb67af8d4 not found: ID does not exist" Oct 31 13:41:26.175196 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:26.175173 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-107955-86f6b68d45-ztxhj"] Oct 31 13:41:26.180124 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:26.180104 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-107955-86f6b68d45-ztxhj"] Oct 31 13:41:26.519122 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:26.519089 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:27.253271 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:27.253237 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:27.644770 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:27.644693 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe" path="/var/lib/kubelet/pods/481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe/volumes" Oct 31 13:41:28.047260 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:28.047225 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:28.825442 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:28.825407 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:29.603206 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:29.603176 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:30.395492 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:30.395459 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:31.313769 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:31.313729 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:31.531608 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:31.531568 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:32.517597 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:32.517562 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:33.520603 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:33.520570 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:33.761434 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:33.761397 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:34.598512 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:34.598477 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:34.931351 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:34.931259 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:35.307011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:35.294125 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:35.758340 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:35.758303 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:36.383070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:36.383029 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd"] Oct 31 13:41:36.383299 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:36.383268 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd" podUID="def1f5e7-9be6-4470-b70c-e387a4469d2f" containerName="maas-api" containerID="cri-o://2fa10498bcf16f65525b47ab458442f284135491eed9b7840c25940a5e9f0af3" gracePeriod=30 Oct 31 13:41:36.531822 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:36.531791 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:36.788921 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:36.788885 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd" Oct 31 13:41:36.835581 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:36.835443 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/def1f5e7-9be6-4470-b70c-e387a4469d2f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/def1f5e7-9be6-4470-b70c-e387a4469d2f-maas-api-tls\") pod \"def1f5e7-9be6-4470-b70c-e387a4469d2f\" (UID: \"def1f5e7-9be6-4470-b70c-e387a4469d2f\") " Oct 31 13:41:36.835581 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:36.835518 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/def1f5e7-9be6-4470-b70c-e387a4469d2f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/def1f5e7-9be6-4470-b70c-e387a4469d2f-metrics-certs\") pod \"def1f5e7-9be6-4470-b70c-e387a4469d2f\" (UID: \"def1f5e7-9be6-4470-b70c-e387a4469d2f\") " Oct 31 13:41:36.835929 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:36.835656 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/def1f5e7-9be6-4470-b70c-e387a4469d2f-kube-api-access-sdltx\" (UniqueName: \"kubernetes.io/projected/def1f5e7-9be6-4470-b70c-e387a4469d2f-kube-api-access-sdltx\") pod \"def1f5e7-9be6-4470-b70c-e387a4469d2f\" (UID: \"def1f5e7-9be6-4470-b70c-e387a4469d2f\") " Oct 31 13:41:36.835929 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:36.835685 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/def1f5e7-9be6-4470-b70c-e387a4469d2f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/def1f5e7-9be6-4470-b70c-e387a4469d2f-openshift-service-ca\") pod \"def1f5e7-9be6-4470-b70c-e387a4469d2f\" (UID: \"def1f5e7-9be6-4470-b70c-e387a4469d2f\") " Oct 31 13:41:36.836638 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:36.836612 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/def1f5e7-9be6-4470-b70c-e387a4469d2f-openshift-service-ca" pod "def1f5e7-9be6-4470-b70c-e387a4469d2f" (UID: "def1f5e7-9be6-4470-b70c-e387a4469d2f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:41:36.839544 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:36.839517 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/def1f5e7-9be6-4470-b70c-e387a4469d2f-metrics-certs" pod "def1f5e7-9be6-4470-b70c-e387a4469d2f" (UID: "def1f5e7-9be6-4470-b70c-e387a4469d2f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:41:36.841448 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:36.841421 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def1f5e7-9be6-4470-b70c-e387a4469d2f-kube-api-access-sdltx" pod "def1f5e7-9be6-4470-b70c-e387a4469d2f" (UID: "def1f5e7-9be6-4470-b70c-e387a4469d2f"). InnerVolumeSpecName "kube-api-access-sdltx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:41:36.848176 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:36.848151 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/def1f5e7-9be6-4470-b70c-e387a4469d2f-maas-api-tls" pod "def1f5e7-9be6-4470-b70c-e387a4469d2f" (UID: "def1f5e7-9be6-4470-b70c-e387a4469d2f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:41:36.938637 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:36.938529 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/def1f5e7-9be6-4470-b70c-e387a4469d2f-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:36.938637 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:36.938564 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/def1f5e7-9be6-4470-b70c-e387a4469d2f-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:36.938637 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:36.938578 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdltx\" (UniqueName: \"kubernetes.io/projected/def1f5e7-9be6-4470-b70c-e387a4469d2f-kube-api-access-sdltx\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:36.938637 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:36.938592 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/def1f5e7-9be6-4470-b70c-e387a4469d2f-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:36.956029 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:36.955004 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:37.239702 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:37.239671 6841 generic.go:356] "Generic (PLEG): container finished" podID="def1f5e7-9be6-4470-b70c-e387a4469d2f" containerID="2fa10498bcf16f65525b47ab458442f284135491eed9b7840c25940a5e9f0af3" exitCode=0 Oct 31 13:41:37.239878 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:37.239727 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd" Oct 31 13:41:37.239878 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:37.239760 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd" event={"ID":"def1f5e7-9be6-4470-b70c-e387a4469d2f","Type":"ContainerDied","Data":"2fa10498bcf16f65525b47ab458442f284135491eed9b7840c25940a5e9f0af3"} Oct 31 13:41:37.239878 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:37.239814 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd" event={"ID":"def1f5e7-9be6-4470-b70c-e387a4469d2f","Type":"ContainerDied","Data":"a4cd1256cb3a51ccd810f551cd1d712374d58cbb1e561951fab1ec5819cb9920"} Oct 31 13:41:37.239878 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:37.239841 6841 scope.go:122] "RemoveContainer" containerID="2fa10498bcf16f65525b47ab458442f284135491eed9b7840c25940a5e9f0af3" Oct 31 13:41:37.254719 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:37.254695 6841 scope.go:122] "RemoveContainer" containerID="2fa10498bcf16f65525b47ab458442f284135491eed9b7840c25940a5e9f0af3" Oct 31 13:41:37.255053 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:41:37.255033 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fa10498bcf16f65525b47ab458442f284135491eed9b7840c25940a5e9f0af3\": container with ID starting with 2fa10498bcf16f65525b47ab458442f284135491eed9b7840c25940a5e9f0af3 not found: ID does not exist" containerID="2fa10498bcf16f65525b47ab458442f284135491eed9b7840c25940a5e9f0af3" Oct 31 13:41:37.255117 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:37.255062 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fa10498bcf16f65525b47ab458442f284135491eed9b7840c25940a5e9f0af3"} err="failed to get container status \"2fa10498bcf16f65525b47ab458442f284135491eed9b7840c25940a5e9f0af3\": rpc error: code = NotFound desc = could not find container \"2fa10498bcf16f65525b47ab458442f284135491eed9b7840c25940a5e9f0af3\": container with ID starting with 2fa10498bcf16f65525b47ab458442f284135491eed9b7840c25940a5e9f0af3 not found: ID does not exist" Oct 31 13:41:37.269482 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:37.269456 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd"] Oct 31 13:41:37.273031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:37.273004 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-54a16b-588dc98b95-hpgjd"] Oct 31 13:41:37.644060 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:37.643962 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="def1f5e7-9be6-4470-b70c-e387a4469d2f" path="/var/lib/kubelet/pods/def1f5e7-9be6-4470-b70c-e387a4469d2f/volumes" Oct 31 13:41:37.795193 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:37.795157 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:38.602306 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:38.602274 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:39.650372 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:39.650334 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:40.534785 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:40.534741 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:40.858187 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:40.858099 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:41.129357 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:41.129328 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:41.334421 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:41.334388 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:41.719392 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:41.719356 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:41.967541 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:41.967500 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx"] Oct 31 13:41:41.968041 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:41.967761 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" podUID="25d7ee65-7958-458c-a202-693bd09893c7" containerName="istio-proxy" containerID="cri-o://33207ea1fd72f6e21d98e41f9682d2611d714ace7b906bb3ddce49bb96306f69" gracePeriod=30 Oct 31 13:41:42.087270 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:42.087160 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:42.379729 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:42.379635 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:42.813132 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:42.813093 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:43.747008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:43.743006 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:44.340184 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:44.340150 6841 patch_prober.go:29] interesting pod/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:41:44.340372 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:44.340203 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" podUID="25d7ee65-7958-458c-a202-693bd09893c7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:41:44.824881 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:44.824843 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:45.839747 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:45.839667 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:46.565398 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.565359 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:46.906471 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.906397 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11391-c38699"] Oct 31 13:41:46.907051 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.907035 6841 cpu_manager.go:414] "containerMap: removing container" podUID="c0573f69-b5ed-475b-b579-4c084744071d" containerName="maas-api" Oct 31 13:41:46.907106 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.907052 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0573f69-b5ed-475b-b579-4c084744071d" containerName="maas-api" Oct 31 13:41:46.907106 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.907063 6841 cpu_manager.go:414] "containerMap: removing container" podUID="a8494c61-00b4-41e6-b8ed-f4c2081f8944" containerName="istio-proxy" Oct 31 13:41:46.907106 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.907068 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8494c61-00b4-41e6-b8ed-f4c2081f8944" containerName="istio-proxy" Oct 31 13:41:46.907106 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.907085 6841 cpu_manager.go:414] "containerMap: removing container" podUID="8437acf6-79cd-4e8e-b0d5-9e81f3602748" containerName="maas-api" Oct 31 13:41:46.907106 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.907091 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8437acf6-79cd-4e8e-b0d5-9e81f3602748" containerName="maas-api" Oct 31 13:41:46.907574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.907109 6841 cpu_manager.go:414] "containerMap: removing container" podUID="3a4747b7-ff5a-4c37-b3f9-5db97fce5d12" containerName="maas-api" Oct 31 13:41:46.907574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.907113 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a4747b7-ff5a-4c37-b3f9-5db97fce5d12" containerName="maas-api" Oct 31 13:41:46.907574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.907123 6841 cpu_manager.go:414] "containerMap: removing container" podUID="c7a4d6d4-06ba-4385-94cc-66f96e07232a" containerName="istio-proxy" Oct 31 13:41:46.907574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.907127 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7a4d6d4-06ba-4385-94cc-66f96e07232a" containerName="istio-proxy" Oct 31 13:41:46.907574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.907138 6841 cpu_manager.go:414] "containerMap: removing container" podUID="d2a5911a-8b22-49ae-b976-dc61217c3dc2" containerName="maas-api" Oct 31 13:41:46.907574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.907143 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2a5911a-8b22-49ae-b976-dc61217c3dc2" containerName="maas-api" Oct 31 13:41:46.907574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.907152 6841 cpu_manager.go:414] "containerMap: removing container" podUID="c7a4d6d4-06ba-4385-94cc-66f96e07232a" containerName="istio-proxy" Oct 31 13:41:46.907574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.907156 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7a4d6d4-06ba-4385-94cc-66f96e07232a" containerName="istio-proxy" Oct 31 13:41:46.907574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.907165 6841 cpu_manager.go:414] "containerMap: removing container" podUID="eee4ba24-3680-41c1-adbf-c7432d2d20fc" containerName="payload-processing" Oct 31 13:41:46.907574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.907169 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eee4ba24-3680-41c1-adbf-c7432d2d20fc" containerName="payload-processing" Oct 31 13:41:46.907574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.907179 6841 cpu_manager.go:414] "containerMap: removing container" podUID="682058fc-61a5-4c39-a42f-ada4dc83a0cc" containerName="maas-api" Oct 31 13:41:46.907574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.907183 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="682058fc-61a5-4c39-a42f-ada4dc83a0cc" containerName="maas-api" Oct 31 13:41:46.907574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.907193 6841 cpu_manager.go:414] "containerMap: removing container" podUID="def1f5e7-9be6-4470-b70c-e387a4469d2f" containerName="maas-api" Oct 31 13:41:46.907574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.907198 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="def1f5e7-9be6-4470-b70c-e387a4469d2f" containerName="maas-api" Oct 31 13:41:46.907574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.907211 6841 cpu_manager.go:414] "containerMap: removing container" podUID="da83bd57-52af-48d3-bd76-42e36c18112f" containerName="main" Oct 31 13:41:46.907574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.907215 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da83bd57-52af-48d3-bd76-42e36c18112f" containerName="main" Oct 31 13:41:46.907574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.907222 6841 cpu_manager.go:414] "containerMap: removing container" podUID="481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe" containerName="payload-processing" Oct 31 13:41:46.907574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.907228 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe" containerName="payload-processing" Oct 31 13:41:46.907574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.907303 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7a4d6d4-06ba-4385-94cc-66f96e07232a" containerName="istio-proxy" Oct 31 13:41:46.907574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.907313 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2a5911a-8b22-49ae-b976-dc61217c3dc2" containerName="maas-api" Oct 31 13:41:46.907574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.907321 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7a4d6d4-06ba-4385-94cc-66f96e07232a" containerName="istio-proxy" Oct 31 13:41:46.907574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.907327 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="eee4ba24-3680-41c1-adbf-c7432d2d20fc" containerName="payload-processing" Oct 31 13:41:46.907574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.907333 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="682058fc-61a5-4c39-a42f-ada4dc83a0cc" containerName="maas-api" Oct 31 13:41:46.907574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.907339 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="def1f5e7-9be6-4470-b70c-e387a4469d2f" containerName="maas-api" Oct 31 13:41:46.907574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.907345 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8494c61-00b4-41e6-b8ed-f4c2081f8944" containerName="istio-proxy" Oct 31 13:41:46.907574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.907350 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="da83bd57-52af-48d3-bd76-42e36c18112f" containerName="main" Oct 31 13:41:46.907574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.907356 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="481c2a58-5caa-41aa-8a0b-bcf6ed63c6fe" containerName="payload-processing" Oct 31 13:41:46.907574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.907362 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0573f69-b5ed-475b-b579-4c084744071d" containerName="maas-api" Oct 31 13:41:46.907574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.907369 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="8437acf6-79cd-4e8e-b0d5-9e81f3602748" containerName="maas-api" Oct 31 13:41:46.907574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.907377 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a4747b7-ff5a-4c37-b3f9-5db97fce5d12" containerName="maas-api" Oct 31 13:41:46.911044 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.911026 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11391-c38699" Oct 31 13:41:46.916322 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.916301 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mn5fx\"" Oct 31 13:41:46.919857 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.919837 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11391-c38699"] Oct 31 13:41:46.945926 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:46.945896 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdjwk\" (UniqueName: \"kubernetes.io/projected/2030e0b5-c644-44d9-9dce-81024a3a84d0-kube-api-access-bdjwk\") pod \"test-curl-11391-c38699\" (UID: \"2030e0b5-c644-44d9-9dce-81024a3a84d0\") " pod="opendatahub/test-curl-11391-c38699" Oct 31 13:41:47.046951 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.046927 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdjwk\" (UniqueName: \"kubernetes.io/projected/2030e0b5-c644-44d9-9dce-81024a3a84d0-kube-api-access-bdjwk\") pod \"test-curl-11391-c38699\" (UID: \"2030e0b5-c644-44d9-9dce-81024a3a84d0\") " pod="opendatahub/test-curl-11391-c38699" Oct 31 13:41:47.076518 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.076488 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdjwk\" (UniqueName: \"kubernetes.io/projected/2030e0b5-c644-44d9-9dce-81024a3a84d0-kube-api-access-bdjwk\") pod \"test-curl-11391-c38699\" (UID: \"2030e0b5-c644-44d9-9dce-81024a3a84d0\") " pod="opendatahub/test-curl-11391-c38699" Oct 31 13:41:47.225012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.224967 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11391-c38699" Oct 31 13:41:47.264388 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.264362 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:41:47.376012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.375629 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25d7ee65-7958-458c-a202-693bd09893c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d7ee65-7958-458c-a202-693bd09893c7-istio-podinfo\") pod \"25d7ee65-7958-458c-a202-693bd09893c7\" (UID: \"25d7ee65-7958-458c-a202-693bd09893c7\") " Oct 31 13:41:47.376012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.375643 6841 generic.go:356] "Generic (PLEG): container finished" podID="25d7ee65-7958-458c-a202-693bd09893c7" containerID="33207ea1fd72f6e21d98e41f9682d2611d714ace7b906bb3ddce49bb96306f69" exitCode=0 Oct 31 13:41:47.376012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.375702 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d7ee65-7958-458c-a202-693bd09893c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d7ee65-7958-458c-a202-693bd09893c7-istio-envoy\") pod \"25d7ee65-7958-458c-a202-693bd09893c7\" (UID: \"25d7ee65-7958-458c-a202-693bd09893c7\") " Oct 31 13:41:47.376012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.375775 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d7ee65-7958-458c-a202-693bd09893c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d7ee65-7958-458c-a202-693bd09893c7-workload-socket\") pod \"25d7ee65-7958-458c-a202-693bd09893c7\" (UID: \"25d7ee65-7958-458c-a202-693bd09893c7\") " Oct 31 13:41:47.376012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.375815 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25d7ee65-7958-458c-a202-693bd09893c7-kube-api-access-gq7b8\" (UniqueName: \"kubernetes.io/projected/25d7ee65-7958-458c-a202-693bd09893c7-kube-api-access-gq7b8\") pod \"25d7ee65-7958-458c-a202-693bd09893c7\" (UID: \"25d7ee65-7958-458c-a202-693bd09893c7\") " Oct 31 13:41:47.376012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.375897 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d7ee65-7958-458c-a202-693bd09893c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d7ee65-7958-458c-a202-693bd09893c7-istio-data\") pod \"25d7ee65-7958-458c-a202-693bd09893c7\" (UID: \"25d7ee65-7958-458c-a202-693bd09893c7\") " Oct 31 13:41:47.376821 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.376608 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25d7ee65-7958-458c-a202-693bd09893c7-istio-token\" (UniqueName: \"kubernetes.io/projected/25d7ee65-7958-458c-a202-693bd09893c7-istio-token\") pod \"25d7ee65-7958-458c-a202-693bd09893c7\" (UID: \"25d7ee65-7958-458c-a202-693bd09893c7\") " Oct 31 13:41:47.376821 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.376661 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d7ee65-7958-458c-a202-693bd09893c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d7ee65-7958-458c-a202-693bd09893c7-workload-certs\") pod \"25d7ee65-7958-458c-a202-693bd09893c7\" (UID: \"25d7ee65-7958-458c-a202-693bd09893c7\") " Oct 31 13:41:47.378275 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.377190 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25d7ee65-7958-458c-a202-693bd09893c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d7ee65-7958-458c-a202-693bd09893c7-istiod-ca-cert\") pod \"25d7ee65-7958-458c-a202-693bd09893c7\" (UID: \"25d7ee65-7958-458c-a202-693bd09893c7\") " Oct 31 13:41:47.378275 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.377274 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d7ee65-7958-458c-a202-693bd09893c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d7ee65-7958-458c-a202-693bd09893c7-credential-socket\") pod \"25d7ee65-7958-458c-a202-693bd09893c7\" (UID: \"25d7ee65-7958-458c-a202-693bd09893c7\") " Oct 31 13:41:47.379319 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.379086 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d7ee65-7958-458c-a202-693bd09893c7-credential-socket" pod "25d7ee65-7958-458c-a202-693bd09893c7" (UID: "25d7ee65-7958-458c-a202-693bd09893c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:47.379830 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.379501 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d7ee65-7958-458c-a202-693bd09893c7-istio-data" pod "25d7ee65-7958-458c-a202-693bd09893c7" (UID: "25d7ee65-7958-458c-a202-693bd09893c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:47.381184 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.379945 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" Oct 31 13:41:47.381184 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.375896 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" event={"ID":"25d7ee65-7958-458c-a202-693bd09893c7","Type":"ContainerDied","Data":"33207ea1fd72f6e21d98e41f9682d2611d714ace7b906bb3ddce49bb96306f69"} Oct 31 13:41:47.381184 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.380424 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx" event={"ID":"25d7ee65-7958-458c-a202-693bd09893c7","Type":"ContainerDied","Data":"60d9cbb0fa30f67d65613da242db24962885c601581160a4742dfb044160f073"} Oct 31 13:41:47.381184 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.380471 6841 scope.go:122] "RemoveContainer" containerID="33207ea1fd72f6e21d98e41f9682d2611d714ace7b906bb3ddce49bb96306f69" Oct 31 13:41:47.381184 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.380731 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25d7ee65-7958-458c-a202-693bd09893c7-istiod-ca-cert" pod "25d7ee65-7958-458c-a202-693bd09893c7" (UID: "25d7ee65-7958-458c-a202-693bd09893c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:41:47.381184 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.381044 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d7ee65-7958-458c-a202-693bd09893c7-workload-socket" pod "25d7ee65-7958-458c-a202-693bd09893c7" (UID: "25d7ee65-7958-458c-a202-693bd09893c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:47.381184 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.381141 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d7ee65-7958-458c-a202-693bd09893c7-workload-certs" pod "25d7ee65-7958-458c-a202-693bd09893c7" (UID: "25d7ee65-7958-458c-a202-693bd09893c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:47.383099 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.383022 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d7ee65-7958-458c-a202-693bd09893c7-istio-token" pod "25d7ee65-7958-458c-a202-693bd09893c7" (UID: "25d7ee65-7958-458c-a202-693bd09893c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:41:47.386486 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.385204 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25d7ee65-7958-458c-a202-693bd09893c7-istio-podinfo" pod "25d7ee65-7958-458c-a202-693bd09893c7" (UID: "25d7ee65-7958-458c-a202-693bd09893c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:41:47.386486 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.385866 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d7ee65-7958-458c-a202-693bd09893c7-kube-api-access-gq7b8" pod "25d7ee65-7958-458c-a202-693bd09893c7" (UID: "25d7ee65-7958-458c-a202-693bd09893c7"). InnerVolumeSpecName "kube-api-access-gq7b8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:41:47.389203 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.389182 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d7ee65-7958-458c-a202-693bd09893c7-istio-envoy" pod "25d7ee65-7958-458c-a202-693bd09893c7" (UID: "25d7ee65-7958-458c-a202-693bd09893c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:47.407045 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.407009 6841 scope.go:122] "RemoveContainer" containerID="e838473111f333938c845e3e045d0bcac8a456dbcaa3563781125ba6980845cb" Oct 31 13:41:47.417648 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.417631 6841 scope.go:122] "RemoveContainer" containerID="33207ea1fd72f6e21d98e41f9682d2611d714ace7b906bb3ddce49bb96306f69" Oct 31 13:41:47.417944 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:41:47.417923 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33207ea1fd72f6e21d98e41f9682d2611d714ace7b906bb3ddce49bb96306f69\": container with ID starting with 33207ea1fd72f6e21d98e41f9682d2611d714ace7b906bb3ddce49bb96306f69 not found: ID does not exist" containerID="33207ea1fd72f6e21d98e41f9682d2611d714ace7b906bb3ddce49bb96306f69" Oct 31 13:41:47.418060 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.417954 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33207ea1fd72f6e21d98e41f9682d2611d714ace7b906bb3ddce49bb96306f69"} err="failed to get container status \"33207ea1fd72f6e21d98e41f9682d2611d714ace7b906bb3ddce49bb96306f69\": rpc error: code = NotFound desc = could not find container \"33207ea1fd72f6e21d98e41f9682d2611d714ace7b906bb3ddce49bb96306f69\": container with ID starting with 33207ea1fd72f6e21d98e41f9682d2611d714ace7b906bb3ddce49bb96306f69 not found: ID does not exist" Oct 31 13:41:47.418060 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.417973 6841 scope.go:122] "RemoveContainer" containerID="e838473111f333938c845e3e045d0bcac8a456dbcaa3563781125ba6980845cb" Oct 31 13:41:47.418252 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:41:47.418235 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e838473111f333938c845e3e045d0bcac8a456dbcaa3563781125ba6980845cb\": container with ID starting with e838473111f333938c845e3e045d0bcac8a456dbcaa3563781125ba6980845cb not found: ID does not exist" containerID="e838473111f333938c845e3e045d0bcac8a456dbcaa3563781125ba6980845cb" Oct 31 13:41:47.418306 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.418259 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e838473111f333938c845e3e045d0bcac8a456dbcaa3563781125ba6980845cb"} err="failed to get container status \"e838473111f333938c845e3e045d0bcac8a456dbcaa3563781125ba6980845cb\": rpc error: code = NotFound desc = could not find container \"e838473111f333938c845e3e045d0bcac8a456dbcaa3563781125ba6980845cb\": container with ID starting with e838473111f333938c845e3e045d0bcac8a456dbcaa3563781125ba6980845cb not found: ID does not exist" Oct 31 13:41:47.479067 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.479001 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d7ee65-7958-458c-a202-693bd09893c7-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:47.479067 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.479030 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d7ee65-7958-458c-a202-693bd09893c7-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:47.479067 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.479043 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d7ee65-7958-458c-a202-693bd09893c7-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:47.479067 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.479052 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d7ee65-7958-458c-a202-693bd09893c7-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:47.479067 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.479061 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq7b8\" (UniqueName: \"kubernetes.io/projected/25d7ee65-7958-458c-a202-693bd09893c7-kube-api-access-gq7b8\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:47.479067 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.479072 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d7ee65-7958-458c-a202-693bd09893c7-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:47.479370 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.479080 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d7ee65-7958-458c-a202-693bd09893c7-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:47.479370 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.479090 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d7ee65-7958-458c-a202-693bd09893c7-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:47.479370 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.479100 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d7ee65-7958-458c-a202-693bd09893c7-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:47.511849 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:41:47.511220 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2030e0b5_c644_44d9_9dce_81024a3a84d0.slice/crio-63d74557a4e449ba550edf6389678ef3894a15ac35936d0a380154150e50b1f9 WatchSource:0}: Error finding container 63d74557a4e449ba550edf6389678ef3894a15ac35936d0a380154150e50b1f9: Status 404 returned error can't find the container with id 63d74557a4e449ba550edf6389678ef3894a15ac35936d0a380154150e50b1f9 Oct 31 13:41:47.511849 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.511325 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11391-c38699"] Oct 31 13:41:47.511849 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.511365 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:47.712204 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.712166 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx"] Oct 31 13:41:47.715439 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:47.715408 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-54a16b-openshift-default-54ff697745-tkfgx"] Oct 31 13:41:48.150890 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:48.150847 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:48.378056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:48.366660 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:48.431003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:48.429115 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11391-c38699" event={"ID":"2030e0b5-c644-44d9-9dce-81024a3a84d0","Type":"ContainerStarted","Data":"aacfcdc39366faa54d44f3aac75f5a41a7de163795b0946e50af9c8d5876a6fd"} Oct 31 13:41:48.431003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:48.429191 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11391-c38699" event={"ID":"2030e0b5-c644-44d9-9dce-81024a3a84d0","Type":"ContainerStarted","Data":"63d74557a4e449ba550edf6389678ef3894a15ac35936d0a380154150e50b1f9"} Oct 31 13:41:48.466317 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:48.466254 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11391-c38699" podStartSLOduration=2.466235197 podStartE2EDuration="2.466235197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:41:48.46416601 +0000 UTC m=+4154.947288912" watchObservedRunningTime="2026-10-31 13:41:48.466235197 +0000 UTC m=+4154.949358095" Oct 31 13:41:48.612931 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:48.612888 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:49.270007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:49.256257 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11391-c38699"] Oct 31 13:41:49.292300 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:49.290648 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11391-c38699"] Oct 31 13:41:49.387595 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:49.387557 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:49.440764 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:49.440722 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11391-77d074"] Oct 31 13:41:49.441493 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:49.441472 6841 cpu_manager.go:414] "containerMap: removing container" podUID="2030e0b5-c644-44d9-9dce-81024a3a84d0" containerName="test-curl-11391-c38699" Oct 31 13:41:49.441624 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:49.441493 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2030e0b5-c644-44d9-9dce-81024a3a84d0" containerName="test-curl-11391-c38699" Oct 31 13:41:49.441624 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:49.441541 6841 cpu_manager.go:414] "containerMap: removing container" podUID="25d7ee65-7958-458c-a202-693bd09893c7" containerName="istio-proxy" Oct 31 13:41:49.441624 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:49.441550 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25d7ee65-7958-458c-a202-693bd09893c7" containerName="istio-proxy" Oct 31 13:41:49.441624 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:49.441580 6841 cpu_manager.go:414] "containerMap: removing container" podUID="a8494c61-00b4-41e6-b8ed-f4c2081f8944" containerName="istio-proxy" Oct 31 13:41:49.441624 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:49.441588 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8494c61-00b4-41e6-b8ed-f4c2081f8944" containerName="istio-proxy" Oct 31 13:41:49.441893 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:49.441741 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="25d7ee65-7958-458c-a202-693bd09893c7" containerName="istio-proxy" Oct 31 13:41:49.441893 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:49.441761 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="2030e0b5-c644-44d9-9dce-81024a3a84d0" containerName="test-curl-11391-c38699" Oct 31 13:41:49.441893 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:49.441780 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8494c61-00b4-41e6-b8ed-f4c2081f8944" containerName="istio-proxy" Oct 31 13:41:49.445617 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:49.445596 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11391-77d074" Oct 31 13:41:49.455243 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:49.455219 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11391-77d074"] Oct 31 13:41:49.529761 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:49.529679 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs95x\" (UniqueName: \"kubernetes.io/projected/99509d76-474b-4b4b-b354-19cf3084ce0d-kube-api-access-gs95x\") pod \"test-curl-11391-77d074\" (UID: \"99509d76-474b-4b4b-b354-19cf3084ce0d\") " pod="opendatahub/test-curl-11391-77d074" Oct 31 13:41:49.630818 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:49.630785 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs95x\" (UniqueName: \"kubernetes.io/projected/99509d76-474b-4b4b-b354-19cf3084ce0d-kube-api-access-gs95x\") pod \"test-curl-11391-77d074\" (UID: \"99509d76-474b-4b4b-b354-19cf3084ce0d\") " pod="opendatahub/test-curl-11391-77d074" Oct 31 13:41:49.643090 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:49.643061 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25d7ee65-7958-458c-a202-693bd09893c7" path="/var/lib/kubelet/pods/25d7ee65-7958-458c-a202-693bd09893c7/volumes" Oct 31 13:41:49.644103 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:49.644084 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs95x\" (UniqueName: \"kubernetes.io/projected/99509d76-474b-4b4b-b354-19cf3084ce0d-kube-api-access-gs95x\") pod \"test-curl-11391-77d074\" (UID: \"99509d76-474b-4b4b-b354-19cf3084ce0d\") " pod="opendatahub/test-curl-11391-77d074" Oct 31 13:41:49.760579 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:49.760543 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11391-77d074" Oct 31 13:41:50.037522 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:41:50.037488 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:41:50.042932 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:50.042904 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11391-77d074"] Oct 31 13:41:50.044187 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:41:50.044152 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99509d76_474b_4b4b_b354_19cf3084ce0d.slice/crio-cfec46f1cbb308b0a0c7506c768d33486e2db3d37489e908ef686949c753960b WatchSource:0}: Error finding container cfec46f1cbb308b0a0c7506c768d33486e2db3d37489e908ef686949c753960b: Status 404 returned error can't find the container with id cfec46f1cbb308b0a0c7506c768d33486e2db3d37489e908ef686949c753960b Oct 31 13:41:50.218286 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:50.218257 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:50.454124 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:50.454085 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11391-77d074" event={"ID":"99509d76-474b-4b4b-b354-19cf3084ce0d","Type":"ContainerStarted","Data":"0da91cbeae3a80e23aca168b2d5ff272439b67edf18a24733bc32cc29d37bfb1"} Oct 31 13:41:50.454618 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:50.454121 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11391-c38699" podUID="2030e0b5-c644-44d9-9dce-81024a3a84d0" containerName="test-curl-11391-c38699" containerID="cri-o://aacfcdc39366faa54d44f3aac75f5a41a7de163795b0946e50af9c8d5876a6fd" gracePeriod=2 Oct 31 13:41:50.454618 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:50.454129 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11391-77d074" event={"ID":"99509d76-474b-4b4b-b354-19cf3084ce0d","Type":"ContainerStarted","Data":"cfec46f1cbb308b0a0c7506c768d33486e2db3d37489e908ef686949c753960b"} Oct 31 13:41:50.475029 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:50.474954 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11391-77d074" podStartSLOduration=1.474940031 podStartE2EDuration="1.474940031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:41:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:41:50.473391078 +0000 UTC m=+4156.956513980" watchObservedRunningTime="2026-10-31 13:41:50.474940031 +0000 UTC m=+4156.958063264" Oct 31 13:41:50.976543 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:50.976502 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:51.188703 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:51.188673 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11391-77d074"] Oct 31 13:41:51.189656 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:51.189625 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:51.191852 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:51.191826 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11391-77d074"] Oct 31 13:41:51.798795 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:51.798761 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:52.485829 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:52.485770 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11391-77d074" podUID="99509d76-474b-4b4b-b354-19cf3084ce0d" containerName="test-curl-11391-77d074" containerID="cri-o://0da91cbeae3a80e23aca168b2d5ff272439b67edf18a24733bc32cc29d37bfb1" gracePeriod=2 Oct 31 13:41:52.522389 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:52.522355 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:52.660704 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:52.660674 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11391-c38699" Oct 31 13:41:52.779673 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:52.779590 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2030e0b5-c644-44d9-9dce-81024a3a84d0-kube-api-access-bdjwk\" (UniqueName: \"kubernetes.io/projected/2030e0b5-c644-44d9-9dce-81024a3a84d0-kube-api-access-bdjwk\") pod \"2030e0b5-c644-44d9-9dce-81024a3a84d0\" (UID: \"2030e0b5-c644-44d9-9dce-81024a3a84d0\") " Oct 31 13:41:52.781713 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:52.781671 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2030e0b5-c644-44d9-9dce-81024a3a84d0-kube-api-access-bdjwk" pod "2030e0b5-c644-44d9-9dce-81024a3a84d0" (UID: "2030e0b5-c644-44d9-9dce-81024a3a84d0"). InnerVolumeSpecName "kube-api-access-bdjwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:41:52.881900 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:52.881869 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdjwk\" (UniqueName: \"kubernetes.io/projected/2030e0b5-c644-44d9-9dce-81024a3a84d0-kube-api-access-bdjwk\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:53.352623 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:53.352563 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:53.494513 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:53.494470 6841 generic.go:356] "Generic (PLEG): container finished" podID="2030e0b5-c644-44d9-9dce-81024a3a84d0" containerID="aacfcdc39366faa54d44f3aac75f5a41a7de163795b0946e50af9c8d5876a6fd" exitCode=137 Oct 31 13:41:53.494701 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:53.494638 6841 scope.go:122] "RemoveContainer" containerID="aacfcdc39366faa54d44f3aac75f5a41a7de163795b0946e50af9c8d5876a6fd" Oct 31 13:41:53.494820 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:53.494804 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11391-c38699" Oct 31 13:41:53.509551 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:41:53.509516 6841 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11391-c38699\" is forbidden: User \"system:node:ip-10-0-138-70.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-70.us-east-2.compute.internal' and this object" podUID="2030e0b5-c644-44d9-9dce-81024a3a84d0" pod="opendatahub/test-curl-11391-c38699" Oct 31 13:41:53.511281 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:53.511261 6841 scope.go:122] "RemoveContainer" containerID="aacfcdc39366faa54d44f3aac75f5a41a7de163795b0946e50af9c8d5876a6fd" Oct 31 13:41:53.511576 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:41:53.511555 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aacfcdc39366faa54d44f3aac75f5a41a7de163795b0946e50af9c8d5876a6fd\": container with ID starting with aacfcdc39366faa54d44f3aac75f5a41a7de163795b0946e50af9c8d5876a6fd not found: ID does not exist" containerID="aacfcdc39366faa54d44f3aac75f5a41a7de163795b0946e50af9c8d5876a6fd" Oct 31 13:41:53.511622 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:53.511587 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aacfcdc39366faa54d44f3aac75f5a41a7de163795b0946e50af9c8d5876a6fd"} err="failed to get container status \"aacfcdc39366faa54d44f3aac75f5a41a7de163795b0946e50af9c8d5876a6fd\": rpc error: code = NotFound desc = could not find container \"aacfcdc39366faa54d44f3aac75f5a41a7de163795b0946e50af9c8d5876a6fd\": container with ID starting with aacfcdc39366faa54d44f3aac75f5a41a7de163795b0946e50af9c8d5876a6fd not found: ID does not exist" Oct 31 13:41:53.641666 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:41:53.641583 6841 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11391-c38699\" is forbidden: User \"system:node:ip-10-0-138-70.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-70.us-east-2.compute.internal' and this object" podUID="2030e0b5-c644-44d9-9dce-81024a3a84d0" pod="opendatahub/test-curl-11391-c38699" Oct 31 13:41:53.642990 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:53.642963 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2030e0b5-c644-44d9-9dce-81024a3a84d0" path="/var/lib/kubelet/pods/2030e0b5-c644-44d9-9dce-81024a3a84d0/volumes" Oct 31 13:41:54.122733 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:54.122695 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:54.695554 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:54.695530 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11391-77d074" Oct 31 13:41:54.822326 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:54.822245 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99509d76-474b-4b4b-b354-19cf3084ce0d-kube-api-access-gs95x\" (UniqueName: \"kubernetes.io/projected/99509d76-474b-4b4b-b354-19cf3084ce0d-kube-api-access-gs95x\") pod \"99509d76-474b-4b4b-b354-19cf3084ce0d\" (UID: \"99509d76-474b-4b4b-b354-19cf3084ce0d\") " Oct 31 13:41:54.824437 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:54.824407 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99509d76-474b-4b4b-b354-19cf3084ce0d-kube-api-access-gs95x" pod "99509d76-474b-4b4b-b354-19cf3084ce0d" (UID: "99509d76-474b-4b4b-b354-19cf3084ce0d"). InnerVolumeSpecName "kube-api-access-gs95x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:41:54.923816 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:54.923782 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs95x\" (UniqueName: \"kubernetes.io/projected/99509d76-474b-4b4b-b354-19cf3084ce0d-kube-api-access-gs95x\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:54.929886 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:54.929852 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:55.512271 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:55.512235 6841 generic.go:356] "Generic (PLEG): container finished" podID="99509d76-474b-4b4b-b354-19cf3084ce0d" containerID="0da91cbeae3a80e23aca168b2d5ff272439b67edf18a24733bc32cc29d37bfb1" exitCode=137 Oct 31 13:41:55.512717 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:55.512276 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11391-77d074" Oct 31 13:41:55.512717 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:55.512322 6841 scope.go:122] "RemoveContainer" containerID="0da91cbeae3a80e23aca168b2d5ff272439b67edf18a24733bc32cc29d37bfb1" Oct 31 13:41:55.522485 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:55.522468 6841 scope.go:122] "RemoveContainer" containerID="0da91cbeae3a80e23aca168b2d5ff272439b67edf18a24733bc32cc29d37bfb1" Oct 31 13:41:55.522756 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:41:55.522731 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0da91cbeae3a80e23aca168b2d5ff272439b67edf18a24733bc32cc29d37bfb1\": container with ID starting with 0da91cbeae3a80e23aca168b2d5ff272439b67edf18a24733bc32cc29d37bfb1 not found: ID does not exist" containerID="0da91cbeae3a80e23aca168b2d5ff272439b67edf18a24733bc32cc29d37bfb1" Oct 31 13:41:55.522827 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:55.522763 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0da91cbeae3a80e23aca168b2d5ff272439b67edf18a24733bc32cc29d37bfb1"} err="failed to get container status \"0da91cbeae3a80e23aca168b2d5ff272439b67edf18a24733bc32cc29d37bfb1\": rpc error: code = NotFound desc = could not find container \"0da91cbeae3a80e23aca168b2d5ff272439b67edf18a24733bc32cc29d37bfb1\": container with ID starting with 0da91cbeae3a80e23aca168b2d5ff272439b67edf18a24733bc32cc29d37bfb1 not found: ID does not exist" Oct 31 13:41:55.523625 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:41:55.523602 6841 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11391-77d074\" is forbidden: User \"system:node:ip-10-0-138-70.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-70.us-east-2.compute.internal' and this object" podUID="99509d76-474b-4b4b-b354-19cf3084ce0d" pod="opendatahub/test-curl-11391-77d074" Oct 31 13:41:55.667132 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:55.667100 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99509d76-474b-4b4b-b354-19cf3084ce0d" path="/var/lib/kubelet/pods/99509d76-474b-4b4b-b354-19cf3084ce0d/volumes" Oct 31 13:41:55.691507 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:55.691460 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:56.426817 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:56.426772 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:57.325381 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:57.325332 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:58.104163 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:58.104128 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:58.315537 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:58.315502 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:59.008976 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:59.008920 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:41:59.768759 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:41:59.768722 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:00.574626 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:00.574577 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:02.021773 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:02.021724 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:02.644802 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:02.644764 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:03.378269 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:03.378213 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:04.131091 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:04.131054 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:04.910649 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:04.910614 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:05.123551 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:05.123491 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:05.407494 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:05.407443 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:05.677787 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:05.677710 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4"] Oct 31 13:42:05.678039 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:05.677966 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" podUID="42529d86-63e7-47dd-a03d-3e2ca7737a52" containerName="istio-proxy" containerID="cri-o://8baeb9c963c3d70611ee309af6bfe62949a32294f521ce2196f83ac19feb28f7" gracePeriod=30 Oct 31 13:42:05.834780 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:05.834749 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:06.052408 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:06.052372 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:06.527775 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:06.527740 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:06.772201 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:06.772166 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:07.597243 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:07.597203 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:08.308077 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:08.308043 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:09.028822 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:09.028759 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:09.807381 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:09.807347 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:10.114168 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:10.114087 6841 patch_prober.go:29] interesting pod/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:42:10.114168 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:10.114150 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" podUID="42529d86-63e7-47dd-a03d-3e2ca7737a52" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:42:10.538457 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:10.538418 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:10.945572 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:10.945550 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:42:11.100207 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.100121 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42529d86-63e7-47dd-a03d-3e2ca7737a52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42529d86-63e7-47dd-a03d-3e2ca7737a52-istiod-ca-cert\") pod \"42529d86-63e7-47dd-a03d-3e2ca7737a52\" (UID: \"42529d86-63e7-47dd-a03d-3e2ca7737a52\") " Oct 31 13:42:11.100207 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.100192 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42529d86-63e7-47dd-a03d-3e2ca7737a52-kube-api-access-s8jfg\" (UniqueName: \"kubernetes.io/projected/42529d86-63e7-47dd-a03d-3e2ca7737a52-kube-api-access-s8jfg\") pod \"42529d86-63e7-47dd-a03d-3e2ca7737a52\" (UID: \"42529d86-63e7-47dd-a03d-3e2ca7737a52\") " Oct 31 13:42:11.100469 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.100211 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42529d86-63e7-47dd-a03d-3e2ca7737a52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42529d86-63e7-47dd-a03d-3e2ca7737a52-workload-certs\") pod \"42529d86-63e7-47dd-a03d-3e2ca7737a52\" (UID: \"42529d86-63e7-47dd-a03d-3e2ca7737a52\") " Oct 31 13:42:11.100469 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.100310 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42529d86-63e7-47dd-a03d-3e2ca7737a52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42529d86-63e7-47dd-a03d-3e2ca7737a52-credential-socket\") pod \"42529d86-63e7-47dd-a03d-3e2ca7737a52\" (UID: \"42529d86-63e7-47dd-a03d-3e2ca7737a52\") " Oct 31 13:42:11.100469 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.100364 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42529d86-63e7-47dd-a03d-3e2ca7737a52-istio-token\" (UniqueName: \"kubernetes.io/projected/42529d86-63e7-47dd-a03d-3e2ca7737a52-istio-token\") pod \"42529d86-63e7-47dd-a03d-3e2ca7737a52\" (UID: \"42529d86-63e7-47dd-a03d-3e2ca7737a52\") " Oct 31 13:42:11.100469 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.100387 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42529d86-63e7-47dd-a03d-3e2ca7737a52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42529d86-63e7-47dd-a03d-3e2ca7737a52-istio-envoy\") pod \"42529d86-63e7-47dd-a03d-3e2ca7737a52\" (UID: \"42529d86-63e7-47dd-a03d-3e2ca7737a52\") " Oct 31 13:42:11.100469 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.100437 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42529d86-63e7-47dd-a03d-3e2ca7737a52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42529d86-63e7-47dd-a03d-3e2ca7737a52-istio-podinfo\") pod \"42529d86-63e7-47dd-a03d-3e2ca7737a52\" (UID: \"42529d86-63e7-47dd-a03d-3e2ca7737a52\") " Oct 31 13:42:11.100725 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.100521 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42529d86-63e7-47dd-a03d-3e2ca7737a52-workload-certs" pod "42529d86-63e7-47dd-a03d-3e2ca7737a52" (UID: "42529d86-63e7-47dd-a03d-3e2ca7737a52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:11.100725 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.100552 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42529d86-63e7-47dd-a03d-3e2ca7737a52-credential-socket" pod "42529d86-63e7-47dd-a03d-3e2ca7737a52" (UID: "42529d86-63e7-47dd-a03d-3e2ca7737a52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:11.100725 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.100567 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42529d86-63e7-47dd-a03d-3e2ca7737a52-istiod-ca-cert" pod "42529d86-63e7-47dd-a03d-3e2ca7737a52" (UID: "42529d86-63e7-47dd-a03d-3e2ca7737a52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:42:11.100725 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.100575 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42529d86-63e7-47dd-a03d-3e2ca7737a52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42529d86-63e7-47dd-a03d-3e2ca7737a52-workload-socket\") pod \"42529d86-63e7-47dd-a03d-3e2ca7737a52\" (UID: \"42529d86-63e7-47dd-a03d-3e2ca7737a52\") " Oct 31 13:42:11.100725 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.100690 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42529d86-63e7-47dd-a03d-3e2ca7737a52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42529d86-63e7-47dd-a03d-3e2ca7737a52-istio-data\") pod \"42529d86-63e7-47dd-a03d-3e2ca7737a52\" (UID: \"42529d86-63e7-47dd-a03d-3e2ca7737a52\") " Oct 31 13:42:11.101042 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.100921 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42529d86-63e7-47dd-a03d-3e2ca7737a52-workload-socket" pod "42529d86-63e7-47dd-a03d-3e2ca7737a52" (UID: "42529d86-63e7-47dd-a03d-3e2ca7737a52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:11.101042 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.100976 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42529d86-63e7-47dd-a03d-3e2ca7737a52-istio-data" pod "42529d86-63e7-47dd-a03d-3e2ca7737a52" (UID: "42529d86-63e7-47dd-a03d-3e2ca7737a52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:11.101477 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.101452 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42529d86-63e7-47dd-a03d-3e2ca7737a52-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:11.101477 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.101479 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42529d86-63e7-47dd-a03d-3e2ca7737a52-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:11.101683 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.101494 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42529d86-63e7-47dd-a03d-3e2ca7737a52-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:11.101683 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.101509 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42529d86-63e7-47dd-a03d-3e2ca7737a52-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:11.101683 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.101525 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42529d86-63e7-47dd-a03d-3e2ca7737a52-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:11.103132 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.103107 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42529d86-63e7-47dd-a03d-3e2ca7737a52-istio-podinfo" pod "42529d86-63e7-47dd-a03d-3e2ca7737a52" (UID: "42529d86-63e7-47dd-a03d-3e2ca7737a52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:42:11.103292 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.103221 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42529d86-63e7-47dd-a03d-3e2ca7737a52-kube-api-access-s8jfg" pod "42529d86-63e7-47dd-a03d-3e2ca7737a52" (UID: "42529d86-63e7-47dd-a03d-3e2ca7737a52"). InnerVolumeSpecName "kube-api-access-s8jfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:42:11.103292 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.103251 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42529d86-63e7-47dd-a03d-3e2ca7737a52-istio-token" pod "42529d86-63e7-47dd-a03d-3e2ca7737a52" (UID: "42529d86-63e7-47dd-a03d-3e2ca7737a52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:42:11.103616 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.103590 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42529d86-63e7-47dd-a03d-3e2ca7737a52-istio-envoy" pod "42529d86-63e7-47dd-a03d-3e2ca7737a52" (UID: "42529d86-63e7-47dd-a03d-3e2ca7737a52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:11.202682 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.202643 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8jfg\" (UniqueName: \"kubernetes.io/projected/42529d86-63e7-47dd-a03d-3e2ca7737a52-kube-api-access-s8jfg\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:11.202682 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.202682 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42529d86-63e7-47dd-a03d-3e2ca7737a52-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:11.203138 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.202692 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42529d86-63e7-47dd-a03d-3e2ca7737a52-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:11.203138 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.202704 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42529d86-63e7-47dd-a03d-3e2ca7737a52-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:11.276283 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.276244 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:11.655036 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.654994 6841 generic.go:356] "Generic (PLEG): container finished" podID="42529d86-63e7-47dd-a03d-3e2ca7737a52" containerID="8baeb9c963c3d70611ee309af6bfe62949a32294f521ce2196f83ac19feb28f7" exitCode=0 Oct 31 13:42:11.655213 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.655032 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" event={"ID":"42529d86-63e7-47dd-a03d-3e2ca7737a52","Type":"ContainerDied","Data":"8baeb9c963c3d70611ee309af6bfe62949a32294f521ce2196f83ac19feb28f7"} Oct 31 13:42:11.655213 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.655066 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" Oct 31 13:42:11.655213 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.655096 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4" event={"ID":"42529d86-63e7-47dd-a03d-3e2ca7737a52","Type":"ContainerDied","Data":"50c7c13fd9d99327dd53fe5547b26d79250f686dac5f222b1c29b6837621a6c3"} Oct 31 13:42:11.655213 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.655126 6841 scope.go:122] "RemoveContainer" containerID="8baeb9c963c3d70611ee309af6bfe62949a32294f521ce2196f83ac19feb28f7" Oct 31 13:42:11.667070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.667050 6841 scope.go:122] "RemoveContainer" containerID="705f880daff07ec87b8b01f01987326bf511cff6093825ebd6f562d4963788e0" Oct 31 13:42:11.676395 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.676373 6841 scope.go:122] "RemoveContainer" containerID="8baeb9c963c3d70611ee309af6bfe62949a32294f521ce2196f83ac19feb28f7" Oct 31 13:42:11.676645 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:42:11.676623 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8baeb9c963c3d70611ee309af6bfe62949a32294f521ce2196f83ac19feb28f7\": container with ID starting with 8baeb9c963c3d70611ee309af6bfe62949a32294f521ce2196f83ac19feb28f7 not found: ID does not exist" containerID="8baeb9c963c3d70611ee309af6bfe62949a32294f521ce2196f83ac19feb28f7" Oct 31 13:42:11.676692 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.676653 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8baeb9c963c3d70611ee309af6bfe62949a32294f521ce2196f83ac19feb28f7"} err="failed to get container status \"8baeb9c963c3d70611ee309af6bfe62949a32294f521ce2196f83ac19feb28f7\": rpc error: code = NotFound desc = could not find container \"8baeb9c963c3d70611ee309af6bfe62949a32294f521ce2196f83ac19feb28f7\": container with ID starting with 8baeb9c963c3d70611ee309af6bfe62949a32294f521ce2196f83ac19feb28f7 not found: ID does not exist" Oct 31 13:42:11.676692 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.676675 6841 scope.go:122] "RemoveContainer" containerID="705f880daff07ec87b8b01f01987326bf511cff6093825ebd6f562d4963788e0" Oct 31 13:42:11.676913 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:42:11.676890 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"705f880daff07ec87b8b01f01987326bf511cff6093825ebd6f562d4963788e0\": container with ID starting with 705f880daff07ec87b8b01f01987326bf511cff6093825ebd6f562d4963788e0 not found: ID does not exist" containerID="705f880daff07ec87b8b01f01987326bf511cff6093825ebd6f562d4963788e0" Oct 31 13:42:11.676963 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.676917 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"705f880daff07ec87b8b01f01987326bf511cff6093825ebd6f562d4963788e0"} err="failed to get container status \"705f880daff07ec87b8b01f01987326bf511cff6093825ebd6f562d4963788e0\": rpc error: code = NotFound desc = could not find container \"705f880daff07ec87b8b01f01987326bf511cff6093825ebd6f562d4963788e0\": container with ID starting with 705f880daff07ec87b8b01f01987326bf511cff6093825ebd6f562d4963788e0 not found: ID does not exist" Oct 31 13:42:11.683773 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.683749 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4"] Oct 31 13:42:11.689374 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:11.689350 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-7n7z4"] Oct 31 13:42:12.344110 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:12.344074 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:12.556232 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:12.551432 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:13.098198 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:13.098160 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:13.625547 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:13.625514 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:13.643747 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:13.643715 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42529d86-63e7-47dd-a03d-3e2ca7737a52" path="/var/lib/kubelet/pods/42529d86-63e7-47dd-a03d-3e2ca7737a52/volumes" Oct 31 13:42:13.825120 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:13.825087 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:14.596050 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:14.596016 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:15.446891 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:15.446855 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:16.143924 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:16.143889 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:16.695266 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:16.695233 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn"] Oct 31 13:42:16.695694 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:16.695438 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn" podUID="d996d76a-0e68-45da-855d-c59a99bb2395" containerName="maas-api" containerID="cri-o://28bff7ddfa58b2a00a60e357e305d9303666f9b8362945e8457e3d5f0e885429" gracePeriod=30 Oct 31 13:42:16.919303 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:16.919269 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:16.972606 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:16.972576 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn" Oct 31 13:42:17.082150 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:17.082112 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d996d76a-0e68-45da-855d-c59a99bb2395-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d996d76a-0e68-45da-855d-c59a99bb2395-maas-api-tls\") pod \"d996d76a-0e68-45da-855d-c59a99bb2395\" (UID: \"d996d76a-0e68-45da-855d-c59a99bb2395\") " Oct 31 13:42:17.082334 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:17.082234 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d996d76a-0e68-45da-855d-c59a99bb2395-kube-api-access-jn4q4\" (UniqueName: \"kubernetes.io/projected/d996d76a-0e68-45da-855d-c59a99bb2395-kube-api-access-jn4q4\") pod \"d996d76a-0e68-45da-855d-c59a99bb2395\" (UID: \"d996d76a-0e68-45da-855d-c59a99bb2395\") " Oct 31 13:42:17.082334 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:17.082263 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d996d76a-0e68-45da-855d-c59a99bb2395-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d996d76a-0e68-45da-855d-c59a99bb2395-openshift-service-ca\") pod \"d996d76a-0e68-45da-855d-c59a99bb2395\" (UID: \"d996d76a-0e68-45da-855d-c59a99bb2395\") " Oct 31 13:42:17.082334 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:17.082288 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d996d76a-0e68-45da-855d-c59a99bb2395-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d996d76a-0e68-45da-855d-c59a99bb2395-metrics-certs\") pod \"d996d76a-0e68-45da-855d-c59a99bb2395\" (UID: \"d996d76a-0e68-45da-855d-c59a99bb2395\") " Oct 31 13:42:17.082672 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:17.082647 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d996d76a-0e68-45da-855d-c59a99bb2395-openshift-service-ca" pod "d996d76a-0e68-45da-855d-c59a99bb2395" (UID: "d996d76a-0e68-45da-855d-c59a99bb2395"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:42:17.082947 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:17.082927 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d996d76a-0e68-45da-855d-c59a99bb2395-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:17.084175 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:17.084154 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d996d76a-0e68-45da-855d-c59a99bb2395-maas-api-tls" pod "d996d76a-0e68-45da-855d-c59a99bb2395" (UID: "d996d76a-0e68-45da-855d-c59a99bb2395"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:42:17.084811 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:17.084789 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d996d76a-0e68-45da-855d-c59a99bb2395-kube-api-access-jn4q4" pod "d996d76a-0e68-45da-855d-c59a99bb2395" (UID: "d996d76a-0e68-45da-855d-c59a99bb2395"). InnerVolumeSpecName "kube-api-access-jn4q4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:42:17.084907 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:17.084810 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d996d76a-0e68-45da-855d-c59a99bb2395-metrics-certs" pod "d996d76a-0e68-45da-855d-c59a99bb2395" (UID: "d996d76a-0e68-45da-855d-c59a99bb2395"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:42:17.185368 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:17.185316 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn4q4\" (UniqueName: \"kubernetes.io/projected/d996d76a-0e68-45da-855d-c59a99bb2395-kube-api-access-jn4q4\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:17.185753 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:17.185728 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d996d76a-0e68-45da-855d-c59a99bb2395-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:17.185872 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:17.185857 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d996d76a-0e68-45da-855d-c59a99bb2395-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:17.392030 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:17.391912 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:17.700204 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:17.700167 6841 generic.go:356] "Generic (PLEG): container finished" podID="d996d76a-0e68-45da-855d-c59a99bb2395" containerID="28bff7ddfa58b2a00a60e357e305d9303666f9b8362945e8457e3d5f0e885429" exitCode=0 Oct 31 13:42:17.700686 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:17.700224 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn" Oct 31 13:42:17.700686 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:17.700257 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn" event={"ID":"d996d76a-0e68-45da-855d-c59a99bb2395","Type":"ContainerDied","Data":"28bff7ddfa58b2a00a60e357e305d9303666f9b8362945e8457e3d5f0e885429"} Oct 31 13:42:17.700686 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:17.700329 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn" event={"ID":"d996d76a-0e68-45da-855d-c59a99bb2395","Type":"ContainerDied","Data":"c15ac616349c17d9b7f21423ddadbd86bbc136049cfd1d13fcb66ada3905a8d3"} Oct 31 13:42:17.700686 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:17.700361 6841 scope.go:122] "RemoveContainer" containerID="28bff7ddfa58b2a00a60e357e305d9303666f9b8362945e8457e3d5f0e885429" Oct 31 13:42:17.711863 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:17.711844 6841 scope.go:122] "RemoveContainer" containerID="28bff7ddfa58b2a00a60e357e305d9303666f9b8362945e8457e3d5f0e885429" Oct 31 13:42:17.712178 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:42:17.712155 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28bff7ddfa58b2a00a60e357e305d9303666f9b8362945e8457e3d5f0e885429\": container with ID starting with 28bff7ddfa58b2a00a60e357e305d9303666f9b8362945e8457e3d5f0e885429 not found: ID does not exist" containerID="28bff7ddfa58b2a00a60e357e305d9303666f9b8362945e8457e3d5f0e885429" Oct 31 13:42:17.712245 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:17.712187 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28bff7ddfa58b2a00a60e357e305d9303666f9b8362945e8457e3d5f0e885429"} err="failed to get container status \"28bff7ddfa58b2a00a60e357e305d9303666f9b8362945e8457e3d5f0e885429\": rpc error: code = NotFound desc = could not find container \"28bff7ddfa58b2a00a60e357e305d9303666f9b8362945e8457e3d5f0e885429\": container with ID starting with 28bff7ddfa58b2a00a60e357e305d9303666f9b8362945e8457e3d5f0e885429 not found: ID does not exist" Oct 31 13:42:17.721159 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:17.721134 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn"] Oct 31 13:42:17.727123 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:17.727101 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6cef79-7bc5ccc957-sn2jn"] Oct 31 13:42:18.217459 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:18.217423 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:18.944886 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:18.944854 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:19.643448 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:19.643411 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d996d76a-0e68-45da-855d-c59a99bb2395" path="/var/lib/kubelet/pods/d996d76a-0e68-45da-855d-c59a99bb2395/volumes" Oct 31 13:42:19.945183 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:19.945149 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:20.036839 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:42:20.036805 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:42:21.242147 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:21.242112 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:22.061621 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:22.061584 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:22.566915 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:22.566871 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:22.810506 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:22.810472 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:23.207638 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:23.207599 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:23.427997 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:23.427950 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:23.478394 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:23.478316 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx"] Oct 31 13:42:23.478566 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:23.478544 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" podUID="32580ef3-c0ea-46bc-8b62-c2b1362e998c" containerName="istio-proxy" containerID="cri-o://858fa07c513333442e79ffe82f049ea19edccc4977872f916e9bf92d6105d76b" gracePeriod=30 Oct 31 13:42:23.640006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:23.639348 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:24.226836 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:24.226803 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:24.445033 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:24.444977 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:24.934289 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:24.934238 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:25.352769 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:25.352737 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng"] Oct 31 13:42:25.506549 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:25.503467 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:25.771459 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:25.771421 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:26.621759 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:26.621729 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:27.001651 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:27.001614 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt"] Oct 31 13:42:27.001885 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:27.001855 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" podUID="954541ff-08a9-4c5e-ab44-a9f6148e0315" containerName="istio-proxy" containerID="cri-o://d7f476fd9acbeb79829c34771a4377a5a1bb38f36f9eb595096f4005328356c0" gracePeriod=30 Oct 31 13:42:27.480484 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:27.480443 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:28.301564 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.301530 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:28.542136 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.542101 6841 patch_prober.go:29] interesting pod/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 31 13:42:28.542283 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.542159 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" podUID="32580ef3-c0ea-46bc-8b62-c2b1362e998c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 31 13:42:28.755895 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.755871 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:42:28.771197 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.771172 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32580ef3-c0ea-46bc-8b62-c2b1362e998c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32580ef3-c0ea-46bc-8b62-c2b1362e998c-istio-data\") pod \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\" (UID: \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\") " Oct 31 13:42:28.771318 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.771218 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32580ef3-c0ea-46bc-8b62-c2b1362e998c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32580ef3-c0ea-46bc-8b62-c2b1362e998c-workload-socket\") pod \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\" (UID: \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\") " Oct 31 13:42:28.771318 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.771277 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32580ef3-c0ea-46bc-8b62-c2b1362e998c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32580ef3-c0ea-46bc-8b62-c2b1362e998c-istio-podinfo\") pod \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\" (UID: \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\") " Oct 31 13:42:28.771318 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.771304 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32580ef3-c0ea-46bc-8b62-c2b1362e998c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32580ef3-c0ea-46bc-8b62-c2b1362e998c-istio-envoy\") pod \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\" (UID: \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\") " Oct 31 13:42:28.771512 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.771334 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32580ef3-c0ea-46bc-8b62-c2b1362e998c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32580ef3-c0ea-46bc-8b62-c2b1362e998c-istiod-ca-cert\") pod \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\" (UID: \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\") " Oct 31 13:42:28.771512 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.771371 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32580ef3-c0ea-46bc-8b62-c2b1362e998c-istio-token\" (UniqueName: \"kubernetes.io/projected/32580ef3-c0ea-46bc-8b62-c2b1362e998c-istio-token\") pod \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\" (UID: \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\") " Oct 31 13:42:28.771512 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.771422 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32580ef3-c0ea-46bc-8b62-c2b1362e998c-kube-api-access-26wxb\" (UniqueName: \"kubernetes.io/projected/32580ef3-c0ea-46bc-8b62-c2b1362e998c-kube-api-access-26wxb\") pod \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\" (UID: \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\") " Oct 31 13:42:28.771512 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.771433 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32580ef3-c0ea-46bc-8b62-c2b1362e998c-istio-data" pod "32580ef3-c0ea-46bc-8b62-c2b1362e998c" (UID: "32580ef3-c0ea-46bc-8b62-c2b1362e998c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:28.771512 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.771473 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32580ef3-c0ea-46bc-8b62-c2b1362e998c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32580ef3-c0ea-46bc-8b62-c2b1362e998c-credential-socket\") pod \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\" (UID: \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\") " Oct 31 13:42:28.771799 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.771557 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32580ef3-c0ea-46bc-8b62-c2b1362e998c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32580ef3-c0ea-46bc-8b62-c2b1362e998c-workload-certs\") pod \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\" (UID: \"32580ef3-c0ea-46bc-8b62-c2b1362e998c\") " Oct 31 13:42:28.772030 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.771555 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32580ef3-c0ea-46bc-8b62-c2b1362e998c-workload-socket" pod "32580ef3-c0ea-46bc-8b62-c2b1362e998c" (UID: "32580ef3-c0ea-46bc-8b62-c2b1362e998c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:28.772030 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.771782 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32580ef3-c0ea-46bc-8b62-c2b1362e998c-credential-socket" pod "32580ef3-c0ea-46bc-8b62-c2b1362e998c" (UID: "32580ef3-c0ea-46bc-8b62-c2b1362e998c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:28.772030 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.771821 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32580ef3-c0ea-46bc-8b62-c2b1362e998c-istiod-ca-cert" pod "32580ef3-c0ea-46bc-8b62-c2b1362e998c" (UID: "32580ef3-c0ea-46bc-8b62-c2b1362e998c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:42:28.772030 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.771968 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32580ef3-c0ea-46bc-8b62-c2b1362e998c-workload-certs" pod "32580ef3-c0ea-46bc-8b62-c2b1362e998c" (UID: "32580ef3-c0ea-46bc-8b62-c2b1362e998c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:28.772313 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.772228 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32580ef3-c0ea-46bc-8b62-c2b1362e998c-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:28.772313 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.772250 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32580ef3-c0ea-46bc-8b62-c2b1362e998c-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:28.772313 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.772264 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32580ef3-c0ea-46bc-8b62-c2b1362e998c-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:28.772313 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.772281 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32580ef3-c0ea-46bc-8b62-c2b1362e998c-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:28.772313 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.772295 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32580ef3-c0ea-46bc-8b62-c2b1362e998c-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:28.774063 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.774035 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32580ef3-c0ea-46bc-8b62-c2b1362e998c-kube-api-access-26wxb" pod "32580ef3-c0ea-46bc-8b62-c2b1362e998c" (UID: "32580ef3-c0ea-46bc-8b62-c2b1362e998c"). InnerVolumeSpecName "kube-api-access-26wxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:42:28.774175 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.774066 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32580ef3-c0ea-46bc-8b62-c2b1362e998c-istio-podinfo" pod "32580ef3-c0ea-46bc-8b62-c2b1362e998c" (UID: "32580ef3-c0ea-46bc-8b62-c2b1362e998c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:42:28.774175 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.774063 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32580ef3-c0ea-46bc-8b62-c2b1362e998c-istio-envoy" pod "32580ef3-c0ea-46bc-8b62-c2b1362e998c" (UID: "32580ef3-c0ea-46bc-8b62-c2b1362e998c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:28.774175 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.774110 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32580ef3-c0ea-46bc-8b62-c2b1362e998c-istio-token" pod "32580ef3-c0ea-46bc-8b62-c2b1362e998c" (UID: "32580ef3-c0ea-46bc-8b62-c2b1362e998c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:42:28.789159 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.789135 6841 generic.go:356] "Generic (PLEG): container finished" podID="32580ef3-c0ea-46bc-8b62-c2b1362e998c" containerID="858fa07c513333442e79ffe82f049ea19edccc4977872f916e9bf92d6105d76b" exitCode=0 Oct 31 13:42:28.789285 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.789249 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" event={"ID":"32580ef3-c0ea-46bc-8b62-c2b1362e998c","Type":"ContainerDied","Data":"858fa07c513333442e79ffe82f049ea19edccc4977872f916e9bf92d6105d76b"} Oct 31 13:42:28.789350 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.789301 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" event={"ID":"32580ef3-c0ea-46bc-8b62-c2b1362e998c","Type":"ContainerDied","Data":"c7c5ccbc499e228d29a5f75f3de236bc0461bc831a3212099218ddbf0f7573c3"} Oct 31 13:42:28.789350 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.789332 6841 scope.go:122] "RemoveContainer" containerID="858fa07c513333442e79ffe82f049ea19edccc4977872f916e9bf92d6105d76b" Oct 31 13:42:28.789514 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.789499 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx" Oct 31 13:42:28.813442 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.813414 6841 scope.go:122] "RemoveContainer" containerID="b3f2b12f6db769d6cda6d6e171cd458e27e99461a775f49ee91314795bf2d136" Oct 31 13:42:28.830781 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.830748 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx"] Oct 31 13:42:28.834023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.832388 6841 scope.go:122] "RemoveContainer" containerID="858fa07c513333442e79ffe82f049ea19edccc4977872f916e9bf92d6105d76b" Oct 31 13:42:28.834023 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:42:28.832753 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"858fa07c513333442e79ffe82f049ea19edccc4977872f916e9bf92d6105d76b\": container with ID starting with 858fa07c513333442e79ffe82f049ea19edccc4977872f916e9bf92d6105d76b not found: ID does not exist" containerID="858fa07c513333442e79ffe82f049ea19edccc4977872f916e9bf92d6105d76b" Oct 31 13:42:28.834023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.832788 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"858fa07c513333442e79ffe82f049ea19edccc4977872f916e9bf92d6105d76b"} err="failed to get container status \"858fa07c513333442e79ffe82f049ea19edccc4977872f916e9bf92d6105d76b\": rpc error: code = NotFound desc = could not find container \"858fa07c513333442e79ffe82f049ea19edccc4977872f916e9bf92d6105d76b\": container with ID starting with 858fa07c513333442e79ffe82f049ea19edccc4977872f916e9bf92d6105d76b not found: ID does not exist" Oct 31 13:42:28.834023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.832814 6841 scope.go:122] "RemoveContainer" containerID="b3f2b12f6db769d6cda6d6e171cd458e27e99461a775f49ee91314795bf2d136" Oct 31 13:42:28.834023 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:42:28.833216 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3f2b12f6db769d6cda6d6e171cd458e27e99461a775f49ee91314795bf2d136\": container with ID starting with b3f2b12f6db769d6cda6d6e171cd458e27e99461a775f49ee91314795bf2d136 not found: ID does not exist" containerID="b3f2b12f6db769d6cda6d6e171cd458e27e99461a775f49ee91314795bf2d136" Oct 31 13:42:28.834023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.833246 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3f2b12f6db769d6cda6d6e171cd458e27e99461a775f49ee91314795bf2d136"} err="failed to get container status \"b3f2b12f6db769d6cda6d6e171cd458e27e99461a775f49ee91314795bf2d136\": rpc error: code = NotFound desc = could not find container \"b3f2b12f6db769d6cda6d6e171cd458e27e99461a775f49ee91314795bf2d136\": container with ID starting with b3f2b12f6db769d6cda6d6e171cd458e27e99461a775f49ee91314795bf2d136 not found: ID does not exist" Oct 31 13:42:28.836723 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.836698 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-6cef79-openshift-default-686ccff857-fn6kx"] Oct 31 13:42:28.855746 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:42:28.855689 6841 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32580ef3_c0ea_46bc_8b62_c2b1362e998c.slice/crio-c7c5ccbc499e228d29a5f75f3de236bc0461bc831a3212099218ddbf0f7573c3\": RecentStats: unable to find data in memory cache]" Oct 31 13:42:28.873659 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.873631 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32580ef3-c0ea-46bc-8b62-c2b1362e998c-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:28.873765 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.873661 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32580ef3-c0ea-46bc-8b62-c2b1362e998c-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:28.873765 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.873677 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32580ef3-c0ea-46bc-8b62-c2b1362e998c-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:28.873765 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:28.873691 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26wxb\" (UniqueName: \"kubernetes.io/projected/32580ef3-c0ea-46bc-8b62-c2b1362e998c-kube-api-access-26wxb\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:29.026538 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:29.026500 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:29.645125 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:29.645088 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32580ef3-c0ea-46bc-8b62-c2b1362e998c" path="/var/lib/kubelet/pods/32580ef3-c0ea-46bc-8b62-c2b1362e998c/volumes" Oct 31 13:42:29.820952 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:29.820918 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:30.250057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:30.249386 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:30.569311 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:30.569219 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:30.778259 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:30.778218 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:30.988357 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:30.988322 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:32.016068 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.016030 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:32.234189 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.234132 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:32.281281 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.281209 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" Oct 31 13:42:32.340378 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.340347 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/954541ff-08a9-4c5e-ab44-a9f6148e0315-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/954541ff-08a9-4c5e-ab44-a9f6148e0315-istiod-ca-cert\") pod \"954541ff-08a9-4c5e-ab44-a9f6148e0315\" (UID: \"954541ff-08a9-4c5e-ab44-a9f6148e0315\") " Oct 31 13:42:32.340571 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.340410 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/954541ff-08a9-4c5e-ab44-a9f6148e0315-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/954541ff-08a9-4c5e-ab44-a9f6148e0315-istio-envoy\") pod \"954541ff-08a9-4c5e-ab44-a9f6148e0315\" (UID: \"954541ff-08a9-4c5e-ab44-a9f6148e0315\") " Oct 31 13:42:32.340571 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.340500 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/954541ff-08a9-4c5e-ab44-a9f6148e0315-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/954541ff-08a9-4c5e-ab44-a9f6148e0315-istio-podinfo\") pod \"954541ff-08a9-4c5e-ab44-a9f6148e0315\" (UID: \"954541ff-08a9-4c5e-ab44-a9f6148e0315\") " Oct 31 13:42:32.340571 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.340533 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/954541ff-08a9-4c5e-ab44-a9f6148e0315-istio-token\" (UniqueName: \"kubernetes.io/projected/954541ff-08a9-4c5e-ab44-a9f6148e0315-istio-token\") pod \"954541ff-08a9-4c5e-ab44-a9f6148e0315\" (UID: \"954541ff-08a9-4c5e-ab44-a9f6148e0315\") " Oct 31 13:42:32.340770 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.340583 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/954541ff-08a9-4c5e-ab44-a9f6148e0315-kube-api-access-hm6js\" (UniqueName: \"kubernetes.io/projected/954541ff-08a9-4c5e-ab44-a9f6148e0315-kube-api-access-hm6js\") pod \"954541ff-08a9-4c5e-ab44-a9f6148e0315\" (UID: \"954541ff-08a9-4c5e-ab44-a9f6148e0315\") " Oct 31 13:42:32.341195 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.341146 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/954541ff-08a9-4c5e-ab44-a9f6148e0315-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/954541ff-08a9-4c5e-ab44-a9f6148e0315-workload-socket\") pod \"954541ff-08a9-4c5e-ab44-a9f6148e0315\" (UID: \"954541ff-08a9-4c5e-ab44-a9f6148e0315\") " Oct 31 13:42:32.341336 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.341281 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/954541ff-08a9-4c5e-ab44-a9f6148e0315-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/954541ff-08a9-4c5e-ab44-a9f6148e0315-credential-socket\") pod \"954541ff-08a9-4c5e-ab44-a9f6148e0315\" (UID: \"954541ff-08a9-4c5e-ab44-a9f6148e0315\") " Oct 31 13:42:32.341418 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.341381 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/954541ff-08a9-4c5e-ab44-a9f6148e0315-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/954541ff-08a9-4c5e-ab44-a9f6148e0315-istio-data\") pod \"954541ff-08a9-4c5e-ab44-a9f6148e0315\" (UID: \"954541ff-08a9-4c5e-ab44-a9f6148e0315\") " Oct 31 13:42:32.341478 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.341419 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/954541ff-08a9-4c5e-ab44-a9f6148e0315-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/954541ff-08a9-4c5e-ab44-a9f6148e0315-workload-certs\") pod \"954541ff-08a9-4c5e-ab44-a9f6148e0315\" (UID: \"954541ff-08a9-4c5e-ab44-a9f6148e0315\") " Oct 31 13:42:32.341599 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.341573 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/954541ff-08a9-4c5e-ab44-a9f6148e0315-istiod-ca-cert" pod "954541ff-08a9-4c5e-ab44-a9f6148e0315" (UID: "954541ff-08a9-4c5e-ab44-a9f6148e0315"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:42:32.341880 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.341851 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954541ff-08a9-4c5e-ab44-a9f6148e0315-workload-socket" pod "954541ff-08a9-4c5e-ab44-a9f6148e0315" (UID: "954541ff-08a9-4c5e-ab44-a9f6148e0315"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:32.341971 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.341932 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954541ff-08a9-4c5e-ab44-a9f6148e0315-workload-certs" pod "954541ff-08a9-4c5e-ab44-a9f6148e0315" (UID: "954541ff-08a9-4c5e-ab44-a9f6148e0315"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:32.342139 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.342109 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954541ff-08a9-4c5e-ab44-a9f6148e0315-credential-socket" pod "954541ff-08a9-4c5e-ab44-a9f6148e0315" (UID: "954541ff-08a9-4c5e-ab44-a9f6148e0315"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:32.342139 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.342128 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954541ff-08a9-4c5e-ab44-a9f6148e0315-istio-data" pod "954541ff-08a9-4c5e-ab44-a9f6148e0315" (UID: "954541ff-08a9-4c5e-ab44-a9f6148e0315"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:32.342308 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.342268 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/954541ff-08a9-4c5e-ab44-a9f6148e0315-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:32.342308 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.342289 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/954541ff-08a9-4c5e-ab44-a9f6148e0315-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:32.342308 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.342305 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/954541ff-08a9-4c5e-ab44-a9f6148e0315-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:32.342470 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.342320 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/954541ff-08a9-4c5e-ab44-a9f6148e0315-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:32.342470 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.342334 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/954541ff-08a9-4c5e-ab44-a9f6148e0315-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:32.343892 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.343865 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/954541ff-08a9-4c5e-ab44-a9f6148e0315-kube-api-access-hm6js" pod "954541ff-08a9-4c5e-ab44-a9f6148e0315" (UID: "954541ff-08a9-4c5e-ab44-a9f6148e0315"). InnerVolumeSpecName "kube-api-access-hm6js". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:42:32.344052 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.343964 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/954541ff-08a9-4c5e-ab44-a9f6148e0315-istio-podinfo" pod "954541ff-08a9-4c5e-ab44-a9f6148e0315" (UID: "954541ff-08a9-4c5e-ab44-a9f6148e0315"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:42:32.345209 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.345183 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954541ff-08a9-4c5e-ab44-a9f6148e0315-istio-envoy" pod "954541ff-08a9-4c5e-ab44-a9f6148e0315" (UID: "954541ff-08a9-4c5e-ab44-a9f6148e0315"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:32.351150 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.351123 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/954541ff-08a9-4c5e-ab44-a9f6148e0315-istio-token" pod "954541ff-08a9-4c5e-ab44-a9f6148e0315" (UID: "954541ff-08a9-4c5e-ab44-a9f6148e0315"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:42:32.444141 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.444100 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/954541ff-08a9-4c5e-ab44-a9f6148e0315-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:32.444141 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.444133 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/954541ff-08a9-4c5e-ab44-a9f6148e0315-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:32.444141 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.444143 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/954541ff-08a9-4c5e-ab44-a9f6148e0315-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:32.444395 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.444152 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm6js\" (UniqueName: \"kubernetes.io/projected/954541ff-08a9-4c5e-ab44-a9f6148e0315-kube-api-access-hm6js\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:32.834450 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.834414 6841 generic.go:356] "Generic (PLEG): container finished" podID="954541ff-08a9-4c5e-ab44-a9f6148e0315" containerID="d7f476fd9acbeb79829c34771a4377a5a1bb38f36f9eb595096f4005328356c0" exitCode=0 Oct 31 13:42:32.834630 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.834458 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" Oct 31 13:42:32.834630 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.834496 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" event={"ID":"954541ff-08a9-4c5e-ab44-a9f6148e0315","Type":"ContainerDied","Data":"d7f476fd9acbeb79829c34771a4377a5a1bb38f36f9eb595096f4005328356c0"} Oct 31 13:42:32.834630 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.834544 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt" event={"ID":"954541ff-08a9-4c5e-ab44-a9f6148e0315","Type":"ContainerDied","Data":"439031760aa5fc56aae4ebe09d90bb242ad54ac6fdd5f179211289538164e6e3"} Oct 31 13:42:32.834630 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.834565 6841 scope.go:122] "RemoveContainer" containerID="d7f476fd9acbeb79829c34771a4377a5a1bb38f36f9eb595096f4005328356c0" Oct 31 13:42:32.846322 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.846302 6841 scope.go:122] "RemoveContainer" containerID="d7f476fd9acbeb79829c34771a4377a5a1bb38f36f9eb595096f4005328356c0" Oct 31 13:42:32.846629 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:42:32.846607 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7f476fd9acbeb79829c34771a4377a5a1bb38f36f9eb595096f4005328356c0\": container with ID starting with d7f476fd9acbeb79829c34771a4377a5a1bb38f36f9eb595096f4005328356c0 not found: ID does not exist" containerID="d7f476fd9acbeb79829c34771a4377a5a1bb38f36f9eb595096f4005328356c0" Oct 31 13:42:32.846686 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.846638 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7f476fd9acbeb79829c34771a4377a5a1bb38f36f9eb595096f4005328356c0"} err="failed to get container status \"d7f476fd9acbeb79829c34771a4377a5a1bb38f36f9eb595096f4005328356c0\": rpc error: code = NotFound desc = could not find container \"d7f476fd9acbeb79829c34771a4377a5a1bb38f36f9eb595096f4005328356c0\": container with ID starting with d7f476fd9acbeb79829c34771a4377a5a1bb38f36f9eb595096f4005328356c0 not found: ID does not exist" Oct 31 13:42:32.903573 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.903533 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt"] Oct 31 13:42:32.919951 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.919911 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-v5rgt"] Oct 31 13:42:32.956635 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:32.956601 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:33.643155 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:33.643126 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="954541ff-08a9-4c5e-ab44-a9f6148e0315" path="/var/lib/kubelet/pods/954541ff-08a9-4c5e-ab44-a9f6148e0315/volumes" Oct 31 13:42:33.944047 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:33.944010 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:34.935533 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:34.935503 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:35.906351 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:35.906318 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:36.767115 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:36.767064 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:37.959924 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:37.959885 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:38.837487 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:38.837457 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:39.151353 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.151255 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:39.743438 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.743405 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn"] Oct 31 13:42:39.744392 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.744364 6841 cpu_manager.go:414] "containerMap: removing container" podUID="954541ff-08a9-4c5e-ab44-a9f6148e0315" containerName="istio-proxy" Oct 31 13:42:39.744392 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.744393 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="954541ff-08a9-4c5e-ab44-a9f6148e0315" containerName="istio-proxy" Oct 31 13:42:39.744612 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.744424 6841 cpu_manager.go:414] "containerMap: removing container" podUID="32580ef3-c0ea-46bc-8b62-c2b1362e998c" containerName="istio-proxy" Oct 31 13:42:39.744612 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.744433 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32580ef3-c0ea-46bc-8b62-c2b1362e998c" containerName="istio-proxy" Oct 31 13:42:39.744612 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.744454 6841 cpu_manager.go:414] "containerMap: removing container" podUID="25d7ee65-7958-458c-a202-693bd09893c7" containerName="istio-proxy" Oct 31 13:42:39.744612 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.744462 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25d7ee65-7958-458c-a202-693bd09893c7" containerName="istio-proxy" Oct 31 13:42:39.744612 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.744487 6841 cpu_manager.go:414] "containerMap: removing container" podUID="32580ef3-c0ea-46bc-8b62-c2b1362e998c" containerName="istio-proxy" Oct 31 13:42:39.744612 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.744494 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32580ef3-c0ea-46bc-8b62-c2b1362e998c" containerName="istio-proxy" Oct 31 13:42:39.744612 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.744515 6841 cpu_manager.go:414] "containerMap: removing container" podUID="d996d76a-0e68-45da-855d-c59a99bb2395" containerName="maas-api" Oct 31 13:42:39.744612 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.744522 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d996d76a-0e68-45da-855d-c59a99bb2395" containerName="maas-api" Oct 31 13:42:39.744612 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.744543 6841 cpu_manager.go:414] "containerMap: removing container" podUID="42529d86-63e7-47dd-a03d-3e2ca7737a52" containerName="istio-proxy" Oct 31 13:42:39.744612 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.744550 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42529d86-63e7-47dd-a03d-3e2ca7737a52" containerName="istio-proxy" Oct 31 13:42:39.744612 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.744565 6841 cpu_manager.go:414] "containerMap: removing container" podUID="99509d76-474b-4b4b-b354-19cf3084ce0d" containerName="test-curl-11391-77d074" Oct 31 13:42:39.744612 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.744573 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99509d76-474b-4b4b-b354-19cf3084ce0d" containerName="test-curl-11391-77d074" Oct 31 13:42:39.744612 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.744601 6841 cpu_manager.go:414] "containerMap: removing container" podUID="42529d86-63e7-47dd-a03d-3e2ca7737a52" containerName="istio-proxy" Oct 31 13:42:39.744612 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.744608 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42529d86-63e7-47dd-a03d-3e2ca7737a52" containerName="istio-proxy" Oct 31 13:42:39.745209 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.744756 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="42529d86-63e7-47dd-a03d-3e2ca7737a52" containerName="istio-proxy" Oct 31 13:42:39.745209 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.744770 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="99509d76-474b-4b4b-b354-19cf3084ce0d" containerName="test-curl-11391-77d074" Oct 31 13:42:39.745209 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.744784 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="42529d86-63e7-47dd-a03d-3e2ca7737a52" containerName="istio-proxy" Oct 31 13:42:39.745209 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.744796 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="954541ff-08a9-4c5e-ab44-a9f6148e0315" containerName="istio-proxy" Oct 31 13:42:39.745209 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.744810 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="32580ef3-c0ea-46bc-8b62-c2b1362e998c" containerName="istio-proxy" Oct 31 13:42:39.745209 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.744826 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="32580ef3-c0ea-46bc-8b62-c2b1362e998c" containerName="istio-proxy" Oct 31 13:42:39.745209 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.744835 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="d996d76a-0e68-45da-855d-c59a99bb2395" containerName="maas-api" Oct 31 13:42:39.745209 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.744847 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="25d7ee65-7958-458c-a202-693bd09893c7" containerName="istio-proxy" Oct 31 13:42:39.751037 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.751020 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn" Oct 31 13:42:39.757567 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.757541 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn"] Oct 31 13:42:39.758956 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.758931 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:39.759560 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.759543 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-6c0612\"/\"sub-test-model-6c0612-kserve-self-signed-certs\"" Oct 31 13:42:39.861413 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.861381 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dc69d96-bfd1-4ec9-8414-75553fde6d87-model-cache\") pod \"sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn\" (UID: \"1dc69d96-bfd1-4ec9-8414-75553fde6d87\") " pod="ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn" Oct 31 13:42:39.861579 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.861423 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dc69d96-bfd1-4ec9-8414-75553fde6d87-home\") pod \"sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn\" (UID: \"1dc69d96-bfd1-4ec9-8414-75553fde6d87\") " pod="ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn" Oct 31 13:42:39.861579 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.861447 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dc69d96-bfd1-4ec9-8414-75553fde6d87-tmp-dir\") pod \"sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn\" (UID: \"1dc69d96-bfd1-4ec9-8414-75553fde6d87\") " pod="ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn" Oct 31 13:42:39.861579 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.861567 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dc69d96-bfd1-4ec9-8414-75553fde6d87-dshm\") pod \"sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn\" (UID: \"1dc69d96-bfd1-4ec9-8414-75553fde6d87\") " pod="ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn" Oct 31 13:42:39.861693 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.861591 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9qcv\" (UniqueName: \"kubernetes.io/projected/1dc69d96-bfd1-4ec9-8414-75553fde6d87-kube-api-access-g9qcv\") pod \"sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn\" (UID: \"1dc69d96-bfd1-4ec9-8414-75553fde6d87\") " pod="ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn" Oct 31 13:42:39.861693 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.861635 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dc69d96-bfd1-4ec9-8414-75553fde6d87-tls-certs\") pod \"sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn\" (UID: \"1dc69d96-bfd1-4ec9-8414-75553fde6d87\") " pod="ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn" Oct 31 13:42:39.963005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.962849 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dc69d96-bfd1-4ec9-8414-75553fde6d87-model-cache\") pod \"sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn\" (UID: \"1dc69d96-bfd1-4ec9-8414-75553fde6d87\") " pod="ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn" Oct 31 13:42:39.963005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.962921 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dc69d96-bfd1-4ec9-8414-75553fde6d87-home\") pod \"sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn\" (UID: \"1dc69d96-bfd1-4ec9-8414-75553fde6d87\") " pod="ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn" Oct 31 13:42:39.963005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.962949 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dc69d96-bfd1-4ec9-8414-75553fde6d87-tmp-dir\") pod \"sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn\" (UID: \"1dc69d96-bfd1-4ec9-8414-75553fde6d87\") " pod="ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn" Oct 31 13:42:39.963327 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.963011 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dc69d96-bfd1-4ec9-8414-75553fde6d87-dshm\") pod \"sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn\" (UID: \"1dc69d96-bfd1-4ec9-8414-75553fde6d87\") " pod="ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn" Oct 31 13:42:39.963327 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.963062 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9qcv\" (UniqueName: \"kubernetes.io/projected/1dc69d96-bfd1-4ec9-8414-75553fde6d87-kube-api-access-g9qcv\") pod \"sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn\" (UID: \"1dc69d96-bfd1-4ec9-8414-75553fde6d87\") " pod="ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn" Oct 31 13:42:39.963327 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.963135 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dc69d96-bfd1-4ec9-8414-75553fde6d87-tls-certs\") pod \"sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn\" (UID: \"1dc69d96-bfd1-4ec9-8414-75553fde6d87\") " pod="ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn" Oct 31 13:42:39.978250 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.964216 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dc69d96-bfd1-4ec9-8414-75553fde6d87-tmp-dir\") pod \"sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn\" (UID: \"1dc69d96-bfd1-4ec9-8414-75553fde6d87\") " pod="ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn" Oct 31 13:42:39.978250 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.964448 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dc69d96-bfd1-4ec9-8414-75553fde6d87-model-cache\") pod \"sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn\" (UID: \"1dc69d96-bfd1-4ec9-8414-75553fde6d87\") " pod="ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn" Oct 31 13:42:39.978250 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:39.964735 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dc69d96-bfd1-4ec9-8414-75553fde6d87-home\") pod \"sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn\" (UID: \"1dc69d96-bfd1-4ec9-8414-75553fde6d87\") " pod="ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn" Oct 31 13:42:40.001892 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.000501 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dc69d96-bfd1-4ec9-8414-75553fde6d87-tls-certs\") pod \"sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn\" (UID: \"1dc69d96-bfd1-4ec9-8414-75553fde6d87\") " pod="ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn" Oct 31 13:42:40.007095 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.003154 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dc69d96-bfd1-4ec9-8414-75553fde6d87-dshm\") pod \"sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn\" (UID: \"1dc69d96-bfd1-4ec9-8414-75553fde6d87\") " pod="ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn" Oct 31 13:42:40.009224 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.009190 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9qcv\" (UniqueName: \"kubernetes.io/projected/1dc69d96-bfd1-4ec9-8414-75553fde6d87-kube-api-access-g9qcv\") pod \"sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn\" (UID: \"1dc69d96-bfd1-4ec9-8414-75553fde6d87\") " pod="ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn" Oct 31 13:42:40.069510 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.069478 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn" Oct 31 13:42:40.298616 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.298528 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:40.354619 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.354594 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn"] Oct 31 13:42:40.355850 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:42:40.355824 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1dc69d96_bfd1_4ec9_8414_75553fde6d87.slice/crio-6401f6ffe868da5a402f87ab87860636bf3ee64620a54f18714823bfc447ccda WatchSource:0}: Error finding container 6401f6ffe868da5a402f87ab87860636bf3ee64620a54f18714823bfc447ccda: Status 404 returned error can't find the container with id 6401f6ffe868da5a402f87ab87860636bf3ee64620a54f18714823bfc447ccda Oct 31 13:42:40.375472 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.375441 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng" podUID="ff57031e-a357-4764-8c12-2463afb93ac7" containerName="main" containerID="cri-o://cf5789507845241b9a6da678bd863fb88b2779e679494fdb6bbeb09d7d5dc461" gracePeriod=45 Oct 31 13:42:40.695759 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.695723 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng" Oct 31 13:42:40.729287 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.729253 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:40.760326 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.760292 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g"] Oct 31 13:42:40.779040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.776586 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff57031e-a357-4764-8c12-2463afb93ac7-kube-api-access-z5vsj\" (UniqueName: \"kubernetes.io/projected/ff57031e-a357-4764-8c12-2463afb93ac7-kube-api-access-z5vsj\") pod \"ff57031e-a357-4764-8c12-2463afb93ac7\" (UID: \"ff57031e-a357-4764-8c12-2463afb93ac7\") " Oct 31 13:42:40.779040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.776688 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff57031e-a357-4764-8c12-2463afb93ac7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff57031e-a357-4764-8c12-2463afb93ac7-dshm\") pod \"ff57031e-a357-4764-8c12-2463afb93ac7\" (UID: \"ff57031e-a357-4764-8c12-2463afb93ac7\") " Oct 31 13:42:40.779040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.776932 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff57031e-a357-4764-8c12-2463afb93ac7-tls-certs\" (UniqueName: \"kubernetes.io/secret/ff57031e-a357-4764-8c12-2463afb93ac7-tls-certs\") pod \"ff57031e-a357-4764-8c12-2463afb93ac7\" (UID: \"ff57031e-a357-4764-8c12-2463afb93ac7\") " Oct 31 13:42:40.779040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.777018 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff57031e-a357-4764-8c12-2463afb93ac7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff57031e-a357-4764-8c12-2463afb93ac7-model-cache\") pod \"ff57031e-a357-4764-8c12-2463afb93ac7\" (UID: \"ff57031e-a357-4764-8c12-2463afb93ac7\") " Oct 31 13:42:40.779040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.777138 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff57031e-a357-4764-8c12-2463afb93ac7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff57031e-a357-4764-8c12-2463afb93ac7-tmp-dir\") pod \"ff57031e-a357-4764-8c12-2463afb93ac7\" (UID: \"ff57031e-a357-4764-8c12-2463afb93ac7\") " Oct 31 13:42:40.779040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.777241 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff57031e-a357-4764-8c12-2463afb93ac7-home\" (UniqueName: \"kubernetes.io/empty-dir/ff57031e-a357-4764-8c12-2463afb93ac7-home\") pod \"ff57031e-a357-4764-8c12-2463afb93ac7\" (UID: \"ff57031e-a357-4764-8c12-2463afb93ac7\") " Oct 31 13:42:40.779040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.777818 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff57031e-a357-4764-8c12-2463afb93ac7-home" pod "ff57031e-a357-4764-8c12-2463afb93ac7" (UID: "ff57031e-a357-4764-8c12-2463afb93ac7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:40.779040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.777966 6841 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff57031e-a357-4764-8c12-2463afb93ac7-home\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:40.779617 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.779197 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff57031e-a357-4764-8c12-2463afb93ac7-model-cache" pod "ff57031e-a357-4764-8c12-2463afb93ac7" (UID: "ff57031e-a357-4764-8c12-2463afb93ac7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:40.779617 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.779452 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff57031e-a357-4764-8c12-2463afb93ac7-tmp-dir" pod "ff57031e-a357-4764-8c12-2463afb93ac7" (UID: "ff57031e-a357-4764-8c12-2463afb93ac7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:40.781842 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.781815 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff57031e-a357-4764-8c12-2463afb93ac7-tls-certs" pod "ff57031e-a357-4764-8c12-2463afb93ac7" (UID: "ff57031e-a357-4764-8c12-2463afb93ac7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:42:40.782000 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.781910 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff57031e-a357-4764-8c12-2463afb93ac7-kube-api-access-z5vsj" pod "ff57031e-a357-4764-8c12-2463afb93ac7" (UID: "ff57031e-a357-4764-8c12-2463afb93ac7"). InnerVolumeSpecName "kube-api-access-z5vsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:42:40.782161 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.782128 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff57031e-a357-4764-8c12-2463afb93ac7-dshm" pod "ff57031e-a357-4764-8c12-2463afb93ac7" (UID: "ff57031e-a357-4764-8c12-2463afb93ac7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:40.896070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.895139 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5vsj\" (UniqueName: \"kubernetes.io/projected/ff57031e-a357-4764-8c12-2463afb93ac7-kube-api-access-z5vsj\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:40.896070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.895179 6841 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff57031e-a357-4764-8c12-2463afb93ac7-dshm\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:40.896070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.895196 6841 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff57031e-a357-4764-8c12-2463afb93ac7-tls-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:40.896070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.895209 6841 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff57031e-a357-4764-8c12-2463afb93ac7-model-cache\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:40.896070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.895223 6841 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff57031e-a357-4764-8c12-2463afb93ac7-tmp-dir\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:40.929437 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.929393 6841 generic.go:356] "Generic (PLEG): container finished" podID="ff57031e-a357-4764-8c12-2463afb93ac7" containerID="cf5789507845241b9a6da678bd863fb88b2779e679494fdb6bbeb09d7d5dc461" exitCode=0 Oct 31 13:42:40.929671 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.929650 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng" event={"ID":"ff57031e-a357-4764-8c12-2463afb93ac7","Type":"ContainerDied","Data":"cf5789507845241b9a6da678bd863fb88b2779e679494fdb6bbeb09d7d5dc461"} Oct 31 13:42:40.929782 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.929693 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng" event={"ID":"ff57031e-a357-4764-8c12-2463afb93ac7","Type":"ContainerDied","Data":"118b03d22ec290f51f980b5ef852b7cdc8fa1b758e5b6463087da3fda6caf7ff"} Oct 31 13:42:40.929782 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.929716 6841 scope.go:122] "RemoveContainer" containerID="cf5789507845241b9a6da678bd863fb88b2779e679494fdb6bbeb09d7d5dc461" Oct 31 13:42:40.929900 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.929858 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng" Oct 31 13:42:40.933144 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.933102 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn" event={"ID":"1dc69d96-bfd1-4ec9-8414-75553fde6d87","Type":"ContainerStarted","Data":"b28d1b9a1243058fe374bbd228d42beba74afb4eea3ed115779ef0e227601da3"} Oct 31 13:42:40.933274 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.933155 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn" event={"ID":"1dc69d96-bfd1-4ec9-8414-75553fde6d87","Type":"ContainerStarted","Data":"6401f6ffe868da5a402f87ab87860636bf3ee64620a54f18714823bfc447ccda"} Oct 31 13:42:40.944027 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.943995 6841 scope.go:122] "RemoveContainer" containerID="cf5789507845241b9a6da678bd863fb88b2779e679494fdb6bbeb09d7d5dc461" Oct 31 13:42:40.944313 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:42:40.944294 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf5789507845241b9a6da678bd863fb88b2779e679494fdb6bbeb09d7d5dc461\": container with ID starting with cf5789507845241b9a6da678bd863fb88b2779e679494fdb6bbeb09d7d5dc461 not found: ID does not exist" containerID="cf5789507845241b9a6da678bd863fb88b2779e679494fdb6bbeb09d7d5dc461" Oct 31 13:42:40.944381 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.944322 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf5789507845241b9a6da678bd863fb88b2779e679494fdb6bbeb09d7d5dc461"} err="failed to get container status \"cf5789507845241b9a6da678bd863fb88b2779e679494fdb6bbeb09d7d5dc461\": rpc error: code = NotFound desc = could not find container \"cf5789507845241b9a6da678bd863fb88b2779e679494fdb6bbeb09d7d5dc461\": container with ID starting with cf5789507845241b9a6da678bd863fb88b2779e679494fdb6bbeb09d7d5dc461 not found: ID does not exist" Oct 31 13:42:40.966433 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.966392 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn" podStartSLOduration=1.9663780979999999 podStartE2EDuration="1.966378098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:42:40.964556277 +0000 UTC m=+4207.447679180" watchObservedRunningTime="2026-10-31 13:42:40.966378098 +0000 UTC m=+4207.449500998" Oct 31 13:42:40.982853 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.982823 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng"] Oct 31 13:42:40.986417 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.986394 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:40.988695 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:40.988675 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-6c0612/rate-test-model-6c0612-kserve-7975cc4dd5-dg4ng"] Oct 31 13:42:41.070100 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:41.070067 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn" Oct 31 13:42:41.607973 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:41.607938 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:41.642938 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:41.642910 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff57031e-a357-4764-8c12-2463afb93ac7" path="/var/lib/kubelet/pods/ff57031e-a357-4764-8c12-2463afb93ac7/volumes" Oct 31 13:42:42.356735 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:42.356699 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:43.245243 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:43.245201 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:44.137435 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:44.137401 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:44.913403 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:44.913367 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:45.587219 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:45.587187 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf"] Oct 31 13:42:45.587403 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:45.587380 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf" podUID="202fa689-242a-4b4d-8ed6-d3992b0883ff" containerName="maas-api" containerID="cri-o://f88dfae730a90c1d0d269b46ae0631a2adac34b6578bdce63abaad2d3c9239e9" gracePeriod=30 Oct 31 13:42:45.876926 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:45.876760 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:45.924967 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:45.924941 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf" Oct 31 13:42:45.975281 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:45.974871 6841 generic.go:356] "Generic (PLEG): container finished" podID="202fa689-242a-4b4d-8ed6-d3992b0883ff" containerID="f88dfae730a90c1d0d269b46ae0631a2adac34b6578bdce63abaad2d3c9239e9" exitCode=0 Oct 31 13:42:45.975281 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:45.974933 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf" Oct 31 13:42:45.975281 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:45.974961 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf" event={"ID":"202fa689-242a-4b4d-8ed6-d3992b0883ff","Type":"ContainerDied","Data":"f88dfae730a90c1d0d269b46ae0631a2adac34b6578bdce63abaad2d3c9239e9"} Oct 31 13:42:45.975281 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:45.975068 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf" event={"ID":"202fa689-242a-4b4d-8ed6-d3992b0883ff","Type":"ContainerDied","Data":"e58ab4c6434d0ab3cbddb5b537a3b5e04c9f2c19f5a5b57297f6eba59244bc3b"} Oct 31 13:42:45.975281 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:45.975094 6841 scope.go:122] "RemoveContainer" containerID="f88dfae730a90c1d0d269b46ae0631a2adac34b6578bdce63abaad2d3c9239e9" Oct 31 13:42:45.978856 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:45.978452 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/202fa689-242a-4b4d-8ed6-d3992b0883ff-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/202fa689-242a-4b4d-8ed6-d3992b0883ff-maas-api-tls\") pod \"202fa689-242a-4b4d-8ed6-d3992b0883ff\" (UID: \"202fa689-242a-4b4d-8ed6-d3992b0883ff\") " Oct 31 13:42:45.978856 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:45.978510 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/202fa689-242a-4b4d-8ed6-d3992b0883ff-kube-api-access-8l8dw\" (UniqueName: \"kubernetes.io/projected/202fa689-242a-4b4d-8ed6-d3992b0883ff-kube-api-access-8l8dw\") pod \"202fa689-242a-4b4d-8ed6-d3992b0883ff\" (UID: \"202fa689-242a-4b4d-8ed6-d3992b0883ff\") " Oct 31 13:42:45.978856 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:45.978647 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/202fa689-242a-4b4d-8ed6-d3992b0883ff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/202fa689-242a-4b4d-8ed6-d3992b0883ff-metrics-certs\") pod \"202fa689-242a-4b4d-8ed6-d3992b0883ff\" (UID: \"202fa689-242a-4b4d-8ed6-d3992b0883ff\") " Oct 31 13:42:45.978856 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:45.978695 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/202fa689-242a-4b4d-8ed6-d3992b0883ff-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/202fa689-242a-4b4d-8ed6-d3992b0883ff-openshift-service-ca\") pod \"202fa689-242a-4b4d-8ed6-d3992b0883ff\" (UID: \"202fa689-242a-4b4d-8ed6-d3992b0883ff\") " Oct 31 13:42:45.979839 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:45.979802 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/202fa689-242a-4b4d-8ed6-d3992b0883ff-openshift-service-ca" pod "202fa689-242a-4b4d-8ed6-d3992b0883ff" (UID: "202fa689-242a-4b4d-8ed6-d3992b0883ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:42:45.995042 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:45.982641 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/202fa689-242a-4b4d-8ed6-d3992b0883ff-maas-api-tls" pod "202fa689-242a-4b4d-8ed6-d3992b0883ff" (UID: "202fa689-242a-4b4d-8ed6-d3992b0883ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:42:46.024291 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:45.996232 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/202fa689-242a-4b4d-8ed6-d3992b0883ff-kube-api-access-8l8dw" pod "202fa689-242a-4b4d-8ed6-d3992b0883ff" (UID: "202fa689-242a-4b4d-8ed6-d3992b0883ff"). InnerVolumeSpecName "kube-api-access-8l8dw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:42:46.053016 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:46.049139 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/202fa689-242a-4b4d-8ed6-d3992b0883ff-metrics-certs" pod "202fa689-242a-4b4d-8ed6-d3992b0883ff" (UID: "202fa689-242a-4b4d-8ed6-d3992b0883ff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:42:46.053016 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:46.049311 6841 scope.go:122] "RemoveContainer" containerID="f88dfae730a90c1d0d269b46ae0631a2adac34b6578bdce63abaad2d3c9239e9" Oct 31 13:42:46.053016 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:42:46.051419 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f88dfae730a90c1d0d269b46ae0631a2adac34b6578bdce63abaad2d3c9239e9\": container with ID starting with f88dfae730a90c1d0d269b46ae0631a2adac34b6578bdce63abaad2d3c9239e9 not found: ID does not exist" containerID="f88dfae730a90c1d0d269b46ae0631a2adac34b6578bdce63abaad2d3c9239e9" Oct 31 13:42:46.053016 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:46.051458 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f88dfae730a90c1d0d269b46ae0631a2adac34b6578bdce63abaad2d3c9239e9"} err="failed to get container status \"f88dfae730a90c1d0d269b46ae0631a2adac34b6578bdce63abaad2d3c9239e9\": rpc error: code = NotFound desc = could not find container \"f88dfae730a90c1d0d269b46ae0631a2adac34b6578bdce63abaad2d3c9239e9\": container with ID starting with f88dfae730a90c1d0d269b46ae0631a2adac34b6578bdce63abaad2d3c9239e9 not found: ID does not exist" Oct 31 13:42:46.082627 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:46.081328 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/202fa689-242a-4b4d-8ed6-d3992b0883ff-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:46.082627 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:46.081363 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l8dw\" (UniqueName: \"kubernetes.io/projected/202fa689-242a-4b4d-8ed6-d3992b0883ff-kube-api-access-8l8dw\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:46.082627 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:46.081375 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/202fa689-242a-4b4d-8ed6-d3992b0883ff-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:46.082627 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:46.081385 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/202fa689-242a-4b4d-8ed6-d3992b0883ff-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:46.301995 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:46.301959 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh"] Oct 31 13:42:46.332738 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:46.332707 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf"] Oct 31 13:42:46.352230 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:46.352201 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c24e1-778c566cc6-dq2vf"] Oct 31 13:42:46.465007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:46.463125 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:46.761159 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:46.761123 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:46.995491 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:46.995454 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:47.348005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:47.347594 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:47.649088 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:47.648243 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="202fa689-242a-4b4d-8ed6-d3992b0883ff" path="/var/lib/kubelet/pods/202fa689-242a-4b4d-8ed6-d3992b0883ff/volumes" Oct 31 13:42:47.737407 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:47.737368 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:47.980956 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:47.980921 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:48.986323 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:48.986283 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:49.526440 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:49.526398 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:49.996369 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:49.996329 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:50.040925 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:42:50.040878 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:42:50.069776 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:50.069748 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn" Oct 31 13:42:50.081970 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:50.081943 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn" Oct 31 13:42:50.823717 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:50.823682 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:51.020662 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.020631 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn" Oct 31 13:42:51.058170 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.058122 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh"] Oct 31 13:42:51.059136 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.059110 6841 cpu_manager.go:414] "containerMap: removing container" podUID="202fa689-242a-4b4d-8ed6-d3992b0883ff" containerName="maas-api" Oct 31 13:42:51.059136 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.059135 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="202fa689-242a-4b4d-8ed6-d3992b0883ff" containerName="maas-api" Oct 31 13:42:51.059343 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.059174 6841 cpu_manager.go:414] "containerMap: removing container" podUID="ff57031e-a357-4764-8c12-2463afb93ac7" containerName="main" Oct 31 13:42:51.059343 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.059183 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff57031e-a357-4764-8c12-2463afb93ac7" containerName="main" Oct 31 13:42:51.059449 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.059345 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="202fa689-242a-4b4d-8ed6-d3992b0883ff" containerName="maas-api" Oct 31 13:42:51.059449 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.059363 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff57031e-a357-4764-8c12-2463afb93ac7" containerName="main" Oct 31 13:42:51.065427 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.065401 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" Oct 31 13:42:51.082048 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.071961 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-5452fc-openshift-default-dockercfg-vgclq\"" Oct 31 13:42:51.082048 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.076258 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh"] Oct 31 13:42:51.174757 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.174718 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7095807-f9e9-47ba-be79-262d1b097fc2-istio-token\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh\" (UID: \"f7095807-f9e9-47ba-be79-262d1b097fc2\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" Oct 31 13:42:51.174944 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.174840 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7095807-f9e9-47ba-be79-262d1b097fc2-istiod-ca-cert\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh\" (UID: \"f7095807-f9e9-47ba-be79-262d1b097fc2\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" Oct 31 13:42:51.174944 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.174878 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7095807-f9e9-47ba-be79-262d1b097fc2-workload-certs\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh\" (UID: \"f7095807-f9e9-47ba-be79-262d1b097fc2\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" Oct 31 13:42:51.175083 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.174949 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7095807-f9e9-47ba-be79-262d1b097fc2-istio-podinfo\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh\" (UID: \"f7095807-f9e9-47ba-be79-262d1b097fc2\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" Oct 31 13:42:51.175083 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.174973 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7095807-f9e9-47ba-be79-262d1b097fc2-credential-socket\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh\" (UID: \"f7095807-f9e9-47ba-be79-262d1b097fc2\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" Oct 31 13:42:51.175083 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.175027 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br2fh\" (UniqueName: \"kubernetes.io/projected/f7095807-f9e9-47ba-be79-262d1b097fc2-kube-api-access-br2fh\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh\" (UID: \"f7095807-f9e9-47ba-be79-262d1b097fc2\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" Oct 31 13:42:51.175202 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.175146 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7095807-f9e9-47ba-be79-262d1b097fc2-istio-envoy\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh\" (UID: \"f7095807-f9e9-47ba-be79-262d1b097fc2\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" Oct 31 13:42:51.175246 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.175224 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7095807-f9e9-47ba-be79-262d1b097fc2-workload-socket\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh\" (UID: \"f7095807-f9e9-47ba-be79-262d1b097fc2\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" Oct 31 13:42:51.175313 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.175299 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7095807-f9e9-47ba-be79-262d1b097fc2-istio-data\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh\" (UID: \"f7095807-f9e9-47ba-be79-262d1b097fc2\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" Oct 31 13:42:51.276002 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.275947 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7095807-f9e9-47ba-be79-262d1b097fc2-istio-token\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh\" (UID: \"f7095807-f9e9-47ba-be79-262d1b097fc2\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" Oct 31 13:42:51.276175 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.276071 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7095807-f9e9-47ba-be79-262d1b097fc2-istiod-ca-cert\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh\" (UID: \"f7095807-f9e9-47ba-be79-262d1b097fc2\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" Oct 31 13:42:51.276175 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.276103 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7095807-f9e9-47ba-be79-262d1b097fc2-workload-certs\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh\" (UID: \"f7095807-f9e9-47ba-be79-262d1b097fc2\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" Oct 31 13:42:51.276175 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.276133 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7095807-f9e9-47ba-be79-262d1b097fc2-istio-podinfo\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh\" (UID: \"f7095807-f9e9-47ba-be79-262d1b097fc2\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" Oct 31 13:42:51.276175 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.276158 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7095807-f9e9-47ba-be79-262d1b097fc2-credential-socket\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh\" (UID: \"f7095807-f9e9-47ba-be79-262d1b097fc2\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" Oct 31 13:42:51.276449 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.276192 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br2fh\" (UniqueName: \"kubernetes.io/projected/f7095807-f9e9-47ba-be79-262d1b097fc2-kube-api-access-br2fh\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh\" (UID: \"f7095807-f9e9-47ba-be79-262d1b097fc2\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" Oct 31 13:42:51.276449 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.276236 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7095807-f9e9-47ba-be79-262d1b097fc2-istio-envoy\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh\" (UID: \"f7095807-f9e9-47ba-be79-262d1b097fc2\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" Oct 31 13:42:51.276449 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.276320 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7095807-f9e9-47ba-be79-262d1b097fc2-workload-socket\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh\" (UID: \"f7095807-f9e9-47ba-be79-262d1b097fc2\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" Oct 31 13:42:51.276449 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.276424 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7095807-f9e9-47ba-be79-262d1b097fc2-istio-data\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh\" (UID: \"f7095807-f9e9-47ba-be79-262d1b097fc2\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" Oct 31 13:42:51.276677 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.276572 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7095807-f9e9-47ba-be79-262d1b097fc2-workload-certs\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh\" (UID: \"f7095807-f9e9-47ba-be79-262d1b097fc2\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" Oct 31 13:42:51.276739 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.276673 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7095807-f9e9-47ba-be79-262d1b097fc2-credential-socket\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh\" (UID: \"f7095807-f9e9-47ba-be79-262d1b097fc2\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" Oct 31 13:42:51.276739 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.276716 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7095807-f9e9-47ba-be79-262d1b097fc2-workload-socket\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh\" (UID: \"f7095807-f9e9-47ba-be79-262d1b097fc2\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" Oct 31 13:42:51.276997 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.276963 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7095807-f9e9-47ba-be79-262d1b097fc2-istiod-ca-cert\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh\" (UID: \"f7095807-f9e9-47ba-be79-262d1b097fc2\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" Oct 31 13:42:51.277096 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.277056 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7095807-f9e9-47ba-be79-262d1b097fc2-istio-data\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh\" (UID: \"f7095807-f9e9-47ba-be79-262d1b097fc2\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" Oct 31 13:42:51.278793 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.278769 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7095807-f9e9-47ba-be79-262d1b097fc2-istio-envoy\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh\" (UID: \"f7095807-f9e9-47ba-be79-262d1b097fc2\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" Oct 31 13:42:51.279349 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.279327 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7095807-f9e9-47ba-be79-262d1b097fc2-istio-podinfo\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh\" (UID: \"f7095807-f9e9-47ba-be79-262d1b097fc2\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" Oct 31 13:42:51.308618 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.308591 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7095807-f9e9-47ba-be79-262d1b097fc2-istio-token\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh\" (UID: \"f7095807-f9e9-47ba-be79-262d1b097fc2\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" Oct 31 13:42:51.308726 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.308703 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br2fh\" (UniqueName: \"kubernetes.io/projected/f7095807-f9e9-47ba-be79-262d1b097fc2-kube-api-access-br2fh\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh\" (UID: \"f7095807-f9e9-47ba-be79-262d1b097fc2\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" Oct 31 13:42:51.385287 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.385200 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" Oct 31 13:42:51.696528 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.696493 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh"] Oct 31 13:42:51.698239 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:42:51.698207 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7095807_f9e9_47ba_be79_262d1b097fc2.slice/crio-fec811275179fc5e8f34bfb0880bcb7b318056adcf826bc232be4c759c12910b WatchSource:0}: Error finding container fec811275179fc5e8f34bfb0880bcb7b318056adcf826bc232be4c759c12910b: Status 404 returned error can't find the container with id fec811275179fc5e8f34bfb0880bcb7b318056adcf826bc232be4c759c12910b Oct 31 13:42:51.701295 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.701262 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:42:51.701395 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.701333 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:42:51.701395 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.701367 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:42:51.768754 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:51.768713 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:52.020182 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:52.020026 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" event={"ID":"f7095807-f9e9-47ba-be79-262d1b097fc2","Type":"ContainerStarted","Data":"65314cc449d23fc5fda8ca78706aee00cae11f94f62b53d31d9d33193838f6cf"} Oct 31 13:42:52.020182 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:52.020089 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" event={"ID":"f7095807-f9e9-47ba-be79-262d1b097fc2","Type":"ContainerStarted","Data":"fec811275179fc5e8f34bfb0880bcb7b318056adcf826bc232be4c759c12910b"} Oct 31 13:42:52.048857 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:52.048794 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" podStartSLOduration=1.048774975 podStartE2EDuration="1.048774975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:42:52.046825759 +0000 UTC m=+4218.529948660" watchObservedRunningTime="2026-10-31 13:42:52.048774975 +0000 UTC m=+4218.531897876" Oct 31 13:42:52.386110 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:52.386019 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" Oct 31 13:42:52.387926 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:52.387896 6841 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 31 13:42:52.388101 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:52.387960 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" podUID="f7095807-f9e9-47ba-be79-262d1b097fc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 31 13:42:52.511326 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:52.511292 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:52.818199 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:52.818162 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:53.327213 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:53.327181 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:53.386306 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:53.386269 6841 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 31 13:42:53.386518 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:53.386328 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" podUID="f7095807-f9e9-47ba-be79-262d1b097fc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 31 13:42:54.386375 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:54.386335 6841 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 31 13:42:54.386790 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:54.386418 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" podUID="f7095807-f9e9-47ba-be79-262d1b097fc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 31 13:42:54.501666 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:54.501626 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:54.894511 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:54.894468 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:55.385949 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:55.385912 6841 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 31 13:42:55.386159 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:55.385970 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" podUID="f7095807-f9e9-47ba-be79-262d1b097fc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 31 13:42:55.449005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:55.441452 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:55.654062 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:55.651228 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:55.784290 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:55.784241 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g" podUID="c017a13d-2772-4379-80e6-6ce0d37af2c5" containerName="main" containerID="cri-o://7f954cf752d58d0a71b24b8f7a69fa208026242bd56aeae58fb6ce2d1575ed71" gracePeriod=45 Oct 31 13:42:56.100933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:56.100902 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g" Oct 31 13:42:56.138354 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:56.138316 6841 generic.go:356] "Generic (PLEG): container finished" podID="c017a13d-2772-4379-80e6-6ce0d37af2c5" containerID="7f954cf752d58d0a71b24b8f7a69fa208026242bd56aeae58fb6ce2d1575ed71" exitCode=0 Oct 31 13:42:56.138555 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:56.138380 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g" Oct 31 13:42:56.138555 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:56.138409 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g" event={"ID":"c017a13d-2772-4379-80e6-6ce0d37af2c5","Type":"ContainerDied","Data":"7f954cf752d58d0a71b24b8f7a69fa208026242bd56aeae58fb6ce2d1575ed71"} Oct 31 13:42:56.138555 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:56.138446 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g" event={"ID":"c017a13d-2772-4379-80e6-6ce0d37af2c5","Type":"ContainerDied","Data":"73e5c992bae9755f9107ba9d8f9e1afead07b33623b166be7f57d42bb015ed72"} Oct 31 13:42:56.138555 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:56.138468 6841 scope.go:122] "RemoveContainer" containerID="7f954cf752d58d0a71b24b8f7a69fa208026242bd56aeae58fb6ce2d1575ed71" Oct 31 13:42:56.166467 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:56.166428 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c017a13d-2772-4379-80e6-6ce0d37af2c5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c017a13d-2772-4379-80e6-6ce0d37af2c5-dshm\") pod \"c017a13d-2772-4379-80e6-6ce0d37af2c5\" (UID: \"c017a13d-2772-4379-80e6-6ce0d37af2c5\") " Oct 31 13:42:56.166664 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:56.166522 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c017a13d-2772-4379-80e6-6ce0d37af2c5-kube-api-access-s9gbm\" (UniqueName: \"kubernetes.io/projected/c017a13d-2772-4379-80e6-6ce0d37af2c5-kube-api-access-s9gbm\") pod \"c017a13d-2772-4379-80e6-6ce0d37af2c5\" (UID: \"c017a13d-2772-4379-80e6-6ce0d37af2c5\") " Oct 31 13:42:56.166664 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:56.166564 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c017a13d-2772-4379-80e6-6ce0d37af2c5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c017a13d-2772-4379-80e6-6ce0d37af2c5-model-cache\") pod \"c017a13d-2772-4379-80e6-6ce0d37af2c5\" (UID: \"c017a13d-2772-4379-80e6-6ce0d37af2c5\") " Oct 31 13:42:56.166795 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:56.166725 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c017a13d-2772-4379-80e6-6ce0d37af2c5-tls-certs\" (UniqueName: \"kubernetes.io/secret/c017a13d-2772-4379-80e6-6ce0d37af2c5-tls-certs\") pod \"c017a13d-2772-4379-80e6-6ce0d37af2c5\" (UID: \"c017a13d-2772-4379-80e6-6ce0d37af2c5\") " Oct 31 13:42:56.166865 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:56.166792 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c017a13d-2772-4379-80e6-6ce0d37af2c5-home\" (UniqueName: \"kubernetes.io/empty-dir/c017a13d-2772-4379-80e6-6ce0d37af2c5-home\") pod \"c017a13d-2772-4379-80e6-6ce0d37af2c5\" (UID: \"c017a13d-2772-4379-80e6-6ce0d37af2c5\") " Oct 31 13:42:56.166865 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:56.166852 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c017a13d-2772-4379-80e6-6ce0d37af2c5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c017a13d-2772-4379-80e6-6ce0d37af2c5-tmp-dir\") pod \"c017a13d-2772-4379-80e6-6ce0d37af2c5\" (UID: \"c017a13d-2772-4379-80e6-6ce0d37af2c5\") " Oct 31 13:42:56.167855 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:56.167286 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c017a13d-2772-4379-80e6-6ce0d37af2c5-model-cache" pod "c017a13d-2772-4379-80e6-6ce0d37af2c5" (UID: "c017a13d-2772-4379-80e6-6ce0d37af2c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:56.167855 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:56.167512 6841 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c017a13d-2772-4379-80e6-6ce0d37af2c5-model-cache\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:56.167855 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:56.167537 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c017a13d-2772-4379-80e6-6ce0d37af2c5-tmp-dir" pod "c017a13d-2772-4379-80e6-6ce0d37af2c5" (UID: "c017a13d-2772-4379-80e6-6ce0d37af2c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:56.168189 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:56.168092 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c017a13d-2772-4379-80e6-6ce0d37af2c5-home" pod "c017a13d-2772-4379-80e6-6ce0d37af2c5" (UID: "c017a13d-2772-4379-80e6-6ce0d37af2c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:56.170934 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:56.170904 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c017a13d-2772-4379-80e6-6ce0d37af2c5-dshm" pod "c017a13d-2772-4379-80e6-6ce0d37af2c5" (UID: "c017a13d-2772-4379-80e6-6ce0d37af2c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:56.171125 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:56.171101 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c017a13d-2772-4379-80e6-6ce0d37af2c5-kube-api-access-s9gbm" pod "c017a13d-2772-4379-80e6-6ce0d37af2c5" (UID: "c017a13d-2772-4379-80e6-6ce0d37af2c5"). InnerVolumeSpecName "kube-api-access-s9gbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:42:56.174425 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:56.174388 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c017a13d-2772-4379-80e6-6ce0d37af2c5-tls-certs" pod "c017a13d-2772-4379-80e6-6ce0d37af2c5" (UID: "c017a13d-2772-4379-80e6-6ce0d37af2c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:42:56.177373 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:56.177347 6841 scope.go:122] "RemoveContainer" containerID="7f954cf752d58d0a71b24b8f7a69fa208026242bd56aeae58fb6ce2d1575ed71" Oct 31 13:42:56.177777 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:42:56.177749 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f954cf752d58d0a71b24b8f7a69fa208026242bd56aeae58fb6ce2d1575ed71\": container with ID starting with 7f954cf752d58d0a71b24b8f7a69fa208026242bd56aeae58fb6ce2d1575ed71 not found: ID does not exist" containerID="7f954cf752d58d0a71b24b8f7a69fa208026242bd56aeae58fb6ce2d1575ed71" Oct 31 13:42:56.177887 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:56.177789 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f954cf752d58d0a71b24b8f7a69fa208026242bd56aeae58fb6ce2d1575ed71"} err="failed to get container status \"7f954cf752d58d0a71b24b8f7a69fa208026242bd56aeae58fb6ce2d1575ed71\": rpc error: code = NotFound desc = could not find container \"7f954cf752d58d0a71b24b8f7a69fa208026242bd56aeae58fb6ce2d1575ed71\": container with ID starting with 7f954cf752d58d0a71b24b8f7a69fa208026242bd56aeae58fb6ce2d1575ed71 not found: ID does not exist" Oct 31 13:42:56.269047 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:56.268943 6841 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c017a13d-2772-4379-80e6-6ce0d37af2c5-dshm\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:56.269047 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:56.268977 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9gbm\" (UniqueName: \"kubernetes.io/projected/c017a13d-2772-4379-80e6-6ce0d37af2c5-kube-api-access-s9gbm\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:56.269047 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:56.269006 6841 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c017a13d-2772-4379-80e6-6ce0d37af2c5-tls-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:56.269047 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:56.269018 6841 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c017a13d-2772-4379-80e6-6ce0d37af2c5-home\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:56.269047 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:56.269032 6841 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c017a13d-2772-4379-80e6-6ce0d37af2c5-tmp-dir\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:56.384276 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:56.384239 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:56.386229 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:56.386203 6841 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 31 13:42:56.386324 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:56.386256 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" podUID="f7095807-f9e9-47ba-be79-262d1b097fc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 31 13:42:56.490760 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:56.490727 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g"] Oct 31 13:42:56.500193 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:56.500161 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-45bcad/sub-test-model-45bcad-kserve-5ff99cc5cf-xk57g"] Oct 31 13:42:57.160805 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:57.160770 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:57.385607 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:57.385567 6841 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 31 13:42:57.385795 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:57.385635 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" podUID="f7095807-f9e9-47ba-be79-262d1b097fc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 31 13:42:57.657382 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:57.657343 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c017a13d-2772-4379-80e6-6ce0d37af2c5" path="/var/lib/kubelet/pods/c017a13d-2772-4379-80e6-6ce0d37af2c5/volumes" Oct 31 13:42:57.973328 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:57.973289 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:58.391256 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:58.391180 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" Oct 31 13:42:58.391538 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:58.391516 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" Oct 31 13:42:58.392238 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:58.392218 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" Oct 31 13:42:58.404584 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:58.404549 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:58.868079 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:58.868046 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:59.131480 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:59.131396 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:42:59.947943 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:42:59.947903 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:00.882521 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:00.882475 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:01.323103 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:01.323063 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh" podUID="12a97fe1-6ffc-4359-a6f7-063d6c065e31" containerName="main" containerID="cri-o://c582e173b95d80405b5cd34379f3172b4a89d3f2b5dfc5c07e418f2d1325ab14" gracePeriod=45 Oct 31 13:43:01.603679 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:01.603652 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh" Oct 31 13:43:01.750906 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:01.750868 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12a97fe1-6ffc-4359-a6f7-063d6c065e31-tls-certs\" (UniqueName: \"kubernetes.io/secret/12a97fe1-6ffc-4359-a6f7-063d6c065e31-tls-certs\") pod \"12a97fe1-6ffc-4359-a6f7-063d6c065e31\" (UID: \"12a97fe1-6ffc-4359-a6f7-063d6c065e31\") " Oct 31 13:43:01.751126 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:01.751070 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12a97fe1-6ffc-4359-a6f7-063d6c065e31-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12a97fe1-6ffc-4359-a6f7-063d6c065e31-model-cache\") pod \"12a97fe1-6ffc-4359-a6f7-063d6c065e31\" (UID: \"12a97fe1-6ffc-4359-a6f7-063d6c065e31\") " Oct 31 13:43:01.751126 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:01.751103 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12a97fe1-6ffc-4359-a6f7-063d6c065e31-dshm\" (UniqueName: \"kubernetes.io/empty-dir/12a97fe1-6ffc-4359-a6f7-063d6c065e31-dshm\") pod \"12a97fe1-6ffc-4359-a6f7-063d6c065e31\" (UID: \"12a97fe1-6ffc-4359-a6f7-063d6c065e31\") " Oct 31 13:43:01.751260 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:01.751180 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12a97fe1-6ffc-4359-a6f7-063d6c065e31-home\" (UniqueName: \"kubernetes.io/empty-dir/12a97fe1-6ffc-4359-a6f7-063d6c065e31-home\") pod \"12a97fe1-6ffc-4359-a6f7-063d6c065e31\" (UID: \"12a97fe1-6ffc-4359-a6f7-063d6c065e31\") " Oct 31 13:43:01.751260 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:01.751243 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12a97fe1-6ffc-4359-a6f7-063d6c065e31-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12a97fe1-6ffc-4359-a6f7-063d6c065e31-tmp-dir\") pod \"12a97fe1-6ffc-4359-a6f7-063d6c065e31\" (UID: \"12a97fe1-6ffc-4359-a6f7-063d6c065e31\") " Oct 31 13:43:01.751373 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:01.751278 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12a97fe1-6ffc-4359-a6f7-063d6c065e31-kube-api-access-bf8ks\" (UniqueName: \"kubernetes.io/projected/12a97fe1-6ffc-4359-a6f7-063d6c065e31-kube-api-access-bf8ks\") pod \"12a97fe1-6ffc-4359-a6f7-063d6c065e31\" (UID: \"12a97fe1-6ffc-4359-a6f7-063d6c065e31\") " Oct 31 13:43:01.751373 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:01.751335 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12a97fe1-6ffc-4359-a6f7-063d6c065e31-model-cache" pod "12a97fe1-6ffc-4359-a6f7-063d6c065e31" (UID: "12a97fe1-6ffc-4359-a6f7-063d6c065e31"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:01.751485 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:01.751458 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12a97fe1-6ffc-4359-a6f7-063d6c065e31-home" pod "12a97fe1-6ffc-4359-a6f7-063d6c065e31" (UID: "12a97fe1-6ffc-4359-a6f7-063d6c065e31"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:01.751540 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:01.751483 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12a97fe1-6ffc-4359-a6f7-063d6c065e31-tmp-dir" pod "12a97fe1-6ffc-4359-a6f7-063d6c065e31" (UID: "12a97fe1-6ffc-4359-a6f7-063d6c065e31"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:01.751832 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:01.751803 6841 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12a97fe1-6ffc-4359-a6f7-063d6c065e31-home\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:01.751832 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:01.751825 6841 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12a97fe1-6ffc-4359-a6f7-063d6c065e31-tmp-dir\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:01.752032 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:01.751839 6841 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12a97fe1-6ffc-4359-a6f7-063d6c065e31-model-cache\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:01.753191 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:01.753162 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12a97fe1-6ffc-4359-a6f7-063d6c065e31-tls-certs" pod "12a97fe1-6ffc-4359-a6f7-063d6c065e31" (UID: "12a97fe1-6ffc-4359-a6f7-063d6c065e31"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:43:01.753306 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:01.753213 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12a97fe1-6ffc-4359-a6f7-063d6c065e31-dshm" pod "12a97fe1-6ffc-4359-a6f7-063d6c065e31" (UID: "12a97fe1-6ffc-4359-a6f7-063d6c065e31"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:01.753421 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:01.753406 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12a97fe1-6ffc-4359-a6f7-063d6c065e31-kube-api-access-bf8ks" pod "12a97fe1-6ffc-4359-a6f7-063d6c065e31" (UID: "12a97fe1-6ffc-4359-a6f7-063d6c065e31"). InnerVolumeSpecName "kube-api-access-bf8ks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:01.853628 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:01.853546 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf8ks\" (UniqueName: \"kubernetes.io/projected/12a97fe1-6ffc-4359-a6f7-063d6c065e31-kube-api-access-bf8ks\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:01.853628 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:01.853577 6841 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12a97fe1-6ffc-4359-a6f7-063d6c065e31-tls-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:01.853628 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:01.853587 6841 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12a97fe1-6ffc-4359-a6f7-063d6c065e31-dshm\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:02.068833 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:02.068796 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:02.199576 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:02.199535 6841 generic.go:356] "Generic (PLEG): container finished" podID="12a97fe1-6ffc-4359-a6f7-063d6c065e31" containerID="c582e173b95d80405b5cd34379f3172b4a89d3f2b5dfc5c07e418f2d1325ab14" exitCode=0 Oct 31 13:43:02.199759 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:02.199584 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh" Oct 31 13:43:02.199759 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:02.199592 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh" event={"ID":"12a97fe1-6ffc-4359-a6f7-063d6c065e31","Type":"ContainerDied","Data":"c582e173b95d80405b5cd34379f3172b4a89d3f2b5dfc5c07e418f2d1325ab14"} Oct 31 13:43:02.199759 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:02.199631 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh" event={"ID":"12a97fe1-6ffc-4359-a6f7-063d6c065e31","Type":"ContainerDied","Data":"3e56823bbd2532440c68cf78f2be63ac990ad61934bcf43dbbd6ca7da7d55712"} Oct 31 13:43:02.199759 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:02.199654 6841 scope.go:122] "RemoveContainer" containerID="c582e173b95d80405b5cd34379f3172b4a89d3f2b5dfc5c07e418f2d1325ab14" Oct 31 13:43:02.210500 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:02.210476 6841 scope.go:122] "RemoveContainer" containerID="c582e173b95d80405b5cd34379f3172b4a89d3f2b5dfc5c07e418f2d1325ab14" Oct 31 13:43:02.210786 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:43:02.210764 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c582e173b95d80405b5cd34379f3172b4a89d3f2b5dfc5c07e418f2d1325ab14\": container with ID starting with c582e173b95d80405b5cd34379f3172b4a89d3f2b5dfc5c07e418f2d1325ab14 not found: ID does not exist" containerID="c582e173b95d80405b5cd34379f3172b4a89d3f2b5dfc5c07e418f2d1325ab14" Oct 31 13:43:02.210841 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:02.210793 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c582e173b95d80405b5cd34379f3172b4a89d3f2b5dfc5c07e418f2d1325ab14"} err="failed to get container status \"c582e173b95d80405b5cd34379f3172b4a89d3f2b5dfc5c07e418f2d1325ab14\": rpc error: code = NotFound desc = could not find container \"c582e173b95d80405b5cd34379f3172b4a89d3f2b5dfc5c07e418f2d1325ab14\": container with ID starting with c582e173b95d80405b5cd34379f3172b4a89d3f2b5dfc5c07e418f2d1325ab14 not found: ID does not exist" Oct 31 13:43:02.226335 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:02.226306 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh"] Oct 31 13:43:02.231068 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:02.231040 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh"] Oct 31 13:43:02.541086 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:02.540977 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-569075/sub-test-model-569075-kserve-696f786fbf-f56sh" podUID="12a97fe1-6ffc-4359-a6f7-063d6c065e31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.229:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:43:02.971831 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:02.971798 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:03.532502 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:03.532460 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:03.671584 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:03.671548 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12a97fe1-6ffc-4359-a6f7-063d6c065e31" path="/var/lib/kubelet/pods/12a97fe1-6ffc-4359-a6f7-063d6c065e31/volumes" Oct 31 13:43:03.740323 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:03.740285 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:04.248308 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:04.248271 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-622e03-7574857797-f4fdg"] Oct 31 13:43:04.266241 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:04.266210 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-622e03-55955d5bc9-nfrwx"] Oct 31 13:43:04.433615 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:04.433574 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:04.576072 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:04.565771 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94"] Oct 31 13:43:04.590004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:04.576521 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94" podUID="b508f7f6-8c58-4f97-bf9b-60ef66ad84db" containerName="maas-api" containerID="cri-o://730188602af72d5f0f82df444657e3ea6896e22e595a333d23e9b30af8231356" gracePeriod=30 Oct 31 13:43:04.656924 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:04.656884 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:04.990140 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:04.990105 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:05.117126 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:05.117102 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94" Oct 31 13:43:05.224211 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:05.224180 6841 generic.go:356] "Generic (PLEG): container finished" podID="b508f7f6-8c58-4f97-bf9b-60ef66ad84db" containerID="730188602af72d5f0f82df444657e3ea6896e22e595a333d23e9b30af8231356" exitCode=0 Oct 31 13:43:05.224374 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:05.224228 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94" Oct 31 13:43:05.224374 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:05.224260 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94" event={"ID":"b508f7f6-8c58-4f97-bf9b-60ef66ad84db","Type":"ContainerDied","Data":"730188602af72d5f0f82df444657e3ea6896e22e595a333d23e9b30af8231356"} Oct 31 13:43:05.224374 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:05.224311 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94" event={"ID":"b508f7f6-8c58-4f97-bf9b-60ef66ad84db","Type":"ContainerDied","Data":"406a67ce52fbf30a7e9606a8199113880effae2fdf7a85ac8f0be42eb83c3fc9"} Oct 31 13:43:05.224374 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:05.224337 6841 scope.go:122] "RemoveContainer" containerID="730188602af72d5f0f82df444657e3ea6896e22e595a333d23e9b30af8231356" Oct 31 13:43:05.234661 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:05.234645 6841 scope.go:122] "RemoveContainer" containerID="730188602af72d5f0f82df444657e3ea6896e22e595a333d23e9b30af8231356" Oct 31 13:43:05.234926 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:43:05.234908 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"730188602af72d5f0f82df444657e3ea6896e22e595a333d23e9b30af8231356\": container with ID starting with 730188602af72d5f0f82df444657e3ea6896e22e595a333d23e9b30af8231356 not found: ID does not exist" containerID="730188602af72d5f0f82df444657e3ea6896e22e595a333d23e9b30af8231356" Oct 31 13:43:05.234977 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:05.234935 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"730188602af72d5f0f82df444657e3ea6896e22e595a333d23e9b30af8231356"} err="failed to get container status \"730188602af72d5f0f82df444657e3ea6896e22e595a333d23e9b30af8231356\": rpc error: code = NotFound desc = could not find container \"730188602af72d5f0f82df444657e3ea6896e22e595a333d23e9b30af8231356\": container with ID starting with 730188602af72d5f0f82df444657e3ea6896e22e595a333d23e9b30af8231356 not found: ID does not exist" Oct 31 13:43:05.240568 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:05.240544 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b508f7f6-8c58-4f97-bf9b-60ef66ad84db-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b508f7f6-8c58-4f97-bf9b-60ef66ad84db-maas-api-tls\") pod \"b508f7f6-8c58-4f97-bf9b-60ef66ad84db\" (UID: \"b508f7f6-8c58-4f97-bf9b-60ef66ad84db\") " Oct 31 13:43:05.240693 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:05.240576 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b508f7f6-8c58-4f97-bf9b-60ef66ad84db-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b508f7f6-8c58-4f97-bf9b-60ef66ad84db-openshift-service-ca\") pod \"b508f7f6-8c58-4f97-bf9b-60ef66ad84db\" (UID: \"b508f7f6-8c58-4f97-bf9b-60ef66ad84db\") " Oct 31 13:43:05.240773 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:05.240755 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b508f7f6-8c58-4f97-bf9b-60ef66ad84db-kube-api-access-cz85j\" (UniqueName: \"kubernetes.io/projected/b508f7f6-8c58-4f97-bf9b-60ef66ad84db-kube-api-access-cz85j\") pod \"b508f7f6-8c58-4f97-bf9b-60ef66ad84db\" (UID: \"b508f7f6-8c58-4f97-bf9b-60ef66ad84db\") " Oct 31 13:43:05.240815 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:05.240803 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b508f7f6-8c58-4f97-bf9b-60ef66ad84db-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b508f7f6-8c58-4f97-bf9b-60ef66ad84db-metrics-certs\") pod \"b508f7f6-8c58-4f97-bf9b-60ef66ad84db\" (UID: \"b508f7f6-8c58-4f97-bf9b-60ef66ad84db\") " Oct 31 13:43:05.240952 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:05.240925 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b508f7f6-8c58-4f97-bf9b-60ef66ad84db-openshift-service-ca" pod "b508f7f6-8c58-4f97-bf9b-60ef66ad84db" (UID: "b508f7f6-8c58-4f97-bf9b-60ef66ad84db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:43:05.241459 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:05.241440 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b508f7f6-8c58-4f97-bf9b-60ef66ad84db-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:05.242667 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:05.242641 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b508f7f6-8c58-4f97-bf9b-60ef66ad84db-maas-api-tls" pod "b508f7f6-8c58-4f97-bf9b-60ef66ad84db" (UID: "b508f7f6-8c58-4f97-bf9b-60ef66ad84db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:43:05.242816 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:05.242796 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b508f7f6-8c58-4f97-bf9b-60ef66ad84db-kube-api-access-cz85j" pod "b508f7f6-8c58-4f97-bf9b-60ef66ad84db" (UID: "b508f7f6-8c58-4f97-bf9b-60ef66ad84db"). InnerVolumeSpecName "kube-api-access-cz85j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:05.242939 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:05.242925 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b508f7f6-8c58-4f97-bf9b-60ef66ad84db-metrics-certs" pod "b508f7f6-8c58-4f97-bf9b-60ef66ad84db" (UID: "b508f7f6-8c58-4f97-bf9b-60ef66ad84db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:43:05.342844 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:05.342808 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz85j\" (UniqueName: \"kubernetes.io/projected/b508f7f6-8c58-4f97-bf9b-60ef66ad84db-kube-api-access-cz85j\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:05.342844 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:05.342842 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b508f7f6-8c58-4f97-bf9b-60ef66ad84db-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:05.343149 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:05.342853 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b508f7f6-8c58-4f97-bf9b-60ef66ad84db-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:05.388067 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:05.387951 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:05.551528 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:05.551496 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94"] Oct 31 13:43:05.560132 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:05.560102 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-5dc570-5b9785b886-dzs94"] Oct 31 13:43:05.642929 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:05.642841 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b508f7f6-8c58-4f97-bf9b-60ef66ad84db" path="/var/lib/kubelet/pods/b508f7f6-8c58-4f97-bf9b-60ef66ad84db/volumes" Oct 31 13:43:06.218723 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:06.218682 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:06.646679 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:06.646589 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:07.031548 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:07.031510 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:08.058098 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:08.058068 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:09.030844 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:09.030811 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:09.249189 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:09.249151 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-622e03-7574857797-f4fdg" podUID="fb6f239c-4a3d-4f98-9b5e-4d775a4c561c" containerName="payload-processing" containerID="cri-o://18de47f6118d9faf6e6dfe92a33834523ed70682bedcb1c91ddb46592c3e11ef" gracePeriod=25 Oct 31 13:43:09.266694 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:09.266649 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-622e03-55955d5bc9-nfrwx" podUID="96229665-4486-4750-becf-d30f28c47dd5" containerName="payload-pre-processing" containerID="cri-o://d4baf7017ccf84c7ec94e5ee6a988983ff78fc8dab38fc6ef15de6fde9b6d2bb" gracePeriod=25 Oct 31 13:43:09.877543 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:09.877498 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:10.726728 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:10.726694 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:11.174296 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:11.169230 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:11.857931 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:11.856852 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:11.926788 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:11.926748 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-622e03-55955d5bc9-nfrwx" podUID="96229665-4486-4750-becf-d30f28c47dd5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:43:11.928168 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:11.928131 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-622e03-7574857797-f4fdg" podUID="fb6f239c-4a3d-4f98-9b5e-4d775a4c561c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:43:12.161147 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:12.158598 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:12.722588 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:12.722548 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:13.172397 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:13.172315 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt"] Oct 31 13:43:13.172798 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:13.172555 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" podUID="de4271dd-8fc1-4768-8dd7-2437ee497f9b" containerName="istio-proxy" containerID="cri-o://7195124b4c67e2f273cfc62efa15afb7d3914a5d9b781101d9f947459bf67750" gracePeriod=30 Oct 31 13:43:13.351662 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:13.351583 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:13.620424 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:13.620389 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:14.108584 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:14.108537 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:14.912756 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:14.912719 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:15.743645 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:15.743608 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:16.432352 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:16.432316 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:16.758109 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:16.758075 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:17.456579 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:17.456540 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm"] Oct 31 13:43:17.457142 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:17.456785 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm" podUID="85eedb21-9713-44f5-aaa7-6e8906b2df27" containerName="maas-api" containerID="cri-o://26f5db959b416458d2a5907d9d29f851da68149a6c0465ca586f56fde54c97ad" gracePeriod=30 Oct 31 13:43:17.778379 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:17.778350 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:17.828467 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:17.828434 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm" Oct 31 13:43:17.963967 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:17.963935 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85eedb21-9713-44f5-aaa7-6e8906b2df27-kube-api-access-2t7sc\" (UniqueName: \"kubernetes.io/projected/85eedb21-9713-44f5-aaa7-6e8906b2df27-kube-api-access-2t7sc\") pod \"85eedb21-9713-44f5-aaa7-6e8906b2df27\" (UID: \"85eedb21-9713-44f5-aaa7-6e8906b2df27\") " Oct 31 13:43:17.964209 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:17.963975 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85eedb21-9713-44f5-aaa7-6e8906b2df27-metrics-certs\" (UniqueName: \"kubernetes.io/secret/85eedb21-9713-44f5-aaa7-6e8906b2df27-metrics-certs\") pod \"85eedb21-9713-44f5-aaa7-6e8906b2df27\" (UID: \"85eedb21-9713-44f5-aaa7-6e8906b2df27\") " Oct 31 13:43:17.964209 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:17.964035 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85eedb21-9713-44f5-aaa7-6e8906b2df27-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85eedb21-9713-44f5-aaa7-6e8906b2df27-openshift-service-ca\") pod \"85eedb21-9713-44f5-aaa7-6e8906b2df27\" (UID: \"85eedb21-9713-44f5-aaa7-6e8906b2df27\") " Oct 31 13:43:17.964209 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:17.964069 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85eedb21-9713-44f5-aaa7-6e8906b2df27-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85eedb21-9713-44f5-aaa7-6e8906b2df27-maas-api-tls\") pod \"85eedb21-9713-44f5-aaa7-6e8906b2df27\" (UID: \"85eedb21-9713-44f5-aaa7-6e8906b2df27\") " Oct 31 13:43:17.964481 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:17.964452 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85eedb21-9713-44f5-aaa7-6e8906b2df27-openshift-service-ca" pod "85eedb21-9713-44f5-aaa7-6e8906b2df27" (UID: "85eedb21-9713-44f5-aaa7-6e8906b2df27"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:43:17.964742 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:17.964720 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85eedb21-9713-44f5-aaa7-6e8906b2df27-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:17.966303 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:17.966274 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85eedb21-9713-44f5-aaa7-6e8906b2df27-metrics-certs" pod "85eedb21-9713-44f5-aaa7-6e8906b2df27" (UID: "85eedb21-9713-44f5-aaa7-6e8906b2df27"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:43:17.966303 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:17.966288 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85eedb21-9713-44f5-aaa7-6e8906b2df27-kube-api-access-2t7sc" pod "85eedb21-9713-44f5-aaa7-6e8906b2df27" (UID: "85eedb21-9713-44f5-aaa7-6e8906b2df27"). InnerVolumeSpecName "kube-api-access-2t7sc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:17.966468 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:17.966418 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85eedb21-9713-44f5-aaa7-6e8906b2df27-maas-api-tls" pod "85eedb21-9713-44f5-aaa7-6e8906b2df27" (UID: "85eedb21-9713-44f5-aaa7-6e8906b2df27"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:43:18.065745 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.065658 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t7sc\" (UniqueName: \"kubernetes.io/projected/85eedb21-9713-44f5-aaa7-6e8906b2df27-kube-api-access-2t7sc\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:18.065745 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.065691 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85eedb21-9713-44f5-aaa7-6e8906b2df27-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:18.065745 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.065701 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85eedb21-9713-44f5-aaa7-6e8906b2df27-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:18.075577 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.075546 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:18.345317 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.345240 6841 generic.go:356] "Generic (PLEG): container finished" podID="de4271dd-8fc1-4768-8dd7-2437ee497f9b" containerID="7195124b4c67e2f273cfc62efa15afb7d3914a5d9b781101d9f947459bf67750" exitCode=0 Oct 31 13:43:18.345473 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.345312 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" event={"ID":"de4271dd-8fc1-4768-8dd7-2437ee497f9b","Type":"ContainerDied","Data":"7195124b4c67e2f273cfc62efa15afb7d3914a5d9b781101d9f947459bf67750"} Oct 31 13:43:18.345473 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.345402 6841 scope.go:122] "RemoveContainer" containerID="7502228c566b3d881de24554dfa7716749434c2266896799063260e4b191daf1" Oct 31 13:43:18.346999 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.346962 6841 generic.go:356] "Generic (PLEG): container finished" podID="85eedb21-9713-44f5-aaa7-6e8906b2df27" containerID="26f5db959b416458d2a5907d9d29f851da68149a6c0465ca586f56fde54c97ad" exitCode=0 Oct 31 13:43:18.347130 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.347032 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm" event={"ID":"85eedb21-9713-44f5-aaa7-6e8906b2df27","Type":"ContainerDied","Data":"26f5db959b416458d2a5907d9d29f851da68149a6c0465ca586f56fde54c97ad"} Oct 31 13:43:18.347130 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.347063 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm" Oct 31 13:43:18.347130 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.347081 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm" event={"ID":"85eedb21-9713-44f5-aaa7-6e8906b2df27","Type":"ContainerDied","Data":"a33fc55606119a78fcb5f0a606b3a58ed5c2275ef1496ff435203240a67ac3c1"} Oct 31 13:43:18.369046 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.369021 6841 scope.go:122] "RemoveContainer" containerID="26f5db959b416458d2a5907d9d29f851da68149a6c0465ca586f56fde54c97ad" Oct 31 13:43:18.382102 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.382077 6841 scope.go:122] "RemoveContainer" containerID="26f5db959b416458d2a5907d9d29f851da68149a6c0465ca586f56fde54c97ad" Oct 31 13:43:18.382437 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:43:18.382415 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26f5db959b416458d2a5907d9d29f851da68149a6c0465ca586f56fde54c97ad\": container with ID starting with 26f5db959b416458d2a5907d9d29f851da68149a6c0465ca586f56fde54c97ad not found: ID does not exist" containerID="26f5db959b416458d2a5907d9d29f851da68149a6c0465ca586f56fde54c97ad" Oct 31 13:43:18.382557 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.382443 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26f5db959b416458d2a5907d9d29f851da68149a6c0465ca586f56fde54c97ad"} err="failed to get container status \"26f5db959b416458d2a5907d9d29f851da68149a6c0465ca586f56fde54c97ad\": rpc error: code = NotFound desc = could not find container \"26f5db959b416458d2a5907d9d29f851da68149a6c0465ca586f56fde54c97ad\": container with ID starting with 26f5db959b416458d2a5907d9d29f851da68149a6c0465ca586f56fde54c97ad not found: ID does not exist" Oct 31 13:43:18.386612 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.386584 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm"] Oct 31 13:43:18.389899 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.389877 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-622e03-749b6b6d68-6f8vm"] Oct 31 13:43:18.441065 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.441042 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:43:18.571703 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.571673 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de4271dd-8fc1-4768-8dd7-2437ee497f9b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4271dd-8fc1-4768-8dd7-2437ee497f9b-credential-socket\") pod \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\" (UID: \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\") " Oct 31 13:43:18.572201 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.571753 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de4271dd-8fc1-4768-8dd7-2437ee497f9b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de4271dd-8fc1-4768-8dd7-2437ee497f9b-istio-data\") pod \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\" (UID: \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\") " Oct 31 13:43:18.572201 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.571772 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de4271dd-8fc1-4768-8dd7-2437ee497f9b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de4271dd-8fc1-4768-8dd7-2437ee497f9b-istiod-ca-cert\") pod \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\" (UID: \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\") " Oct 31 13:43:18.572201 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.571845 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de4271dd-8fc1-4768-8dd7-2437ee497f9b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de4271dd-8fc1-4768-8dd7-2437ee497f9b-workload-certs\") pod \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\" (UID: \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\") " Oct 31 13:43:18.572201 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.571892 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de4271dd-8fc1-4768-8dd7-2437ee497f9b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4271dd-8fc1-4768-8dd7-2437ee497f9b-workload-socket\") pod \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\" (UID: \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\") " Oct 31 13:43:18.572201 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.572025 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de4271dd-8fc1-4768-8dd7-2437ee497f9b-credential-socket" pod "de4271dd-8fc1-4768-8dd7-2437ee497f9b" (UID: "de4271dd-8fc1-4768-8dd7-2437ee497f9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:18.572201 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.572033 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de4271dd-8fc1-4768-8dd7-2437ee497f9b-kube-api-access-7zzh8\" (UniqueName: \"kubernetes.io/projected/de4271dd-8fc1-4768-8dd7-2437ee497f9b-kube-api-access-7zzh8\") pod \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\" (UID: \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\") " Oct 31 13:43:18.572201 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.572063 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de4271dd-8fc1-4768-8dd7-2437ee497f9b-istio-data" pod "de4271dd-8fc1-4768-8dd7-2437ee497f9b" (UID: "de4271dd-8fc1-4768-8dd7-2437ee497f9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:18.572201 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.572100 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de4271dd-8fc1-4768-8dd7-2437ee497f9b-istio-token\" (UniqueName: \"kubernetes.io/projected/de4271dd-8fc1-4768-8dd7-2437ee497f9b-istio-token\") pod \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\" (UID: \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\") " Oct 31 13:43:18.572630 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.572227 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de4271dd-8fc1-4768-8dd7-2437ee497f9b-workload-socket" pod "de4271dd-8fc1-4768-8dd7-2437ee497f9b" (UID: "de4271dd-8fc1-4768-8dd7-2437ee497f9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:18.572630 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.572243 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de4271dd-8fc1-4768-8dd7-2437ee497f9b-workload-certs" pod "de4271dd-8fc1-4768-8dd7-2437ee497f9b" (UID: "de4271dd-8fc1-4768-8dd7-2437ee497f9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:18.572630 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.572258 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de4271dd-8fc1-4768-8dd7-2437ee497f9b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de4271dd-8fc1-4768-8dd7-2437ee497f9b-istio-podinfo\") pod \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\" (UID: \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\") " Oct 31 13:43:18.572630 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.572340 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de4271dd-8fc1-4768-8dd7-2437ee497f9b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de4271dd-8fc1-4768-8dd7-2437ee497f9b-istio-envoy\") pod \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\" (UID: \"de4271dd-8fc1-4768-8dd7-2437ee497f9b\") " Oct 31 13:43:18.572630 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.572339 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de4271dd-8fc1-4768-8dd7-2437ee497f9b-istiod-ca-cert" pod "de4271dd-8fc1-4768-8dd7-2437ee497f9b" (UID: "de4271dd-8fc1-4768-8dd7-2437ee497f9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:43:18.572900 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.572874 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de4271dd-8fc1-4768-8dd7-2437ee497f9b-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:18.572964 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.572902 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4271dd-8fc1-4768-8dd7-2437ee497f9b-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:18.572964 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.572920 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4271dd-8fc1-4768-8dd7-2437ee497f9b-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:18.572964 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.572934 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de4271dd-8fc1-4768-8dd7-2437ee497f9b-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:18.572964 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.572949 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de4271dd-8fc1-4768-8dd7-2437ee497f9b-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:18.574276 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.574255 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de4271dd-8fc1-4768-8dd7-2437ee497f9b-istio-token" pod "de4271dd-8fc1-4768-8dd7-2437ee497f9b" (UID: "de4271dd-8fc1-4768-8dd7-2437ee497f9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:18.574405 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.574306 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de4271dd-8fc1-4768-8dd7-2437ee497f9b-kube-api-access-7zzh8" pod "de4271dd-8fc1-4768-8dd7-2437ee497f9b" (UID: "de4271dd-8fc1-4768-8dd7-2437ee497f9b"). InnerVolumeSpecName "kube-api-access-7zzh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:18.574665 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.574650 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de4271dd-8fc1-4768-8dd7-2437ee497f9b-istio-envoy" pod "de4271dd-8fc1-4768-8dd7-2437ee497f9b" (UID: "de4271dd-8fc1-4768-8dd7-2437ee497f9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:18.574722 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.574676 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de4271dd-8fc1-4768-8dd7-2437ee497f9b-istio-podinfo" pod "de4271dd-8fc1-4768-8dd7-2437ee497f9b" (UID: "de4271dd-8fc1-4768-8dd7-2437ee497f9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:43:18.674395 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.674304 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zzh8\" (UniqueName: \"kubernetes.io/projected/de4271dd-8fc1-4768-8dd7-2437ee497f9b-kube-api-access-7zzh8\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:18.674395 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.674338 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de4271dd-8fc1-4768-8dd7-2437ee497f9b-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:18.674395 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.674348 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de4271dd-8fc1-4768-8dd7-2437ee497f9b-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:18.674395 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.674357 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de4271dd-8fc1-4768-8dd7-2437ee497f9b-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:18.839683 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:18.839636 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:19.355847 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:19.355819 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" Oct 31 13:43:19.355847 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:19.355834 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt" event={"ID":"de4271dd-8fc1-4768-8dd7-2437ee497f9b","Type":"ContainerDied","Data":"ea974dc1d8632beb4e73b563293641ead1a43446acfd6b79b366c37047127c2b"} Oct 31 13:43:19.356117 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:19.355888 6841 scope.go:122] "RemoveContainer" containerID="7195124b4c67e2f273cfc62efa15afb7d3914a5d9b781101d9f947459bf67750" Oct 31 13:43:19.401732 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:19.390245 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt"] Oct 31 13:43:19.435092 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:19.435045 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-48kdt"] Oct 31 13:43:19.501458 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:19.501405 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:19.641904 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:19.641828 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85eedb21-9713-44f5-aaa7-6e8906b2df27" path="/var/lib/kubelet/pods/85eedb21-9713-44f5-aaa7-6e8906b2df27/volumes" Oct 31 13:43:19.642401 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:19.642373 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de4271dd-8fc1-4768-8dd7-2437ee497f9b" path="/var/lib/kubelet/pods/de4271dd-8fc1-4768-8dd7-2437ee497f9b/volumes" Oct 31 13:43:19.829375 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:19.829340 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:20.035721 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:43:20.035683 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:43:20.107662 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:20.107629 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:21.213036 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:21.213005 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:21.926221 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:21.926182 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-622e03-55955d5bc9-nfrwx" podUID="96229665-4486-4750-becf-d30f28c47dd5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:43:21.926221 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:21.926205 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-622e03-7574857797-f4fdg" podUID="fb6f239c-4a3d-4f98-9b5e-4d775a4c561c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:43:22.347771 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:22.347739 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:22.957216 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:22.957178 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:23.220063 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:23.219939 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:23.631373 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:23.631281 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:23.882812 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:23.882729 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:24.365431 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.365394 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:24.662444 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.662360 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq"] Oct 31 13:43:24.662967 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.662943 6841 cpu_manager.go:414] "containerMap: removing container" podUID="12a97fe1-6ffc-4359-a6f7-063d6c065e31" containerName="main" Oct 31 13:43:24.662967 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.662966 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12a97fe1-6ffc-4359-a6f7-063d6c065e31" containerName="main" Oct 31 13:43:24.663106 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.662978 6841 cpu_manager.go:414] "containerMap: removing container" podUID="85eedb21-9713-44f5-aaa7-6e8906b2df27" containerName="maas-api" Oct 31 13:43:24.663106 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.663000 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85eedb21-9713-44f5-aaa7-6e8906b2df27" containerName="maas-api" Oct 31 13:43:24.663106 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.663014 6841 cpu_manager.go:414] "containerMap: removing container" podUID="c017a13d-2772-4379-80e6-6ce0d37af2c5" containerName="main" Oct 31 13:43:24.663106 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.663022 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c017a13d-2772-4379-80e6-6ce0d37af2c5" containerName="main" Oct 31 13:43:24.663106 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.663034 6841 cpu_manager.go:414] "containerMap: removing container" podUID="b508f7f6-8c58-4f97-bf9b-60ef66ad84db" containerName="maas-api" Oct 31 13:43:24.663106 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.663041 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b508f7f6-8c58-4f97-bf9b-60ef66ad84db" containerName="maas-api" Oct 31 13:43:24.663106 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.663066 6841 cpu_manager.go:414] "containerMap: removing container" podUID="de4271dd-8fc1-4768-8dd7-2437ee497f9b" containerName="istio-proxy" Oct 31 13:43:24.663106 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.663070 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de4271dd-8fc1-4768-8dd7-2437ee497f9b" containerName="istio-proxy" Oct 31 13:43:24.663106 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.663088 6841 cpu_manager.go:414] "containerMap: removing container" podUID="de4271dd-8fc1-4768-8dd7-2437ee497f9b" containerName="istio-proxy" Oct 31 13:43:24.663106 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.663093 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de4271dd-8fc1-4768-8dd7-2437ee497f9b" containerName="istio-proxy" Oct 31 13:43:24.663429 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.663194 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="de4271dd-8fc1-4768-8dd7-2437ee497f9b" containerName="istio-proxy" Oct 31 13:43:24.663429 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.663206 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="12a97fe1-6ffc-4359-a6f7-063d6c065e31" containerName="main" Oct 31 13:43:24.663429 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.663212 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="85eedb21-9713-44f5-aaa7-6e8906b2df27" containerName="maas-api" Oct 31 13:43:24.663429 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.663219 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="c017a13d-2772-4379-80e6-6ce0d37af2c5" containerName="main" Oct 31 13:43:24.663429 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.663226 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="b508f7f6-8c58-4f97-bf9b-60ef66ad84db" containerName="maas-api" Oct 31 13:43:24.668206 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.668185 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" Oct 31 13:43:24.678104 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.678080 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-622e03-openshift-default-dockercfg-br2cc\"" Oct 31 13:43:24.683061 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.683032 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6"] Oct 31 13:43:24.684198 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.684171 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="de4271dd-8fc1-4768-8dd7-2437ee497f9b" containerName="istio-proxy" Oct 31 13:43:24.687726 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.687706 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" Oct 31 13:43:24.695643 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.695611 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq"] Oct 31 13:43:24.704224 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.703118 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6"] Oct 31 13:43:24.828151 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.828116 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:24.869472 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.869436 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a899eb-181a-4f40-a64e-2953e94e6dd4-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6\" (UID: \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" Oct 31 13:43:24.869621 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.869478 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a0d55fc-97da-4fba-a45a-68a229f87264-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq\" (UID: \"8a0d55fc-97da-4fba-a45a-68a229f87264\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" Oct 31 13:43:24.869621 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.869506 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a0d55fc-97da-4fba-a45a-68a229f87264-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq\" (UID: \"8a0d55fc-97da-4fba-a45a-68a229f87264\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" Oct 31 13:43:24.869621 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.869599 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2q5h\" (UniqueName: \"kubernetes.io/projected/a3a899eb-181a-4f40-a64e-2953e94e6dd4-kube-api-access-n2q5h\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6\" (UID: \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" Oct 31 13:43:24.869748 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.869626 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a899eb-181a-4f40-a64e-2953e94e6dd4-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6\" (UID: \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" Oct 31 13:43:24.869748 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.869719 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0d55fc-97da-4fba-a45a-68a229f87264-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq\" (UID: \"8a0d55fc-97da-4fba-a45a-68a229f87264\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" Oct 31 13:43:24.869815 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.869794 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3a899eb-181a-4f40-a64e-2953e94e6dd4-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6\" (UID: \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" Oct 31 13:43:24.869854 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.869815 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a0d55fc-97da-4fba-a45a-68a229f87264-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq\" (UID: \"8a0d55fc-97da-4fba-a45a-68a229f87264\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" Oct 31 13:43:24.869889 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.869855 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a899eb-181a-4f40-a64e-2953e94e6dd4-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6\" (UID: \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" Oct 31 13:43:24.869956 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.869926 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a899eb-181a-4f40-a64e-2953e94e6dd4-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6\" (UID: \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" Oct 31 13:43:24.870026 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.869976 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a899eb-181a-4f40-a64e-2953e94e6dd4-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6\" (UID: \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" Oct 31 13:43:24.870082 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.870033 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a0d55fc-97da-4fba-a45a-68a229f87264-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq\" (UID: \"8a0d55fc-97da-4fba-a45a-68a229f87264\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" Oct 31 13:43:24.870082 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.870062 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0d55fc-97da-4fba-a45a-68a229f87264-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq\" (UID: \"8a0d55fc-97da-4fba-a45a-68a229f87264\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" Oct 31 13:43:24.870155 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.870089 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a899eb-181a-4f40-a64e-2953e94e6dd4-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6\" (UID: \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" Oct 31 13:43:24.870191 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.870164 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a0d55fc-97da-4fba-a45a-68a229f87264-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq\" (UID: \"8a0d55fc-97da-4fba-a45a-68a229f87264\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" Oct 31 13:43:24.870229 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.870194 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a899eb-181a-4f40-a64e-2953e94e6dd4-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6\" (UID: \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" Oct 31 13:43:24.870269 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.870229 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqq5w\" (UniqueName: \"kubernetes.io/projected/8a0d55fc-97da-4fba-a45a-68a229f87264-kube-api-access-pqq5w\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq\" (UID: \"8a0d55fc-97da-4fba-a45a-68a229f87264\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" Oct 31 13:43:24.870360 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.870341 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a0d55fc-97da-4fba-a45a-68a229f87264-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq\" (UID: \"8a0d55fc-97da-4fba-a45a-68a229f87264\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" Oct 31 13:43:24.971148 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.971120 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0d55fc-97da-4fba-a45a-68a229f87264-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq\" (UID: \"8a0d55fc-97da-4fba-a45a-68a229f87264\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" Oct 31 13:43:24.971319 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.971186 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3a899eb-181a-4f40-a64e-2953e94e6dd4-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6\" (UID: \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" Oct 31 13:43:24.971319 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.971211 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a0d55fc-97da-4fba-a45a-68a229f87264-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq\" (UID: \"8a0d55fc-97da-4fba-a45a-68a229f87264\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" Oct 31 13:43:24.971319 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.971255 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a899eb-181a-4f40-a64e-2953e94e6dd4-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6\" (UID: \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" Oct 31 13:43:24.971319 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.971316 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a899eb-181a-4f40-a64e-2953e94e6dd4-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6\" (UID: \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" Oct 31 13:43:24.971554 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.971339 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a899eb-181a-4f40-a64e-2953e94e6dd4-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6\" (UID: \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" Oct 31 13:43:24.971554 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.971382 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a0d55fc-97da-4fba-a45a-68a229f87264-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq\" (UID: \"8a0d55fc-97da-4fba-a45a-68a229f87264\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" Oct 31 13:43:24.971554 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.971409 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0d55fc-97da-4fba-a45a-68a229f87264-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq\" (UID: \"8a0d55fc-97da-4fba-a45a-68a229f87264\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" Oct 31 13:43:24.971554 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.971433 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a899eb-181a-4f40-a64e-2953e94e6dd4-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6\" (UID: \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" Oct 31 13:43:24.971554 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.971476 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a0d55fc-97da-4fba-a45a-68a229f87264-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq\" (UID: \"8a0d55fc-97da-4fba-a45a-68a229f87264\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" Oct 31 13:43:24.971554 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.971507 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a899eb-181a-4f40-a64e-2953e94e6dd4-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6\" (UID: \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" Oct 31 13:43:24.971554 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.971540 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqq5w\" (UniqueName: \"kubernetes.io/projected/8a0d55fc-97da-4fba-a45a-68a229f87264-kube-api-access-pqq5w\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq\" (UID: \"8a0d55fc-97da-4fba-a45a-68a229f87264\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" Oct 31 13:43:24.971923 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.971572 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a0d55fc-97da-4fba-a45a-68a229f87264-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq\" (UID: \"8a0d55fc-97da-4fba-a45a-68a229f87264\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" Oct 31 13:43:24.971923 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.971592 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0d55fc-97da-4fba-a45a-68a229f87264-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq\" (UID: \"8a0d55fc-97da-4fba-a45a-68a229f87264\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" Oct 31 13:43:24.971923 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.971624 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a899eb-181a-4f40-a64e-2953e94e6dd4-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6\" (UID: \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" Oct 31 13:43:24.971923 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.971657 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a0d55fc-97da-4fba-a45a-68a229f87264-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq\" (UID: \"8a0d55fc-97da-4fba-a45a-68a229f87264\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" Oct 31 13:43:24.971923 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.971685 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a0d55fc-97da-4fba-a45a-68a229f87264-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq\" (UID: \"8a0d55fc-97da-4fba-a45a-68a229f87264\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" Oct 31 13:43:24.971923 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.971723 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2q5h\" (UniqueName: \"kubernetes.io/projected/a3a899eb-181a-4f40-a64e-2953e94e6dd4-kube-api-access-n2q5h\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6\" (UID: \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" Oct 31 13:43:24.971923 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.971760 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a899eb-181a-4f40-a64e-2953e94e6dd4-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6\" (UID: \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" Oct 31 13:43:24.971923 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.971803 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a899eb-181a-4f40-a64e-2953e94e6dd4-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6\" (UID: \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" Oct 31 13:43:24.972347 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.971917 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a899eb-181a-4f40-a64e-2953e94e6dd4-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6\" (UID: \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" Oct 31 13:43:24.973569 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.972781 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a0d55fc-97da-4fba-a45a-68a229f87264-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq\" (UID: \"8a0d55fc-97da-4fba-a45a-68a229f87264\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" Oct 31 13:43:24.973569 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.973048 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a899eb-181a-4f40-a64e-2953e94e6dd4-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6\" (UID: \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" Oct 31 13:43:24.973569 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.973273 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0d55fc-97da-4fba-a45a-68a229f87264-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq\" (UID: \"8a0d55fc-97da-4fba-a45a-68a229f87264\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" Oct 31 13:43:24.973569 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.973333 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a899eb-181a-4f40-a64e-2953e94e6dd4-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6\" (UID: \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" Oct 31 13:43:24.973569 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.973357 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a0d55fc-97da-4fba-a45a-68a229f87264-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq\" (UID: \"8a0d55fc-97da-4fba-a45a-68a229f87264\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" Oct 31 13:43:24.974338 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.974313 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a899eb-181a-4f40-a64e-2953e94e6dd4-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6\" (UID: \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" Oct 31 13:43:24.974622 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.974599 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a0d55fc-97da-4fba-a45a-68a229f87264-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq\" (UID: \"8a0d55fc-97da-4fba-a45a-68a229f87264\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" Oct 31 13:43:24.974708 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.974617 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a0d55fc-97da-4fba-a45a-68a229f87264-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq\" (UID: \"8a0d55fc-97da-4fba-a45a-68a229f87264\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" Oct 31 13:43:24.974708 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.974696 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a899eb-181a-4f40-a64e-2953e94e6dd4-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6\" (UID: \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" Oct 31 13:43:24.974821 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.974766 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a899eb-181a-4f40-a64e-2953e94e6dd4-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6\" (UID: \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" Oct 31 13:43:24.975098 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.975078 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a0d55fc-97da-4fba-a45a-68a229f87264-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq\" (UID: \"8a0d55fc-97da-4fba-a45a-68a229f87264\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" Oct 31 13:43:24.986445 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.986416 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a0d55fc-97da-4fba-a45a-68a229f87264-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq\" (UID: \"8a0d55fc-97da-4fba-a45a-68a229f87264\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" Oct 31 13:43:24.986594 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.986555 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3a899eb-181a-4f40-a64e-2953e94e6dd4-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6\" (UID: \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" Oct 31 13:43:24.986803 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.986785 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqq5w\" (UniqueName: \"kubernetes.io/projected/8a0d55fc-97da-4fba-a45a-68a229f87264-kube-api-access-pqq5w\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq\" (UID: \"8a0d55fc-97da-4fba-a45a-68a229f87264\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" Oct 31 13:43:24.987497 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:24.987476 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2q5h\" (UniqueName: \"kubernetes.io/projected/a3a899eb-181a-4f40-a64e-2953e94e6dd4-kube-api-access-n2q5h\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6\" (UID: \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" Oct 31 13:43:25.018555 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:25.018268 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" Oct 31 13:43:25.059865 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:25.059826 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:25.283092 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:25.283018 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" Oct 31 13:43:25.293993 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:25.293954 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:25.313723 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:25.313693 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6"] Oct 31 13:43:25.317550 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:43:25.317515 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3a899eb_181a_4f40_a64e_2953e94e6dd4.slice/crio-78c54ff5c6b891277666ae814619ef71fbd527c407818a2b6414af95f7378b25 WatchSource:0}: Error finding container 78c54ff5c6b891277666ae814619ef71fbd527c407818a2b6414af95f7378b25: Status 404 returned error can't find the container with id 78c54ff5c6b891277666ae814619ef71fbd527c407818a2b6414af95f7378b25 Oct 31 13:43:25.320555 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:25.320522 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:43:25.320646 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:25.320591 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:43:25.320646 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:25.320620 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:43:25.405052 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:25.405018 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" event={"ID":"a3a899eb-181a-4f40-a64e-2953e94e6dd4","Type":"ContainerStarted","Data":"78c54ff5c6b891277666ae814619ef71fbd527c407818a2b6414af95f7378b25"} Oct 31 13:43:25.573958 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:25.573929 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq"] Oct 31 13:43:25.574519 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:43:25.574492 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a0d55fc_97da_4fba_a45a_68a229f87264.slice/crio-8c33aee7a8e41ab1f976c74a8fceaa4e83fd684fca4d95c8ec193a395d9d0938 WatchSource:0}: Error finding container 8c33aee7a8e41ab1f976c74a8fceaa4e83fd684fca4d95c8ec193a395d9d0938: Status 404 returned error can't find the container with id 8c33aee7a8e41ab1f976c74a8fceaa4e83fd684fca4d95c8ec193a395d9d0938 Oct 31 13:43:25.577216 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:25.577184 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:43:25.577305 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:25.577250 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:43:25.577305 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:25.577280 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:43:25.821028 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:25.820994 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:26.416510 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:26.416473 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" event={"ID":"a3a899eb-181a-4f40-a64e-2953e94e6dd4","Type":"ContainerStarted","Data":"a1f521b043083d6b868fafb2790b1deddbdd895806d73284d6345cf60dd5a311"} Oct 31 13:43:26.418176 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:26.418152 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" event={"ID":"8a0d55fc-97da-4fba-a45a-68a229f87264","Type":"ContainerStarted","Data":"fbc1436d0ef53d0461521ad35e9f37e74877f0f65a7cdda96d39771d15f9b651"} Oct 31 13:43:26.418280 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:26.418194 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" event={"ID":"8a0d55fc-97da-4fba-a45a-68a229f87264","Type":"ContainerStarted","Data":"8c33aee7a8e41ab1f976c74a8fceaa4e83fd684fca4d95c8ec193a395d9d0938"} Oct 31 13:43:26.470019 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:26.468257 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" podStartSLOduration=2.468236589 podStartE2EDuration="2.468236589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:43:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:43:26.467313036 +0000 UTC m=+4252.950435938" watchObservedRunningTime="2026-10-31 13:43:26.468236589 +0000 UTC m=+4252.951359491" Oct 31 13:43:26.521203 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:26.520911 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" podStartSLOduration=2.520886122 podStartE2EDuration="2.520886122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:43:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:43:26.518881469 +0000 UTC m=+4253.002004371" watchObservedRunningTime="2026-10-31 13:43:26.520886122 +0000 UTC m=+4253.004009061" Oct 31 13:43:26.796006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:26.795790 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:27.021212 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:27.021172 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" Oct 31 13:43:27.022505 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:27.022475 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 31 13:43:27.022585 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:27.022546 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" podUID="a3a899eb-181a-4f40-a64e-2953e94e6dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 31 13:43:27.290103 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:27.290059 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" Oct 31 13:43:27.291446 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:27.291422 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 31 13:43:27.291628 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:27.291607 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" podUID="8a0d55fc-97da-4fba-a45a-68a229f87264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 31 13:43:27.667066 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:27.666775 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:28.021482 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:28.021451 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 31 13:43:28.021928 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:28.021511 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" podUID="a3a899eb-181a-4f40-a64e-2953e94e6dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 31 13:43:28.285495 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:28.285397 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 31 13:43:28.285804 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:28.285765 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" podUID="8a0d55fc-97da-4fba-a45a-68a229f87264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 31 13:43:28.530721 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:28.530682 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:29.019755 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:29.019717 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:29.045417 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:29.043417 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 31 13:43:29.045417 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:29.043481 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" podUID="a3a899eb-181a-4f40-a64e-2953e94e6dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 31 13:43:29.275536 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:29.275428 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:29.296357 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:29.296139 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 31 13:43:29.296357 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:29.296202 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" podUID="8a0d55fc-97da-4fba-a45a-68a229f87264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 31 13:43:29.397924 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:43:29.397883 6841 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb6f239c_4a3d_4f98_9b5e_4d775a4c561c.slice/crio-conmon-18de47f6118d9faf6e6dfe92a33834523ed70682bedcb1c91ddb46592c3e11ef.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb6f239c_4a3d_4f98_9b5e_4d775a4c561c.slice/crio-18de47f6118d9faf6e6dfe92a33834523ed70682bedcb1c91ddb46592c3e11ef.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96229665_4486_4750_becf_d30f28c47dd5.slice/crio-d4baf7017ccf84c7ec94e5ee6a988983ff78fc8dab38fc6ef15de6fde9b6d2bb.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96229665_4486_4750_becf_d30f28c47dd5.slice/crio-conmon-d4baf7017ccf84c7ec94e5ee6a988983ff78fc8dab38fc6ef15de6fde9b6d2bb.scope\": RecentStats: unable to find data in memory cache]" Oct 31 13:43:29.450017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:29.445232 6841 generic.go:356] "Generic (PLEG): container finished" podID="96229665-4486-4750-becf-d30f28c47dd5" containerID="d4baf7017ccf84c7ec94e5ee6a988983ff78fc8dab38fc6ef15de6fde9b6d2bb" exitCode=0 Oct 31 13:43:29.450017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:29.445367 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-622e03-55955d5bc9-nfrwx" event={"ID":"96229665-4486-4750-becf-d30f28c47dd5","Type":"ContainerDied","Data":"d4baf7017ccf84c7ec94e5ee6a988983ff78fc8dab38fc6ef15de6fde9b6d2bb"} Oct 31 13:43:29.459017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:29.456999 6841 generic.go:356] "Generic (PLEG): container finished" podID="fb6f239c-4a3d-4f98-9b5e-4d775a4c561c" containerID="18de47f6118d9faf6e6dfe92a33834523ed70682bedcb1c91ddb46592c3e11ef" exitCode=0 Oct 31 13:43:29.459017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:29.457134 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-622e03-7574857797-f4fdg" event={"ID":"fb6f239c-4a3d-4f98-9b5e-4d775a4c561c","Type":"ContainerDied","Data":"18de47f6118d9faf6e6dfe92a33834523ed70682bedcb1c91ddb46592c3e11ef"} Oct 31 13:43:29.496643 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:29.496552 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:29.707541 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:29.707509 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:29.837617 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:29.837583 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-622e03-7574857797-f4fdg" Oct 31 13:43:29.908070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:29.906889 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-622e03-55955d5bc9-nfrwx" Oct 31 13:43:29.995008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:29.994433 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb6f239c-4a3d-4f98-9b5e-4d775a4c561c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb6f239c-4a3d-4f98-9b5e-4d775a4c561c-plugins-config-volume\") pod \"fb6f239c-4a3d-4f98-9b5e-4d775a4c561c\" (UID: \"fb6f239c-4a3d-4f98-9b5e-4d775a4c561c\") " Oct 31 13:43:29.995008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:29.994502 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb6f239c-4a3d-4f98-9b5e-4d775a4c561c-kube-api-access-xktpv\" (UniqueName: \"kubernetes.io/projected/fb6f239c-4a3d-4f98-9b5e-4d775a4c561c-kube-api-access-xktpv\") pod \"fb6f239c-4a3d-4f98-9b5e-4d775a4c561c\" (UID: \"fb6f239c-4a3d-4f98-9b5e-4d775a4c561c\") " Oct 31 13:43:29.995008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:29.994619 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96229665-4486-4750-becf-d30f28c47dd5-kube-api-access-dmtbz\" (UniqueName: \"kubernetes.io/projected/96229665-4486-4750-becf-d30f28c47dd5-kube-api-access-dmtbz\") pod \"96229665-4486-4750-becf-d30f28c47dd5\" (UID: \"96229665-4486-4750-becf-d30f28c47dd5\") " Oct 31 13:43:29.995008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:29.994683 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96229665-4486-4750-becf-d30f28c47dd5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96229665-4486-4750-becf-d30f28c47dd5-plugins-config-volume\") pod \"96229665-4486-4750-becf-d30f28c47dd5\" (UID: \"96229665-4486-4750-becf-d30f28c47dd5\") " Oct 31 13:43:30.036045 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:30.034607 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb6f239c-4a3d-4f98-9b5e-4d775a4c561c-kube-api-access-xktpv" pod "fb6f239c-4a3d-4f98-9b5e-4d775a4c561c" (UID: "fb6f239c-4a3d-4f98-9b5e-4d775a4c561c"). InnerVolumeSpecName "kube-api-access-xktpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:30.036045 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:30.035061 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb6f239c-4a3d-4f98-9b5e-4d775a4c561c-plugins-config-volume" pod "fb6f239c-4a3d-4f98-9b5e-4d775a4c561c" (UID: "fb6f239c-4a3d-4f98-9b5e-4d775a4c561c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:43:30.052802 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:30.051290 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 31 13:43:30.052802 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:30.051350 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" podUID="a3a899eb-181a-4f40-a64e-2953e94e6dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 31 13:43:30.061695 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:30.061200 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96229665-4486-4750-becf-d30f28c47dd5-plugins-config-volume" pod "96229665-4486-4750-becf-d30f28c47dd5" (UID: "96229665-4486-4750-becf-d30f28c47dd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:43:30.092857 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:30.092815 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:30.093085 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:30.093039 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96229665-4486-4750-becf-d30f28c47dd5-kube-api-access-dmtbz" pod "96229665-4486-4750-becf-d30f28c47dd5" (UID: "96229665-4486-4750-becf-d30f28c47dd5"). InnerVolumeSpecName "kube-api-access-dmtbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:30.104029 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:30.103751 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmtbz\" (UniqueName: \"kubernetes.io/projected/96229665-4486-4750-becf-d30f28c47dd5-kube-api-access-dmtbz\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:30.104029 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:30.103779 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96229665-4486-4750-becf-d30f28c47dd5-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:30.104029 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:30.103796 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb6f239c-4a3d-4f98-9b5e-4d775a4c561c-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:30.104029 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:30.103811 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xktpv\" (UniqueName: \"kubernetes.io/projected/fb6f239c-4a3d-4f98-9b5e-4d775a4c561c-kube-api-access-xktpv\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:30.284505 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:30.284213 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 31 13:43:30.284505 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:30.284266 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" podUID="8a0d55fc-97da-4fba-a45a-68a229f87264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 31 13:43:30.487005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:30.486209 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-622e03-7574857797-f4fdg" event={"ID":"fb6f239c-4a3d-4f98-9b5e-4d775a4c561c","Type":"ContainerDied","Data":"316527ca7f4739546addbf4f9485ae462f44537b9433ee496261650d98a8cd03"} Oct 31 13:43:30.487005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:30.486282 6841 scope.go:122] "RemoveContainer" containerID="18de47f6118d9faf6e6dfe92a33834523ed70682bedcb1c91ddb46592c3e11ef" Oct 31 13:43:30.487005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:30.486427 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-622e03-7574857797-f4fdg" Oct 31 13:43:30.494399 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:30.494366 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-622e03-55955d5bc9-nfrwx" event={"ID":"96229665-4486-4750-becf-d30f28c47dd5","Type":"ContainerDied","Data":"1213b6c878fa186eb454338ff986e19fec610adccda6160969b5cb1041d3d584"} Oct 31 13:43:30.494556 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:30.494474 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-622e03-55955d5bc9-nfrwx" Oct 31 13:43:30.510213 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:30.510191 6841 scope.go:122] "RemoveContainer" containerID="d4baf7017ccf84c7ec94e5ee6a988983ff78fc8dab38fc6ef15de6fde9b6d2bb" Oct 31 13:43:30.520232 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:30.520208 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-622e03-7574857797-f4fdg"] Oct 31 13:43:30.529487 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:30.529463 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-622e03-7574857797-f4fdg"] Oct 31 13:43:30.586147 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:30.586037 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-622e03-55955d5bc9-nfrwx"] Oct 31 13:43:30.589304 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:30.589273 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-622e03-55955d5bc9-nfrwx"] Oct 31 13:43:30.624270 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:30.624224 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:31.020703 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:31.020671 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 31 13:43:31.020897 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:31.020725 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" podUID="a3a899eb-181a-4f40-a64e-2953e94e6dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 31 13:43:31.286063 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:31.284403 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 31 13:43:31.286063 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:31.284466 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" podUID="8a0d55fc-97da-4fba-a45a-68a229f87264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 31 13:43:31.656506 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:31.652674 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96229665-4486-4750-becf-d30f28c47dd5" path="/var/lib/kubelet/pods/96229665-4486-4750-becf-d30f28c47dd5/volumes" Oct 31 13:43:31.656506 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:31.653343 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb6f239c-4a3d-4f98-9b5e-4d775a4c561c" path="/var/lib/kubelet/pods/fb6f239c-4a3d-4f98-9b5e-4d775a4c561c/volumes" Oct 31 13:43:31.676093 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:31.676057 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:32.022144 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:32.022103 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 31 13:43:32.022356 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:32.022195 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" podUID="a3a899eb-181a-4f40-a64e-2953e94e6dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 31 13:43:32.283860 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:32.283771 6841 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 31 13:43:32.283860 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:32.283837 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" podUID="8a0d55fc-97da-4fba-a45a-68a229f87264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 31 13:43:32.637889 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:32.637794 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:33.026132 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:33.026089 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" Oct 31 13:43:33.026475 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:33.026452 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" Oct 31 13:43:33.027223 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:33.027188 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" Oct 31 13:43:33.288218 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:33.288127 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" Oct 31 13:43:33.288437 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:33.288408 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" Oct 31 13:43:33.289288 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:33.289266 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" Oct 31 13:43:33.339121 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:33.339093 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:33.624808 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:33.624717 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:33.978201 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:33.978165 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:34.270302 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:34.270216 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht"] Oct 31 13:43:34.270505 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:34.270440 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" podUID="cfd61135-1d8a-44a5-9771-a66f1c88cd18" containerName="istio-proxy" containerID="cri-o://a82cf6f5f0155063fd87d7d9d075dc4a06b0e77b0b499a8edd985e802abeac57" gracePeriod=30 Oct 31 13:43:34.336003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:34.335872 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:34.600753 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:34.600665 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:34.956864 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:34.956814 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:35.263915 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:35.263825 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:35.712135 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:35.712101 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:36.343276 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:36.343241 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:36.704123 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:36.704091 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:36.989082 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:36.988495 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq"] Oct 31 13:43:36.989082 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:36.988804 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" podUID="8a0d55fc-97da-4fba-a45a-68a229f87264" containerName="istio-proxy" containerID="cri-o://fbc1436d0ef53d0461521ad35e9f37e74877f0f65a7cdda96d39771d15f9b651" gracePeriod=30 Oct 31 13:43:36.995433 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:36.995402 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6"] Oct 31 13:43:36.995856 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:36.995823 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" podUID="a3a899eb-181a-4f40-a64e-2953e94e6dd4" containerName="istio-proxy" containerID="cri-o://a1f521b043083d6b868fafb2790b1deddbdd895806d73284d6345cf60dd5a311" gracePeriod=30 Oct 31 13:43:37.106795 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:37.106756 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:37.915587 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:37.915550 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:39.354617 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.354585 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:39.533692 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.533668 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:43:39.560386 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.560325 6841 generic.go:356] "Generic (PLEG): container finished" podID="cfd61135-1d8a-44a5-9771-a66f1c88cd18" containerID="a82cf6f5f0155063fd87d7d9d075dc4a06b0e77b0b499a8edd985e802abeac57" exitCode=0 Oct 31 13:43:39.560386 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.560370 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" Oct 31 13:43:39.560526 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.560405 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" event={"ID":"cfd61135-1d8a-44a5-9771-a66f1c88cd18","Type":"ContainerDied","Data":"a82cf6f5f0155063fd87d7d9d075dc4a06b0e77b0b499a8edd985e802abeac57"} Oct 31 13:43:39.560526 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.560457 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht" event={"ID":"cfd61135-1d8a-44a5-9771-a66f1c88cd18","Type":"ContainerDied","Data":"f828a7191bcae5112272097267dc1365d8d8ec188c4a0fa342353e5bf7da663e"} Oct 31 13:43:39.560526 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.560485 6841 scope.go:122] "RemoveContainer" containerID="a82cf6f5f0155063fd87d7d9d075dc4a06b0e77b0b499a8edd985e802abeac57" Oct 31 13:43:39.570467 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.570451 6841 scope.go:122] "RemoveContainer" containerID="9a1b6eabb4d08a835841e53610bafb1356c9c10fc97dc39635d66a00bd40bf90" Oct 31 13:43:39.580647 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.580621 6841 scope.go:122] "RemoveContainer" containerID="a82cf6f5f0155063fd87d7d9d075dc4a06b0e77b0b499a8edd985e802abeac57" Oct 31 13:43:39.581042 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:43:39.580970 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a82cf6f5f0155063fd87d7d9d075dc4a06b0e77b0b499a8edd985e802abeac57\": container with ID starting with a82cf6f5f0155063fd87d7d9d075dc4a06b0e77b0b499a8edd985e802abeac57 not found: ID does not exist" containerID="a82cf6f5f0155063fd87d7d9d075dc4a06b0e77b0b499a8edd985e802abeac57" Oct 31 13:43:39.581188 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.581057 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a82cf6f5f0155063fd87d7d9d075dc4a06b0e77b0b499a8edd985e802abeac57"} err="failed to get container status \"a82cf6f5f0155063fd87d7d9d075dc4a06b0e77b0b499a8edd985e802abeac57\": rpc error: code = NotFound desc = could not find container \"a82cf6f5f0155063fd87d7d9d075dc4a06b0e77b0b499a8edd985e802abeac57\": container with ID starting with a82cf6f5f0155063fd87d7d9d075dc4a06b0e77b0b499a8edd985e802abeac57 not found: ID does not exist" Oct 31 13:43:39.581188 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.581081 6841 scope.go:122] "RemoveContainer" containerID="9a1b6eabb4d08a835841e53610bafb1356c9c10fc97dc39635d66a00bd40bf90" Oct 31 13:43:39.581406 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:43:39.581380 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a1b6eabb4d08a835841e53610bafb1356c9c10fc97dc39635d66a00bd40bf90\": container with ID starting with 9a1b6eabb4d08a835841e53610bafb1356c9c10fc97dc39635d66a00bd40bf90 not found: ID does not exist" containerID="9a1b6eabb4d08a835841e53610bafb1356c9c10fc97dc39635d66a00bd40bf90" Oct 31 13:43:39.581458 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.581417 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a1b6eabb4d08a835841e53610bafb1356c9c10fc97dc39635d66a00bd40bf90"} err="failed to get container status \"9a1b6eabb4d08a835841e53610bafb1356c9c10fc97dc39635d66a00bd40bf90\": rpc error: code = NotFound desc = could not find container \"9a1b6eabb4d08a835841e53610bafb1356c9c10fc97dc39635d66a00bd40bf90\": container with ID starting with 9a1b6eabb4d08a835841e53610bafb1356c9c10fc97dc39635d66a00bd40bf90 not found: ID does not exist" Oct 31 13:43:39.694379 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.694349 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cfd61135-1d8a-44a5-9771-a66f1c88cd18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfd61135-1d8a-44a5-9771-a66f1c88cd18-istio-podinfo\") pod \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\" (UID: \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\") " Oct 31 13:43:39.694565 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.694392 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd61135-1d8a-44a5-9771-a66f1c88cd18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfd61135-1d8a-44a5-9771-a66f1c88cd18-workload-certs\") pod \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\" (UID: \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\") " Oct 31 13:43:39.694565 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.694421 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd61135-1d8a-44a5-9771-a66f1c88cd18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd61135-1d8a-44a5-9771-a66f1c88cd18-credential-socket\") pod \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\" (UID: \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\") " Oct 31 13:43:39.694565 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.694458 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfd61135-1d8a-44a5-9771-a66f1c88cd18-istio-token\" (UniqueName: \"kubernetes.io/projected/cfd61135-1d8a-44a5-9771-a66f1c88cd18-istio-token\") pod \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\" (UID: \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\") " Oct 31 13:43:39.694565 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.694497 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd61135-1d8a-44a5-9771-a66f1c88cd18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfd61135-1d8a-44a5-9771-a66f1c88cd18-istio-envoy\") pod \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\" (UID: \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\") " Oct 31 13:43:39.694786 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.694591 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfd61135-1d8a-44a5-9771-a66f1c88cd18-kube-api-access-jb2z4\" (UniqueName: \"kubernetes.io/projected/cfd61135-1d8a-44a5-9771-a66f1c88cd18-kube-api-access-jb2z4\") pod \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\" (UID: \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\") " Oct 31 13:43:39.694786 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.694657 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd61135-1d8a-44a5-9771-a66f1c88cd18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd61135-1d8a-44a5-9771-a66f1c88cd18-workload-socket\") pod \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\" (UID: \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\") " Oct 31 13:43:39.694786 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.694708 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd61135-1d8a-44a5-9771-a66f1c88cd18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfd61135-1d8a-44a5-9771-a66f1c88cd18-istio-data\") pod \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\" (UID: \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\") " Oct 31 13:43:39.694786 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.694724 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd61135-1d8a-44a5-9771-a66f1c88cd18-credential-socket" pod "cfd61135-1d8a-44a5-9771-a66f1c88cd18" (UID: "cfd61135-1d8a-44a5-9771-a66f1c88cd18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:39.694786 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.694670 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd61135-1d8a-44a5-9771-a66f1c88cd18-workload-certs" pod "cfd61135-1d8a-44a5-9771-a66f1c88cd18" (UID: "cfd61135-1d8a-44a5-9771-a66f1c88cd18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:39.695080 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.694792 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfd61135-1d8a-44a5-9771-a66f1c88cd18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfd61135-1d8a-44a5-9771-a66f1c88cd18-istiod-ca-cert\") pod \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\" (UID: \"cfd61135-1d8a-44a5-9771-a66f1c88cd18\") " Oct 31 13:43:39.695080 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.694975 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd61135-1d8a-44a5-9771-a66f1c88cd18-istio-data" pod "cfd61135-1d8a-44a5-9771-a66f1c88cd18" (UID: "cfd61135-1d8a-44a5-9771-a66f1c88cd18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:39.695080 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.695010 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd61135-1d8a-44a5-9771-a66f1c88cd18-workload-socket" pod "cfd61135-1d8a-44a5-9771-a66f1c88cd18" (UID: "cfd61135-1d8a-44a5-9771-a66f1c88cd18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:39.695376 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.695349 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfd61135-1d8a-44a5-9771-a66f1c88cd18-istiod-ca-cert" pod "cfd61135-1d8a-44a5-9771-a66f1c88cd18" (UID: "cfd61135-1d8a-44a5-9771-a66f1c88cd18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:43:39.695499 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.695483 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd61135-1d8a-44a5-9771-a66f1c88cd18-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:39.695555 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.695505 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfd61135-1d8a-44a5-9771-a66f1c88cd18-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:39.695555 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.695517 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfd61135-1d8a-44a5-9771-a66f1c88cd18-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:39.695555 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.695531 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd61135-1d8a-44a5-9771-a66f1c88cd18-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:39.697199 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.697171 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfd61135-1d8a-44a5-9771-a66f1c88cd18-istio-podinfo" pod "cfd61135-1d8a-44a5-9771-a66f1c88cd18" (UID: "cfd61135-1d8a-44a5-9771-a66f1c88cd18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:43:39.697314 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.697247 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd61135-1d8a-44a5-9771-a66f1c88cd18-istio-envoy" pod "cfd61135-1d8a-44a5-9771-a66f1c88cd18" (UID: "cfd61135-1d8a-44a5-9771-a66f1c88cd18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:39.697387 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.697313 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfd61135-1d8a-44a5-9771-a66f1c88cd18-istio-token" pod "cfd61135-1d8a-44a5-9771-a66f1c88cd18" (UID: "cfd61135-1d8a-44a5-9771-a66f1c88cd18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:39.697387 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.697346 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfd61135-1d8a-44a5-9771-a66f1c88cd18-kube-api-access-jb2z4" pod "cfd61135-1d8a-44a5-9771-a66f1c88cd18" (UID: "cfd61135-1d8a-44a5-9771-a66f1c88cd18"). InnerVolumeSpecName "kube-api-access-jb2z4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:39.797017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.796965 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfd61135-1d8a-44a5-9771-a66f1c88cd18-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:39.797017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.797010 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfd61135-1d8a-44a5-9771-a66f1c88cd18-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:39.797017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.797026 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfd61135-1d8a-44a5-9771-a66f1c88cd18-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:39.797338 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.797035 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfd61135-1d8a-44a5-9771-a66f1c88cd18-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:39.797338 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.797047 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb2z4\" (UniqueName: \"kubernetes.io/projected/cfd61135-1d8a-44a5-9771-a66f1c88cd18-kube-api-access-jb2z4\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:39.888352 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.888324 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht"] Oct 31 13:43:39.909667 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:39.909635 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-fpxht"] Oct 31 13:43:40.380616 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:40.379437 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:40.613095 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:40.613055 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:41.041473 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:41.041441 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:41.549824 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:41.549787 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:41.642518 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:41.642488 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfd61135-1d8a-44a5-9771-a66f1c88cd18" path="/var/lib/kubelet/pods/cfd61135-1d8a-44a5-9771-a66f1c88cd18/volumes" Oct 31 13:43:42.164449 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.164404 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-45bcad-6f4486b69f-qtdsk"] Oct 31 13:43:42.348135 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.348106 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:42.582049 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.582009 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:42.584788 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.584760 6841 generic.go:356] "Generic (PLEG): container finished" podID="a3a899eb-181a-4f40-a64e-2953e94e6dd4" containerID="a1f521b043083d6b868fafb2790b1deddbdd895806d73284d6345cf60dd5a311" exitCode=0 Oct 31 13:43:42.585009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.584866 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" event={"ID":"a3a899eb-181a-4f40-a64e-2953e94e6dd4","Type":"ContainerDied","Data":"a1f521b043083d6b868fafb2790b1deddbdd895806d73284d6345cf60dd5a311"} Oct 31 13:43:42.587456 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.587052 6841 generic.go:356] "Generic (PLEG): container finished" podID="8a0d55fc-97da-4fba-a45a-68a229f87264" containerID="fbc1436d0ef53d0461521ad35e9f37e74877f0f65a7cdda96d39771d15f9b651" exitCode=0 Oct 31 13:43:42.587456 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.587126 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" event={"ID":"8a0d55fc-97da-4fba-a45a-68a229f87264","Type":"ContainerDied","Data":"fbc1436d0ef53d0461521ad35e9f37e74877f0f65a7cdda96d39771d15f9b651"} Oct 31 13:43:42.724249 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.724220 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" Oct 31 13:43:42.737854 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.736957 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a899eb-181a-4f40-a64e-2953e94e6dd4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a899eb-181a-4f40-a64e-2953e94e6dd4-workload-socket\") pod \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\" (UID: \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\") " Oct 31 13:43:42.737854 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.737024 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3a899eb-181a-4f40-a64e-2953e94e6dd4-kube-api-access-n2q5h\" (UniqueName: \"kubernetes.io/projected/a3a899eb-181a-4f40-a64e-2953e94e6dd4-kube-api-access-n2q5h\") pod \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\" (UID: \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\") " Oct 31 13:43:42.737854 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.737059 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a899eb-181a-4f40-a64e-2953e94e6dd4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a899eb-181a-4f40-a64e-2953e94e6dd4-istio-envoy\") pod \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\" (UID: \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\") " Oct 31 13:43:42.737854 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.737116 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3a899eb-181a-4f40-a64e-2953e94e6dd4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a899eb-181a-4f40-a64e-2953e94e6dd4-istiod-ca-cert\") pod \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\" (UID: \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\") " Oct 31 13:43:42.737854 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.737164 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3a899eb-181a-4f40-a64e-2953e94e6dd4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a899eb-181a-4f40-a64e-2953e94e6dd4-istio-podinfo\") pod \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\" (UID: \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\") " Oct 31 13:43:42.737854 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.737199 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3a899eb-181a-4f40-a64e-2953e94e6dd4-istio-token\" (UniqueName: \"kubernetes.io/projected/a3a899eb-181a-4f40-a64e-2953e94e6dd4-istio-token\") pod \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\" (UID: \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\") " Oct 31 13:43:42.737854 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.737281 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a899eb-181a-4f40-a64e-2953e94e6dd4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a899eb-181a-4f40-a64e-2953e94e6dd4-credential-socket\") pod \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\" (UID: \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\") " Oct 31 13:43:42.737854 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.737356 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a899eb-181a-4f40-a64e-2953e94e6dd4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a899eb-181a-4f40-a64e-2953e94e6dd4-istio-data\") pod \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\" (UID: \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\") " Oct 31 13:43:42.737854 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.737397 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a899eb-181a-4f40-a64e-2953e94e6dd4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a899eb-181a-4f40-a64e-2953e94e6dd4-workload-certs\") pod \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\" (UID: \"a3a899eb-181a-4f40-a64e-2953e94e6dd4\") " Oct 31 13:43:42.737854 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.737746 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3a899eb-181a-4f40-a64e-2953e94e6dd4-istiod-ca-cert" pod "a3a899eb-181a-4f40-a64e-2953e94e6dd4" (UID: "a3a899eb-181a-4f40-a64e-2953e94e6dd4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:43:42.738523 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.737970 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a899eb-181a-4f40-a64e-2953e94e6dd4-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:42.738523 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.738023 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a899eb-181a-4f40-a64e-2953e94e6dd4-workload-socket" pod "a3a899eb-181a-4f40-a64e-2953e94e6dd4" (UID: "a3a899eb-181a-4f40-a64e-2953e94e6dd4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:42.738641 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.738530 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a899eb-181a-4f40-a64e-2953e94e6dd4-workload-certs" pod "a3a899eb-181a-4f40-a64e-2953e94e6dd4" (UID: "a3a899eb-181a-4f40-a64e-2953e94e6dd4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:42.740023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.739801 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a899eb-181a-4f40-a64e-2953e94e6dd4-credential-socket" pod "a3a899eb-181a-4f40-a64e-2953e94e6dd4" (UID: "a3a899eb-181a-4f40-a64e-2953e94e6dd4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:42.740179 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.740154 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a899eb-181a-4f40-a64e-2953e94e6dd4-istio-data" pod "a3a899eb-181a-4f40-a64e-2953e94e6dd4" (UID: "a3a899eb-181a-4f40-a64e-2953e94e6dd4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:42.741315 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.741223 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a899eb-181a-4f40-a64e-2953e94e6dd4-kube-api-access-n2q5h" pod "a3a899eb-181a-4f40-a64e-2953e94e6dd4" (UID: "a3a899eb-181a-4f40-a64e-2953e94e6dd4"). InnerVolumeSpecName "kube-api-access-n2q5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:42.741315 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.741294 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a899eb-181a-4f40-a64e-2953e94e6dd4-istio-token" pod "a3a899eb-181a-4f40-a64e-2953e94e6dd4" (UID: "a3a899eb-181a-4f40-a64e-2953e94e6dd4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:42.743211 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.743188 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a899eb-181a-4f40-a64e-2953e94e6dd4-istio-envoy" pod "a3a899eb-181a-4f40-a64e-2953e94e6dd4" (UID: "a3a899eb-181a-4f40-a64e-2953e94e6dd4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:42.743326 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.743295 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3a899eb-181a-4f40-a64e-2953e94e6dd4-istio-podinfo" pod "a3a899eb-181a-4f40-a64e-2953e94e6dd4" (UID: "a3a899eb-181a-4f40-a64e-2953e94e6dd4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:43:42.773387 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.773085 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" Oct 31 13:43:42.839077 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.839037 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a0d55fc-97da-4fba-a45a-68a229f87264-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a0d55fc-97da-4fba-a45a-68a229f87264-istio-envoy\") pod \"8a0d55fc-97da-4fba-a45a-68a229f87264\" (UID: \"8a0d55fc-97da-4fba-a45a-68a229f87264\") " Oct 31 13:43:42.839276 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.839104 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a0d55fc-97da-4fba-a45a-68a229f87264-istio-token\" (UniqueName: \"kubernetes.io/projected/8a0d55fc-97da-4fba-a45a-68a229f87264-istio-token\") pod \"8a0d55fc-97da-4fba-a45a-68a229f87264\" (UID: \"8a0d55fc-97da-4fba-a45a-68a229f87264\") " Oct 31 13:43:42.839276 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.839150 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a0d55fc-97da-4fba-a45a-68a229f87264-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0d55fc-97da-4fba-a45a-68a229f87264-workload-socket\") pod \"8a0d55fc-97da-4fba-a45a-68a229f87264\" (UID: \"8a0d55fc-97da-4fba-a45a-68a229f87264\") " Oct 31 13:43:42.839276 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.839179 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a0d55fc-97da-4fba-a45a-68a229f87264-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a0d55fc-97da-4fba-a45a-68a229f87264-istiod-ca-cert\") pod \"8a0d55fc-97da-4fba-a45a-68a229f87264\" (UID: \"8a0d55fc-97da-4fba-a45a-68a229f87264\") " Oct 31 13:43:42.839276 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.839217 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a0d55fc-97da-4fba-a45a-68a229f87264-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0d55fc-97da-4fba-a45a-68a229f87264-credential-socket\") pod \"8a0d55fc-97da-4fba-a45a-68a229f87264\" (UID: \"8a0d55fc-97da-4fba-a45a-68a229f87264\") " Oct 31 13:43:42.839456 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.839280 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a0d55fc-97da-4fba-a45a-68a229f87264-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a0d55fc-97da-4fba-a45a-68a229f87264-istio-data\") pod \"8a0d55fc-97da-4fba-a45a-68a229f87264\" (UID: \"8a0d55fc-97da-4fba-a45a-68a229f87264\") " Oct 31 13:43:42.839456 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.839338 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a0d55fc-97da-4fba-a45a-68a229f87264-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a0d55fc-97da-4fba-a45a-68a229f87264-workload-certs\") pod \"8a0d55fc-97da-4fba-a45a-68a229f87264\" (UID: \"8a0d55fc-97da-4fba-a45a-68a229f87264\") " Oct 31 13:43:42.839456 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.839361 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a0d55fc-97da-4fba-a45a-68a229f87264-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a0d55fc-97da-4fba-a45a-68a229f87264-istio-podinfo\") pod \"8a0d55fc-97da-4fba-a45a-68a229f87264\" (UID: \"8a0d55fc-97da-4fba-a45a-68a229f87264\") " Oct 31 13:43:42.839456 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.839385 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a0d55fc-97da-4fba-a45a-68a229f87264-kube-api-access-pqq5w\" (UniqueName: \"kubernetes.io/projected/8a0d55fc-97da-4fba-a45a-68a229f87264-kube-api-access-pqq5w\") pod \"8a0d55fc-97da-4fba-a45a-68a229f87264\" (UID: \"8a0d55fc-97da-4fba-a45a-68a229f87264\") " Oct 31 13:43:42.839738 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.839722 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a899eb-181a-4f40-a64e-2953e94e6dd4-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:42.839789 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.839741 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2q5h\" (UniqueName: \"kubernetes.io/projected/a3a899eb-181a-4f40-a64e-2953e94e6dd4-kube-api-access-n2q5h\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:42.839789 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.839751 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a899eb-181a-4f40-a64e-2953e94e6dd4-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:42.839789 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.839760 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a899eb-181a-4f40-a64e-2953e94e6dd4-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:42.839789 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.839769 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3a899eb-181a-4f40-a64e-2953e94e6dd4-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:42.839789 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.839777 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a899eb-181a-4f40-a64e-2953e94e6dd4-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:42.839789 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.839785 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a899eb-181a-4f40-a64e-2953e94e6dd4-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:42.839977 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.839794 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a899eb-181a-4f40-a64e-2953e94e6dd4-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:42.841277 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.841181 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0d55fc-97da-4fba-a45a-68a229f87264-istio-envoy" pod "8a0d55fc-97da-4fba-a45a-68a229f87264" (UID: "8a0d55fc-97da-4fba-a45a-68a229f87264"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:42.841277 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.841245 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0d55fc-97da-4fba-a45a-68a229f87264-workload-socket" pod "8a0d55fc-97da-4fba-a45a-68a229f87264" (UID: "8a0d55fc-97da-4fba-a45a-68a229f87264"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:42.841420 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.841313 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0d55fc-97da-4fba-a45a-68a229f87264-istio-data" pod "8a0d55fc-97da-4fba-a45a-68a229f87264" (UID: "8a0d55fc-97da-4fba-a45a-68a229f87264"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:42.841481 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.841423 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a0d55fc-97da-4fba-a45a-68a229f87264-istiod-ca-cert" pod "8a0d55fc-97da-4fba-a45a-68a229f87264" (UID: "8a0d55fc-97da-4fba-a45a-68a229f87264"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:43:42.841663 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.841637 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0d55fc-97da-4fba-a45a-68a229f87264-workload-certs" pod "8a0d55fc-97da-4fba-a45a-68a229f87264" (UID: "8a0d55fc-97da-4fba-a45a-68a229f87264"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:42.841864 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.841841 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0d55fc-97da-4fba-a45a-68a229f87264-credential-socket" pod "8a0d55fc-97da-4fba-a45a-68a229f87264" (UID: "8a0d55fc-97da-4fba-a45a-68a229f87264"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:42.843183 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.843160 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a0d55fc-97da-4fba-a45a-68a229f87264-istio-token" pod "8a0d55fc-97da-4fba-a45a-68a229f87264" (UID: "8a0d55fc-97da-4fba-a45a-68a229f87264"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:42.843354 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.843340 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a0d55fc-97da-4fba-a45a-68a229f87264-kube-api-access-pqq5w" pod "8a0d55fc-97da-4fba-a45a-68a229f87264" (UID: "8a0d55fc-97da-4fba-a45a-68a229f87264"). InnerVolumeSpecName "kube-api-access-pqq5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:42.843399 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.843379 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a0d55fc-97da-4fba-a45a-68a229f87264-istio-podinfo" pod "8a0d55fc-97da-4fba-a45a-68a229f87264" (UID: "8a0d55fc-97da-4fba-a45a-68a229f87264"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:43:42.941512 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.941483 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0d55fc-97da-4fba-a45a-68a229f87264-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:42.941512 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.941510 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a0d55fc-97da-4fba-a45a-68a229f87264-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:42.941512 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.941520 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0d55fc-97da-4fba-a45a-68a229f87264-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:42.941753 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.941529 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a0d55fc-97da-4fba-a45a-68a229f87264-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:42.941753 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.941538 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a0d55fc-97da-4fba-a45a-68a229f87264-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:42.941753 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.941546 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a0d55fc-97da-4fba-a45a-68a229f87264-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:42.941753 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.941555 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqq5w\" (UniqueName: \"kubernetes.io/projected/8a0d55fc-97da-4fba-a45a-68a229f87264-kube-api-access-pqq5w\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:42.941753 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.941564 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a0d55fc-97da-4fba-a45a-68a229f87264-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:42.941753 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:42.941573 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a0d55fc-97da-4fba-a45a-68a229f87264-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:43.218285 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.218251 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:43.254797 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.254762 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc"] Oct 31 13:43:43.255008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.254968 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc" podUID="aa093405-83f7-4d3e-b984-bb188e5171d7" containerName="maas-api" containerID="cri-o://f79fb0a090f0b78c1b70fd48f5bc621166ea07367abf3b293858afb4a714097b" gracePeriod=30 Oct 31 13:43:43.461572 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.461536 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:43.607012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.605721 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc" Oct 31 13:43:43.613166 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.613139 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" Oct 31 13:43:43.613289 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.613162 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6" event={"ID":"a3a899eb-181a-4f40-a64e-2953e94e6dd4","Type":"ContainerDied","Data":"78c54ff5c6b891277666ae814619ef71fbd527c407818a2b6414af95f7378b25"} Oct 31 13:43:43.613289 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.613230 6841 scope.go:122] "RemoveContainer" containerID="a1f521b043083d6b868fafb2790b1deddbdd895806d73284d6345cf60dd5a311" Oct 31 13:43:43.614711 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.614687 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" Oct 31 13:43:43.614794 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.614719 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq" event={"ID":"8a0d55fc-97da-4fba-a45a-68a229f87264","Type":"ContainerDied","Data":"8c33aee7a8e41ab1f976c74a8fceaa4e83fd684fca4d95c8ec193a395d9d0938"} Oct 31 13:43:43.616424 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.616401 6841 generic.go:356] "Generic (PLEG): container finished" podID="aa093405-83f7-4d3e-b984-bb188e5171d7" containerID="f79fb0a090f0b78c1b70fd48f5bc621166ea07367abf3b293858afb4a714097b" exitCode=0 Oct 31 13:43:43.616520 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.616430 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc" event={"ID":"aa093405-83f7-4d3e-b984-bb188e5171d7","Type":"ContainerDied","Data":"f79fb0a090f0b78c1b70fd48f5bc621166ea07367abf3b293858afb4a714097b"} Oct 31 13:43:43.616520 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.616463 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc" Oct 31 13:43:43.616621 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.616473 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc" event={"ID":"aa093405-83f7-4d3e-b984-bb188e5171d7","Type":"ContainerDied","Data":"7c32c1cd30a5eeacdedd803583bf4ed99e293f18966496116415467de6c076a1"} Oct 31 13:43:43.625680 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.625661 6841 scope.go:122] "RemoveContainer" containerID="fbc1436d0ef53d0461521ad35e9f37e74877f0f65a7cdda96d39771d15f9b651" Oct 31 13:43:43.635808 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.635601 6841 scope.go:122] "RemoveContainer" containerID="f79fb0a090f0b78c1b70fd48f5bc621166ea07367abf3b293858afb4a714097b" Oct 31 13:43:43.646913 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.646888 6841 scope.go:122] "RemoveContainer" containerID="f79fb0a090f0b78c1b70fd48f5bc621166ea07367abf3b293858afb4a714097b" Oct 31 13:43:43.647229 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:43:43.647207 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f79fb0a090f0b78c1b70fd48f5bc621166ea07367abf3b293858afb4a714097b\": container with ID starting with f79fb0a090f0b78c1b70fd48f5bc621166ea07367abf3b293858afb4a714097b not found: ID does not exist" containerID="f79fb0a090f0b78c1b70fd48f5bc621166ea07367abf3b293858afb4a714097b" Oct 31 13:43:43.647307 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.647239 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f79fb0a090f0b78c1b70fd48f5bc621166ea07367abf3b293858afb4a714097b"} err="failed to get container status \"f79fb0a090f0b78c1b70fd48f5bc621166ea07367abf3b293858afb4a714097b\": rpc error: code = NotFound desc = could not find container \"f79fb0a090f0b78c1b70fd48f5bc621166ea07367abf3b293858afb4a714097b\": container with ID starting with f79fb0a090f0b78c1b70fd48f5bc621166ea07367abf3b293858afb4a714097b not found: ID does not exist" Oct 31 13:43:43.666900 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.666872 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq"] Oct 31 13:43:43.675084 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.675018 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-cwhmq"] Oct 31 13:43:43.697344 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.697321 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6"] Oct 31 13:43:43.711100 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.711077 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-gsrh6"] Oct 31 13:43:43.752572 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.752544 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa093405-83f7-4d3e-b984-bb188e5171d7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa093405-83f7-4d3e-b984-bb188e5171d7-metrics-certs\") pod \"aa093405-83f7-4d3e-b984-bb188e5171d7\" (UID: \"aa093405-83f7-4d3e-b984-bb188e5171d7\") " Oct 31 13:43:43.752691 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.752632 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa093405-83f7-4d3e-b984-bb188e5171d7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa093405-83f7-4d3e-b984-bb188e5171d7-openshift-service-ca\") pod \"aa093405-83f7-4d3e-b984-bb188e5171d7\" (UID: \"aa093405-83f7-4d3e-b984-bb188e5171d7\") " Oct 31 13:43:43.752691 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.752661 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa093405-83f7-4d3e-b984-bb188e5171d7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa093405-83f7-4d3e-b984-bb188e5171d7-maas-api-tls\") pod \"aa093405-83f7-4d3e-b984-bb188e5171d7\" (UID: \"aa093405-83f7-4d3e-b984-bb188e5171d7\") " Oct 31 13:43:43.752779 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.752694 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa093405-83f7-4d3e-b984-bb188e5171d7-kube-api-access-5khnl\" (UniqueName: \"kubernetes.io/projected/aa093405-83f7-4d3e-b984-bb188e5171d7-kube-api-access-5khnl\") pod \"aa093405-83f7-4d3e-b984-bb188e5171d7\" (UID: \"aa093405-83f7-4d3e-b984-bb188e5171d7\") " Oct 31 13:43:43.753060 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.753032 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa093405-83f7-4d3e-b984-bb188e5171d7-openshift-service-ca" pod "aa093405-83f7-4d3e-b984-bb188e5171d7" (UID: "aa093405-83f7-4d3e-b984-bb188e5171d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:43:43.754907 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.754887 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa093405-83f7-4d3e-b984-bb188e5171d7-maas-api-tls" pod "aa093405-83f7-4d3e-b984-bb188e5171d7" (UID: "aa093405-83f7-4d3e-b984-bb188e5171d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:43:43.755011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.754923 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa093405-83f7-4d3e-b984-bb188e5171d7-kube-api-access-5khnl" pod "aa093405-83f7-4d3e-b984-bb188e5171d7" (UID: "aa093405-83f7-4d3e-b984-bb188e5171d7"). InnerVolumeSpecName "kube-api-access-5khnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:43.755011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.754971 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa093405-83f7-4d3e-b984-bb188e5171d7-metrics-certs" pod "aa093405-83f7-4d3e-b984-bb188e5171d7" (UID: "aa093405-83f7-4d3e-b984-bb188e5171d7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:43:43.803844 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.803760 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:43.854236 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.854210 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa093405-83f7-4d3e-b984-bb188e5171d7-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:43.854236 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.854234 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa093405-83f7-4d3e-b984-bb188e5171d7-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:43.854420 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.854246 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5khnl\" (UniqueName: \"kubernetes.io/projected/aa093405-83f7-4d3e-b984-bb188e5171d7-kube-api-access-5khnl\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:43.854420 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.854256 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa093405-83f7-4d3e-b984-bb188e5171d7-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:43.943902 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.943873 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc"] Oct 31 13:43:43.953879 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:43.953850 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2ef686-687c6b75f5-hznhc"] Oct 31 13:43:44.091654 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:44.091572 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:44.547639 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:44.547600 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:45.071178 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:45.071140 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:45.643916 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:45.643883 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a0d55fc-97da-4fba-a45a-68a229f87264" path="/var/lib/kubelet/pods/8a0d55fc-97da-4fba-a45a-68a229f87264/volumes" Oct 31 13:43:45.644358 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:45.644344 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3a899eb-181a-4f40-a64e-2953e94e6dd4" path="/var/lib/kubelet/pods/a3a899eb-181a-4f40-a64e-2953e94e6dd4/volumes" Oct 31 13:43:45.644734 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:45.644722 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa093405-83f7-4d3e-b984-bb188e5171d7" path="/var/lib/kubelet/pods/aa093405-83f7-4d3e-b984-bb188e5171d7/volumes" Oct 31 13:43:45.860916 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:45.860867 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:46.775207 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:46.775165 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:47.165668 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:47.165580 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-45bcad-6f4486b69f-qtdsk" podUID="68db5d02-47c3-4605-8967-30a007c782a7" containerName="payload-pre-processing" containerID="cri-o://f4f2ab140e6c994376056efa1b15143f9f69db4fba640bc7aa2b8e20047a7453" gracePeriod=25 Oct 31 13:43:47.874001 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:47.871687 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:48.108279 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:48.108237 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:48.417487 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:48.417450 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:48.667329 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:48.667294 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl"] Oct 31 13:43:48.667626 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:48.667553 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" podUID="6e2a46f8-aab8-482f-8edb-67800fa4960e" containerName="istio-proxy" containerID="cri-o://87d1573007990af25b8140be7754ad1af13dd956627c21cb9771456fc377791a" gracePeriod=30 Oct 31 13:43:48.803998 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:48.803892 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:49.012395 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:49.012363 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:49.658090 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:49.658036 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:49.960995 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:49.960956 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:50.036410 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:43:50.036377 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:43:50.882778 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:50.882739 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:51.760294 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:51.760240 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:52.557825 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:52.557789 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:52.780318 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:52.780285 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:53.156529 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:53.156490 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:53.422494 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:53.422409 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt"] Oct 31 13:43:53.422650 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:53.422614 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" podUID="c85bee9f-4267-44ec-a03f-0b97c6b95d24" containerName="istio-proxy" containerID="cri-o://1b0b0012d99c80ccff6505c5d68aa38f4e624f2ee77b13af29ca79b0d0d08ec3" gracePeriod=30 Oct 31 13:43:53.568945 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:53.568700 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:53.823930 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:53.823893 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:53.932763 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:53.932704 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:43:53.994943 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:53.994913 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e2a46f8-aab8-482f-8edb-67800fa4960e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e2a46f8-aab8-482f-8edb-67800fa4960e-credential-socket\") pod \"6e2a46f8-aab8-482f-8edb-67800fa4960e\" (UID: \"6e2a46f8-aab8-482f-8edb-67800fa4960e\") " Oct 31 13:43:53.995453 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:53.994965 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e2a46f8-aab8-482f-8edb-67800fa4960e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e2a46f8-aab8-482f-8edb-67800fa4960e-istio-envoy\") pod \"6e2a46f8-aab8-482f-8edb-67800fa4960e\" (UID: \"6e2a46f8-aab8-482f-8edb-67800fa4960e\") " Oct 31 13:43:53.995453 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:53.995034 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e2a46f8-aab8-482f-8edb-67800fa4960e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e2a46f8-aab8-482f-8edb-67800fa4960e-istiod-ca-cert\") pod \"6e2a46f8-aab8-482f-8edb-67800fa4960e\" (UID: \"6e2a46f8-aab8-482f-8edb-67800fa4960e\") " Oct 31 13:43:53.995453 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:53.995065 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e2a46f8-aab8-482f-8edb-67800fa4960e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e2a46f8-aab8-482f-8edb-67800fa4960e-workload-certs\") pod \"6e2a46f8-aab8-482f-8edb-67800fa4960e\" (UID: \"6e2a46f8-aab8-482f-8edb-67800fa4960e\") " Oct 31 13:43:53.995453 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:53.995091 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e2a46f8-aab8-482f-8edb-67800fa4960e-kube-api-access-djfv6\" (UniqueName: \"kubernetes.io/projected/6e2a46f8-aab8-482f-8edb-67800fa4960e-kube-api-access-djfv6\") pod \"6e2a46f8-aab8-482f-8edb-67800fa4960e\" (UID: \"6e2a46f8-aab8-482f-8edb-67800fa4960e\") " Oct 31 13:43:53.995453 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:53.995149 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e2a46f8-aab8-482f-8edb-67800fa4960e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e2a46f8-aab8-482f-8edb-67800fa4960e-istio-podinfo\") pod \"6e2a46f8-aab8-482f-8edb-67800fa4960e\" (UID: \"6e2a46f8-aab8-482f-8edb-67800fa4960e\") " Oct 31 13:43:53.995453 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:53.995219 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e2a46f8-aab8-482f-8edb-67800fa4960e-credential-socket" pod "6e2a46f8-aab8-482f-8edb-67800fa4960e" (UID: "6e2a46f8-aab8-482f-8edb-67800fa4960e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:53.995453 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:53.995236 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e2a46f8-aab8-482f-8edb-67800fa4960e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e2a46f8-aab8-482f-8edb-67800fa4960e-workload-socket\") pod \"6e2a46f8-aab8-482f-8edb-67800fa4960e\" (UID: \"6e2a46f8-aab8-482f-8edb-67800fa4960e\") " Oct 31 13:43:53.995453 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:53.995360 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e2a46f8-aab8-482f-8edb-67800fa4960e-istio-token\" (UniqueName: \"kubernetes.io/projected/6e2a46f8-aab8-482f-8edb-67800fa4960e-istio-token\") pod \"6e2a46f8-aab8-482f-8edb-67800fa4960e\" (UID: \"6e2a46f8-aab8-482f-8edb-67800fa4960e\") " Oct 31 13:43:53.995453 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:53.995390 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e2a46f8-aab8-482f-8edb-67800fa4960e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e2a46f8-aab8-482f-8edb-67800fa4960e-istio-data\") pod \"6e2a46f8-aab8-482f-8edb-67800fa4960e\" (UID: \"6e2a46f8-aab8-482f-8edb-67800fa4960e\") " Oct 31 13:43:53.996043 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:53.995433 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e2a46f8-aab8-482f-8edb-67800fa4960e-workload-certs" pod "6e2a46f8-aab8-482f-8edb-67800fa4960e" (UID: "6e2a46f8-aab8-482f-8edb-67800fa4960e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:53.996043 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:53.995643 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e2a46f8-aab8-482f-8edb-67800fa4960e-workload-socket" pod "6e2a46f8-aab8-482f-8edb-67800fa4960e" (UID: "6e2a46f8-aab8-482f-8edb-67800fa4960e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:53.996043 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:53.995851 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e2a46f8-aab8-482f-8edb-67800fa4960e-istio-data" pod "6e2a46f8-aab8-482f-8edb-67800fa4960e" (UID: "6e2a46f8-aab8-482f-8edb-67800fa4960e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:53.996286 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:53.996083 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e2a46f8-aab8-482f-8edb-67800fa4960e-istiod-ca-cert" pod "6e2a46f8-aab8-482f-8edb-67800fa4960e" (UID: "6e2a46f8-aab8-482f-8edb-67800fa4960e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:43:53.996286 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:53.996109 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e2a46f8-aab8-482f-8edb-67800fa4960e-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:53.996286 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:53.996132 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e2a46f8-aab8-482f-8edb-67800fa4960e-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:53.996286 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:53.996147 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e2a46f8-aab8-482f-8edb-67800fa4960e-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:53.996286 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:53.996164 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e2a46f8-aab8-482f-8edb-67800fa4960e-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:53.997740 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:53.997712 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e2a46f8-aab8-482f-8edb-67800fa4960e-kube-api-access-djfv6" pod "6e2a46f8-aab8-482f-8edb-67800fa4960e" (UID: "6e2a46f8-aab8-482f-8edb-67800fa4960e"). InnerVolumeSpecName "kube-api-access-djfv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:53.997891 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:53.997871 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e2a46f8-aab8-482f-8edb-67800fa4960e-istio-podinfo" pod "6e2a46f8-aab8-482f-8edb-67800fa4960e" (UID: "6e2a46f8-aab8-482f-8edb-67800fa4960e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:43:53.997961 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:53.997874 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e2a46f8-aab8-482f-8edb-67800fa4960e-istio-envoy" pod "6e2a46f8-aab8-482f-8edb-67800fa4960e" (UID: "6e2a46f8-aab8-482f-8edb-67800fa4960e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:53.998136 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:53.998112 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e2a46f8-aab8-482f-8edb-67800fa4960e-istio-token" pod "6e2a46f8-aab8-482f-8edb-67800fa4960e" (UID: "6e2a46f8-aab8-482f-8edb-67800fa4960e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:54.097117 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:54.097077 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e2a46f8-aab8-482f-8edb-67800fa4960e-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:54.097117 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:54.097118 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e2a46f8-aab8-482f-8edb-67800fa4960e-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:54.097333 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:54.097136 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djfv6\" (UniqueName: \"kubernetes.io/projected/6e2a46f8-aab8-482f-8edb-67800fa4960e-kube-api-access-djfv6\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:54.097333 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:54.097152 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e2a46f8-aab8-482f-8edb-67800fa4960e-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:54.097333 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:54.097166 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e2a46f8-aab8-482f-8edb-67800fa4960e-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:54.416644 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:54.416601 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:54.653493 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:54.653455 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:54.703953 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:54.703921 6841 generic.go:356] "Generic (PLEG): container finished" podID="6e2a46f8-aab8-482f-8edb-67800fa4960e" containerID="87d1573007990af25b8140be7754ad1af13dd956627c21cb9771456fc377791a" exitCode=0 Oct 31 13:43:54.704169 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:54.704024 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" Oct 31 13:43:54.704169 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:54.704024 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" event={"ID":"6e2a46f8-aab8-482f-8edb-67800fa4960e","Type":"ContainerDied","Data":"87d1573007990af25b8140be7754ad1af13dd956627c21cb9771456fc377791a"} Oct 31 13:43:54.704169 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:54.704150 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl" event={"ID":"6e2a46f8-aab8-482f-8edb-67800fa4960e","Type":"ContainerDied","Data":"8a1b3987a4d7905568a71973d1d878e67cec57d3366efd086d9de4f815f3ad65"} Oct 31 13:43:54.704350 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:54.704178 6841 scope.go:122] "RemoveContainer" containerID="87d1573007990af25b8140be7754ad1af13dd956627c21cb9771456fc377791a" Oct 31 13:43:54.715465 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:54.715450 6841 scope.go:122] "RemoveContainer" containerID="fec8706920e9bd7d9213ad585b2dc602732f62dc3d5afb107b76d2ed2303e143" Oct 31 13:43:54.725353 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:54.725327 6841 scope.go:122] "RemoveContainer" containerID="87d1573007990af25b8140be7754ad1af13dd956627c21cb9771456fc377791a" Oct 31 13:43:54.725627 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:43:54.725608 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87d1573007990af25b8140be7754ad1af13dd956627c21cb9771456fc377791a\": container with ID starting with 87d1573007990af25b8140be7754ad1af13dd956627c21cb9771456fc377791a not found: ID does not exist" containerID="87d1573007990af25b8140be7754ad1af13dd956627c21cb9771456fc377791a" Oct 31 13:43:54.725679 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:54.725637 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87d1573007990af25b8140be7754ad1af13dd956627c21cb9771456fc377791a"} err="failed to get container status \"87d1573007990af25b8140be7754ad1af13dd956627c21cb9771456fc377791a\": rpc error: code = NotFound desc = could not find container \"87d1573007990af25b8140be7754ad1af13dd956627c21cb9771456fc377791a\": container with ID starting with 87d1573007990af25b8140be7754ad1af13dd956627c21cb9771456fc377791a not found: ID does not exist" Oct 31 13:43:54.725679 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:54.725655 6841 scope.go:122] "RemoveContainer" containerID="fec8706920e9bd7d9213ad585b2dc602732f62dc3d5afb107b76d2ed2303e143" Oct 31 13:43:54.725894 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:43:54.725872 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fec8706920e9bd7d9213ad585b2dc602732f62dc3d5afb107b76d2ed2303e143\": container with ID starting with fec8706920e9bd7d9213ad585b2dc602732f62dc3d5afb107b76d2ed2303e143 not found: ID does not exist" containerID="fec8706920e9bd7d9213ad585b2dc602732f62dc3d5afb107b76d2ed2303e143" Oct 31 13:43:54.725954 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:54.725904 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fec8706920e9bd7d9213ad585b2dc602732f62dc3d5afb107b76d2ed2303e143"} err="failed to get container status \"fec8706920e9bd7d9213ad585b2dc602732f62dc3d5afb107b76d2ed2303e143\": rpc error: code = NotFound desc = could not find container \"fec8706920e9bd7d9213ad585b2dc602732f62dc3d5afb107b76d2ed2303e143\": container with ID starting with fec8706920e9bd7d9213ad585b2dc602732f62dc3d5afb107b76d2ed2303e143 not found: ID does not exist" Oct 31 13:43:54.733239 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:54.733216 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl"] Oct 31 13:43:54.738087 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:54.738066 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-2ef686-openshift-default-777849d465-gn8vl"] Oct 31 13:43:55.500448 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:55.500410 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:55.643006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:55.642941 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e2a46f8-aab8-482f-8edb-67800fa4960e" path="/var/lib/kubelet/pods/6e2a46f8-aab8-482f-8edb-67800fa4960e/volumes" Oct 31 13:43:55.708559 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:55.708523 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-45bcad-6f4486b69f-qtdsk" podUID="68db5d02-47c3-4605-8967-30a007c782a7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:43:56.719355 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:56.719298 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:57.665581 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:57.665551 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:58.708600 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.708575 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:43:58.736962 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.736929 6841 generic.go:356] "Generic (PLEG): container finished" podID="c85bee9f-4267-44ec-a03f-0b97c6b95d24" containerID="1b0b0012d99c80ccff6505c5d68aa38f4e624f2ee77b13af29ca79b0d0d08ec3" exitCode=0 Oct 31 13:43:58.737152 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.736976 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" Oct 31 13:43:58.737152 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.737017 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" event={"ID":"c85bee9f-4267-44ec-a03f-0b97c6b95d24","Type":"ContainerDied","Data":"1b0b0012d99c80ccff6505c5d68aa38f4e624f2ee77b13af29ca79b0d0d08ec3"} Oct 31 13:43:58.737152 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.737086 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt" event={"ID":"c85bee9f-4267-44ec-a03f-0b97c6b95d24","Type":"ContainerDied","Data":"da69a7f4ee393af06fc3cd53158e23f03b92a40bbf1d196fe88866a3ccf36f04"} Oct 31 13:43:58.737152 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.737134 6841 scope.go:122] "RemoveContainer" containerID="1b0b0012d99c80ccff6505c5d68aa38f4e624f2ee77b13af29ca79b0d0d08ec3" Oct 31 13:43:58.749038 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.749015 6841 scope.go:122] "RemoveContainer" containerID="1b1b3d24f4bcadf276d1dcf4ebedd9018d267f346abb926a4c64ac2f352d023f" Oct 31 13:43:58.758349 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.758330 6841 scope.go:122] "RemoveContainer" containerID="1b0b0012d99c80ccff6505c5d68aa38f4e624f2ee77b13af29ca79b0d0d08ec3" Oct 31 13:43:58.758613 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:43:58.758593 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b0b0012d99c80ccff6505c5d68aa38f4e624f2ee77b13af29ca79b0d0d08ec3\": container with ID starting with 1b0b0012d99c80ccff6505c5d68aa38f4e624f2ee77b13af29ca79b0d0d08ec3 not found: ID does not exist" containerID="1b0b0012d99c80ccff6505c5d68aa38f4e624f2ee77b13af29ca79b0d0d08ec3" Oct 31 13:43:58.758692 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.758625 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b0b0012d99c80ccff6505c5d68aa38f4e624f2ee77b13af29ca79b0d0d08ec3"} err="failed to get container status \"1b0b0012d99c80ccff6505c5d68aa38f4e624f2ee77b13af29ca79b0d0d08ec3\": rpc error: code = NotFound desc = could not find container \"1b0b0012d99c80ccff6505c5d68aa38f4e624f2ee77b13af29ca79b0d0d08ec3\": container with ID starting with 1b0b0012d99c80ccff6505c5d68aa38f4e624f2ee77b13af29ca79b0d0d08ec3 not found: ID does not exist" Oct 31 13:43:58.758692 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.758642 6841 scope.go:122] "RemoveContainer" containerID="1b1b3d24f4bcadf276d1dcf4ebedd9018d267f346abb926a4c64ac2f352d023f" Oct 31 13:43:58.758917 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:43:58.758903 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b1b3d24f4bcadf276d1dcf4ebedd9018d267f346abb926a4c64ac2f352d023f\": container with ID starting with 1b1b3d24f4bcadf276d1dcf4ebedd9018d267f346abb926a4c64ac2f352d023f not found: ID does not exist" containerID="1b1b3d24f4bcadf276d1dcf4ebedd9018d267f346abb926a4c64ac2f352d023f" Oct 31 13:43:58.759017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.758922 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b1b3d24f4bcadf276d1dcf4ebedd9018d267f346abb926a4c64ac2f352d023f"} err="failed to get container status \"1b1b3d24f4bcadf276d1dcf4ebedd9018d267f346abb926a4c64ac2f352d023f\": rpc error: code = NotFound desc = could not find container \"1b1b3d24f4bcadf276d1dcf4ebedd9018d267f346abb926a4c64ac2f352d023f\": container with ID starting with 1b1b3d24f4bcadf276d1dcf4ebedd9018d267f346abb926a4c64ac2f352d023f not found: ID does not exist" Oct 31 13:43:58.882883 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.875310 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c85bee9f-4267-44ec-a03f-0b97c6b95d24-kube-api-access-7zrtf\" (UniqueName: \"kubernetes.io/projected/c85bee9f-4267-44ec-a03f-0b97c6b95d24-kube-api-access-7zrtf\") pod \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\" (UID: \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\") " Oct 31 13:43:58.882883 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.875372 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c85bee9f-4267-44ec-a03f-0b97c6b95d24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c85bee9f-4267-44ec-a03f-0b97c6b95d24-credential-socket\") pod \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\" (UID: \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\") " Oct 31 13:43:58.882883 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.875440 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c85bee9f-4267-44ec-a03f-0b97c6b95d24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c85bee9f-4267-44ec-a03f-0b97c6b95d24-workload-certs\") pod \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\" (UID: \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\") " Oct 31 13:43:58.882883 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.875491 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c85bee9f-4267-44ec-a03f-0b97c6b95d24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c85bee9f-4267-44ec-a03f-0b97c6b95d24-istio-data\") pod \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\" (UID: \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\") " Oct 31 13:43:58.882883 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.875558 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c85bee9f-4267-44ec-a03f-0b97c6b95d24-istio-token\" (UniqueName: \"kubernetes.io/projected/c85bee9f-4267-44ec-a03f-0b97c6b95d24-istio-token\") pod \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\" (UID: \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\") " Oct 31 13:43:58.882883 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.875591 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c85bee9f-4267-44ec-a03f-0b97c6b95d24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c85bee9f-4267-44ec-a03f-0b97c6b95d24-istio-envoy\") pod \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\" (UID: \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\") " Oct 31 13:43:58.882883 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.875619 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c85bee9f-4267-44ec-a03f-0b97c6b95d24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c85bee9f-4267-44ec-a03f-0b97c6b95d24-istiod-ca-cert\") pod \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\" (UID: \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\") " Oct 31 13:43:58.882883 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.875693 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c85bee9f-4267-44ec-a03f-0b97c6b95d24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c85bee9f-4267-44ec-a03f-0b97c6b95d24-istio-podinfo\") pod \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\" (UID: \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\") " Oct 31 13:43:58.882883 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.875791 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c85bee9f-4267-44ec-a03f-0b97c6b95d24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c85bee9f-4267-44ec-a03f-0b97c6b95d24-workload-socket\") pod \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\" (UID: \"c85bee9f-4267-44ec-a03f-0b97c6b95d24\") " Oct 31 13:43:58.887247 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.887177 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c85bee9f-4267-44ec-a03f-0b97c6b95d24-istio-data" pod "c85bee9f-4267-44ec-a03f-0b97c6b95d24" (UID: "c85bee9f-4267-44ec-a03f-0b97c6b95d24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:58.888473 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.888144 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c85bee9f-4267-44ec-a03f-0b97c6b95d24-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:58.888945 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.888916 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c85bee9f-4267-44ec-a03f-0b97c6b95d24-credential-socket" pod "c85bee9f-4267-44ec-a03f-0b97c6b95d24" (UID: "c85bee9f-4267-44ec-a03f-0b97c6b95d24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:58.889467 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.889443 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c85bee9f-4267-44ec-a03f-0b97c6b95d24-workload-certs" pod "c85bee9f-4267-44ec-a03f-0b97c6b95d24" (UID: "c85bee9f-4267-44ec-a03f-0b97c6b95d24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:58.890256 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.890228 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c85bee9f-4267-44ec-a03f-0b97c6b95d24-istiod-ca-cert" pod "c85bee9f-4267-44ec-a03f-0b97c6b95d24" (UID: "c85bee9f-4267-44ec-a03f-0b97c6b95d24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:43:58.892748 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.891829 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c85bee9f-4267-44ec-a03f-0b97c6b95d24-workload-socket" pod "c85bee9f-4267-44ec-a03f-0b97c6b95d24" (UID: "c85bee9f-4267-44ec-a03f-0b97c6b95d24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:58.907025 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.897186 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c85bee9f-4267-44ec-a03f-0b97c6b95d24-istio-podinfo" pod "c85bee9f-4267-44ec-a03f-0b97c6b95d24" (UID: "c85bee9f-4267-44ec-a03f-0b97c6b95d24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:43:58.907025 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.899142 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c85bee9f-4267-44ec-a03f-0b97c6b95d24-istio-token" pod "c85bee9f-4267-44ec-a03f-0b97c6b95d24" (UID: "c85bee9f-4267-44ec-a03f-0b97c6b95d24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:58.907025 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.904337 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c85bee9f-4267-44ec-a03f-0b97c6b95d24-istio-envoy" pod "c85bee9f-4267-44ec-a03f-0b97c6b95d24" (UID: "c85bee9f-4267-44ec-a03f-0b97c6b95d24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:58.913196 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.913146 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c85bee9f-4267-44ec-a03f-0b97c6b95d24-kube-api-access-7zrtf" pod "c85bee9f-4267-44ec-a03f-0b97c6b95d24" (UID: "c85bee9f-4267-44ec-a03f-0b97c6b95d24"). InnerVolumeSpecName "kube-api-access-7zrtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:58.989777 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.989708 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c85bee9f-4267-44ec-a03f-0b97c6b95d24-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:58.989777 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.989735 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c85bee9f-4267-44ec-a03f-0b97c6b95d24-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:58.989777 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.989744 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c85bee9f-4267-44ec-a03f-0b97c6b95d24-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:58.989777 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.989754 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c85bee9f-4267-44ec-a03f-0b97c6b95d24-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:58.989777 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.989765 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c85bee9f-4267-44ec-a03f-0b97c6b95d24-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:58.989777 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.989775 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c85bee9f-4267-44ec-a03f-0b97c6b95d24-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:58.990139 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.989789 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zrtf\" (UniqueName: \"kubernetes.io/projected/c85bee9f-4267-44ec-a03f-0b97c6b95d24-kube-api-access-7zrtf\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:58.990139 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:58.989799 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c85bee9f-4267-44ec-a03f-0b97c6b95d24-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:59.004579 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:59.004550 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:59.070210 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:59.070170 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt"] Oct 31 13:43:59.076911 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:59.076877 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-j85qt"] Oct 31 13:43:59.317015 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:59.316870 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:43:59.642512 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:59.642421 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c85bee9f-4267-44ec-a03f-0b97c6b95d24" path="/var/lib/kubelet/pods/c85bee9f-4267-44ec-a03f-0b97c6b95d24/volumes" Oct 31 13:43:59.762804 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:43:59.762767 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:00.888898 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:00.888847 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:01.653580 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:01.653547 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:01.876036 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:01.875999 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:02.883639 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:02.883600 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:03.362795 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:03.362766 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:03.795823 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:03.795788 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:04.697173 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:04.697129 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:05.521927 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:05.521898 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:05.708533 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:05.708498 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-45bcad-6f4486b69f-qtdsk" podUID="68db5d02-47c3-4605-8967-30a007c782a7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:44:06.176232 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:06.176194 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:06.497920 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:06.497890 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:07.043037 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:07.042969 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:07.336064 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:07.335911 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:07.428722 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:07.428698 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-45bcad-6f4486b69f-qtdsk" Oct 31 13:44:07.491655 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:07.491624 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68db5d02-47c3-4605-8967-30a007c782a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68db5d02-47c3-4605-8967-30a007c782a7-plugins-config-volume\") pod \"68db5d02-47c3-4605-8967-30a007c782a7\" (UID: \"68db5d02-47c3-4605-8967-30a007c782a7\") " Oct 31 13:44:07.491837 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:07.491707 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68db5d02-47c3-4605-8967-30a007c782a7-kube-api-access-dz8x4\" (UniqueName: \"kubernetes.io/projected/68db5d02-47c3-4605-8967-30a007c782a7-kube-api-access-dz8x4\") pod \"68db5d02-47c3-4605-8967-30a007c782a7\" (UID: \"68db5d02-47c3-4605-8967-30a007c782a7\") " Oct 31 13:44:07.492070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:07.492043 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68db5d02-47c3-4605-8967-30a007c782a7-plugins-config-volume" pod "68db5d02-47c3-4605-8967-30a007c782a7" (UID: "68db5d02-47c3-4605-8967-30a007c782a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:07.496229 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:07.496205 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68db5d02-47c3-4605-8967-30a007c782a7-kube-api-access-dz8x4" pod "68db5d02-47c3-4605-8967-30a007c782a7" (UID: "68db5d02-47c3-4605-8967-30a007c782a7"). InnerVolumeSpecName "kube-api-access-dz8x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:07.593215 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:07.593133 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68db5d02-47c3-4605-8967-30a007c782a7-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:07.593215 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:07.593161 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz8x4\" (UniqueName: \"kubernetes.io/projected/68db5d02-47c3-4605-8967-30a007c782a7-kube-api-access-dz8x4\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:07.801683 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:07.801649 6841 generic.go:356] "Generic (PLEG): container finished" podID="68db5d02-47c3-4605-8967-30a007c782a7" containerID="f4f2ab140e6c994376056efa1b15143f9f69db4fba640bc7aa2b8e20047a7453" exitCode=0 Oct 31 13:44:07.801840 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:07.801705 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-45bcad-6f4486b69f-qtdsk" Oct 31 13:44:07.801840 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:07.801732 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-45bcad-6f4486b69f-qtdsk" event={"ID":"68db5d02-47c3-4605-8967-30a007c782a7","Type":"ContainerDied","Data":"f4f2ab140e6c994376056efa1b15143f9f69db4fba640bc7aa2b8e20047a7453"} Oct 31 13:44:07.801840 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:07.801792 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-45bcad-6f4486b69f-qtdsk" event={"ID":"68db5d02-47c3-4605-8967-30a007c782a7","Type":"ContainerDied","Data":"13b0e62220ad3a4c07d5503d0200e4c6592ccdf7e30e723bedd40e98e658a340"} Oct 31 13:44:07.801840 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:07.801816 6841 scope.go:122] "RemoveContainer" containerID="f4f2ab140e6c994376056efa1b15143f9f69db4fba640bc7aa2b8e20047a7453" Oct 31 13:44:07.811431 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:07.811413 6841 scope.go:122] "RemoveContainer" containerID="f4f2ab140e6c994376056efa1b15143f9f69db4fba640bc7aa2b8e20047a7453" Oct 31 13:44:07.811725 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:44:07.811700 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4f2ab140e6c994376056efa1b15143f9f69db4fba640bc7aa2b8e20047a7453\": container with ID starting with f4f2ab140e6c994376056efa1b15143f9f69db4fba640bc7aa2b8e20047a7453 not found: ID does not exist" containerID="f4f2ab140e6c994376056efa1b15143f9f69db4fba640bc7aa2b8e20047a7453" Oct 31 13:44:07.811806 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:07.811734 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4f2ab140e6c994376056efa1b15143f9f69db4fba640bc7aa2b8e20047a7453"} err="failed to get container status \"f4f2ab140e6c994376056efa1b15143f9f69db4fba640bc7aa2b8e20047a7453\": rpc error: code = NotFound desc = could not find container \"f4f2ab140e6c994376056efa1b15143f9f69db4fba640bc7aa2b8e20047a7453\": container with ID starting with f4f2ab140e6c994376056efa1b15143f9f69db4fba640bc7aa2b8e20047a7453 not found: ID does not exist" Oct 31 13:44:07.829596 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:07.829569 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-45bcad-6f4486b69f-qtdsk"] Oct 31 13:44:07.833579 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:07.833560 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-45bcad-6f4486b69f-qtdsk"] Oct 31 13:44:08.272487 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:08.272432 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:09.128875 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:09.128842 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm"] Oct 31 13:44:09.129114 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:09.129076 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm" podUID="481838f4-af98-4e2e-8f4e-2db1897925c4" containerName="maas-api" containerID="cri-o://cc4ac0333c956e5a4b8d57f2f0cedde2b11235a4030515db7bfdbeaeefa1054f" gracePeriod=30 Oct 31 13:44:09.268703 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:09.268640 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:09.442243 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:09.442213 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm" Oct 31 13:44:09.521416 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:09.520552 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/481838f4-af98-4e2e-8f4e-2db1897925c4-kube-api-access-xb8kd\" (UniqueName: \"kubernetes.io/projected/481838f4-af98-4e2e-8f4e-2db1897925c4-kube-api-access-xb8kd\") pod \"481838f4-af98-4e2e-8f4e-2db1897925c4\" (UID: \"481838f4-af98-4e2e-8f4e-2db1897925c4\") " Oct 31 13:44:09.521416 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:09.520602 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/481838f4-af98-4e2e-8f4e-2db1897925c4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/481838f4-af98-4e2e-8f4e-2db1897925c4-metrics-certs\") pod \"481838f4-af98-4e2e-8f4e-2db1897925c4\" (UID: \"481838f4-af98-4e2e-8f4e-2db1897925c4\") " Oct 31 13:44:09.521416 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:09.520691 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/481838f4-af98-4e2e-8f4e-2db1897925c4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/481838f4-af98-4e2e-8f4e-2db1897925c4-openshift-service-ca\") pod \"481838f4-af98-4e2e-8f4e-2db1897925c4\" (UID: \"481838f4-af98-4e2e-8f4e-2db1897925c4\") " Oct 31 13:44:09.521416 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:09.520796 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/481838f4-af98-4e2e-8f4e-2db1897925c4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/481838f4-af98-4e2e-8f4e-2db1897925c4-maas-api-tls\") pod \"481838f4-af98-4e2e-8f4e-2db1897925c4\" (UID: \"481838f4-af98-4e2e-8f4e-2db1897925c4\") " Oct 31 13:44:09.521920 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:09.521684 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/481838f4-af98-4e2e-8f4e-2db1897925c4-openshift-service-ca" pod "481838f4-af98-4e2e-8f4e-2db1897925c4" (UID: "481838f4-af98-4e2e-8f4e-2db1897925c4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:09.526523 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:09.526478 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/481838f4-af98-4e2e-8f4e-2db1897925c4-maas-api-tls" pod "481838f4-af98-4e2e-8f4e-2db1897925c4" (UID: "481838f4-af98-4e2e-8f4e-2db1897925c4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:44:09.527095 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:09.527051 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/481838f4-af98-4e2e-8f4e-2db1897925c4-metrics-certs" pod "481838f4-af98-4e2e-8f4e-2db1897925c4" (UID: "481838f4-af98-4e2e-8f4e-2db1897925c4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:44:09.528339 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:09.528308 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/481838f4-af98-4e2e-8f4e-2db1897925c4-kube-api-access-xb8kd" pod "481838f4-af98-4e2e-8f4e-2db1897925c4" (UID: "481838f4-af98-4e2e-8f4e-2db1897925c4"). InnerVolumeSpecName "kube-api-access-xb8kd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:09.572600 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:09.572564 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:09.622560 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:09.622530 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/481838f4-af98-4e2e-8f4e-2db1897925c4-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:09.622560 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:09.622558 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb8kd\" (UniqueName: \"kubernetes.io/projected/481838f4-af98-4e2e-8f4e-2db1897925c4-kube-api-access-xb8kd\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:09.622769 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:09.622570 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/481838f4-af98-4e2e-8f4e-2db1897925c4-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:09.622769 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:09.622580 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/481838f4-af98-4e2e-8f4e-2db1897925c4-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:09.642277 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:09.642245 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68db5d02-47c3-4605-8967-30a007c782a7" path="/var/lib/kubelet/pods/68db5d02-47c3-4605-8967-30a007c782a7/volumes" Oct 31 13:44:09.817629 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:09.817595 6841 generic.go:356] "Generic (PLEG): container finished" podID="481838f4-af98-4e2e-8f4e-2db1897925c4" containerID="cc4ac0333c956e5a4b8d57f2f0cedde2b11235a4030515db7bfdbeaeefa1054f" exitCode=0 Oct 31 13:44:09.817850 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:09.817646 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm" Oct 31 13:44:09.817850 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:09.817680 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm" event={"ID":"481838f4-af98-4e2e-8f4e-2db1897925c4","Type":"ContainerDied","Data":"cc4ac0333c956e5a4b8d57f2f0cedde2b11235a4030515db7bfdbeaeefa1054f"} Oct 31 13:44:09.817850 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:09.817733 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm" event={"ID":"481838f4-af98-4e2e-8f4e-2db1897925c4","Type":"ContainerDied","Data":"88a86332e00ca4d79baa69250cb4a3c2748965c9ab4a3eb6ebbe395d071f9721"} Oct 31 13:44:09.817850 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:09.817764 6841 scope.go:122] "RemoveContainer" containerID="cc4ac0333c956e5a4b8d57f2f0cedde2b11235a4030515db7bfdbeaeefa1054f" Oct 31 13:44:09.827509 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:09.827491 6841 scope.go:122] "RemoveContainer" containerID="cc4ac0333c956e5a4b8d57f2f0cedde2b11235a4030515db7bfdbeaeefa1054f" Oct 31 13:44:09.827798 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:44:09.827774 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc4ac0333c956e5a4b8d57f2f0cedde2b11235a4030515db7bfdbeaeefa1054f\": container with ID starting with cc4ac0333c956e5a4b8d57f2f0cedde2b11235a4030515db7bfdbeaeefa1054f not found: ID does not exist" containerID="cc4ac0333c956e5a4b8d57f2f0cedde2b11235a4030515db7bfdbeaeefa1054f" Oct 31 13:44:09.827875 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:09.827806 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc4ac0333c956e5a4b8d57f2f0cedde2b11235a4030515db7bfdbeaeefa1054f"} err="failed to get container status \"cc4ac0333c956e5a4b8d57f2f0cedde2b11235a4030515db7bfdbeaeefa1054f\": rpc error: code = NotFound desc = could not find container \"cc4ac0333c956e5a4b8d57f2f0cedde2b11235a4030515db7bfdbeaeefa1054f\": container with ID starting with cc4ac0333c956e5a4b8d57f2f0cedde2b11235a4030515db7bfdbeaeefa1054f not found: ID does not exist" Oct 31 13:44:09.844658 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:09.844635 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm"] Oct 31 13:44:09.864273 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:09.864247 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5a9e45-5f5bcb544-g8ktm"] Oct 31 13:44:10.418138 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:10.418099 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:11.146454 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:11.146401 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:11.359603 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:11.359567 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:11.642327 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:11.642294 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="481838f4-af98-4e2e-8f4e-2db1897925c4" path="/var/lib/kubelet/pods/481838f4-af98-4e2e-8f4e-2db1897925c4/volumes" Oct 31 13:44:11.674730 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:11.674698 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:11.940116 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:11.940030 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df"] Oct 31 13:44:11.940305 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:11.940280 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" podUID="e4e95366-875b-4fda-8e1b-6e441df2f5b2" containerName="istio-proxy" containerID="cri-o://a23137f1b1b265f1cf0325beaa79d15d14a373a04245b16622e8c6d98aec9cf1" gracePeriod=30 Oct 31 13:44:12.071976 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:12.071938 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:12.318064 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:12.318032 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:12.553708 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:12.553669 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:13.262442 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:13.262401 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:14.210502 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:14.210464 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:14.859194 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:14.859158 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:15.227448 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:15.227415 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:15.469834 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:15.469800 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:15.695900 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:15.695858 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:15.914439 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:15.914398 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:16.351798 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:16.351752 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:16.667282 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:16.667199 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:17.247171 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.247143 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:44:17.265214 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.265185 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:17.319117 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.319088 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4e95366-875b-4fda-8e1b-6e441df2f5b2-istio-token\" (UniqueName: \"kubernetes.io/projected/e4e95366-875b-4fda-8e1b-6e441df2f5b2-istio-token\") pod \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\" (UID: \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\") " Oct 31 13:44:17.319324 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.319133 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4e95366-875b-4fda-8e1b-6e441df2f5b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4e95366-875b-4fda-8e1b-6e441df2f5b2-istiod-ca-cert\") pod \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\" (UID: \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\") " Oct 31 13:44:17.319324 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.319171 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4e95366-875b-4fda-8e1b-6e441df2f5b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e95366-875b-4fda-8e1b-6e441df2f5b2-workload-certs\") pod \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\" (UID: \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\") " Oct 31 13:44:17.319324 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.319272 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4e95366-875b-4fda-8e1b-6e441df2f5b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4e95366-875b-4fda-8e1b-6e441df2f5b2-istio-podinfo\") pod \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\" (UID: \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\") " Oct 31 13:44:17.319524 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.319327 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4e95366-875b-4fda-8e1b-6e441df2f5b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4e95366-875b-4fda-8e1b-6e441df2f5b2-istio-data\") pod \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\" (UID: \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\") " Oct 31 13:44:17.319524 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.319383 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4e95366-875b-4fda-8e1b-6e441df2f5b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4e95366-875b-4fda-8e1b-6e441df2f5b2-istio-envoy\") pod \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\" (UID: \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\") " Oct 31 13:44:17.319524 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.319417 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4e95366-875b-4fda-8e1b-6e441df2f5b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e95366-875b-4fda-8e1b-6e441df2f5b2-credential-socket\") pod \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\" (UID: \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\") " Oct 31 13:44:17.319524 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.319479 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4e95366-875b-4fda-8e1b-6e441df2f5b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e95366-875b-4fda-8e1b-6e441df2f5b2-workload-socket\") pod \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\" (UID: \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\") " Oct 31 13:44:17.319524 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.319517 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4e95366-875b-4fda-8e1b-6e441df2f5b2-kube-api-access-n8t5z\" (UniqueName: \"kubernetes.io/projected/e4e95366-875b-4fda-8e1b-6e441df2f5b2-kube-api-access-n8t5z\") pod \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\" (UID: \"e4e95366-875b-4fda-8e1b-6e441df2f5b2\") " Oct 31 13:44:17.319803 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.319580 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e95366-875b-4fda-8e1b-6e441df2f5b2-workload-certs" pod "e4e95366-875b-4fda-8e1b-6e441df2f5b2" (UID: "e4e95366-875b-4fda-8e1b-6e441df2f5b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:17.320252 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.320087 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e95366-875b-4fda-8e1b-6e441df2f5b2-workload-socket" pod "e4e95366-875b-4fda-8e1b-6e441df2f5b2" (UID: "e4e95366-875b-4fda-8e1b-6e441df2f5b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:17.320252 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.320210 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e95366-875b-4fda-8e1b-6e441df2f5b2-credential-socket" pod "e4e95366-875b-4fda-8e1b-6e441df2f5b2" (UID: "e4e95366-875b-4fda-8e1b-6e441df2f5b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:17.320472 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.320408 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4e95366-875b-4fda-8e1b-6e441df2f5b2-istiod-ca-cert" pod "e4e95366-875b-4fda-8e1b-6e441df2f5b2" (UID: "e4e95366-875b-4fda-8e1b-6e441df2f5b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:17.320543 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.320472 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e95366-875b-4fda-8e1b-6e441df2f5b2-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:17.320543 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.320494 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e95366-875b-4fda-8e1b-6e441df2f5b2-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:17.320543 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.320508 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e95366-875b-4fda-8e1b-6e441df2f5b2-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:17.320730 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.320608 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e95366-875b-4fda-8e1b-6e441df2f5b2-istio-data" pod "e4e95366-875b-4fda-8e1b-6e441df2f5b2" (UID: "e4e95366-875b-4fda-8e1b-6e441df2f5b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:17.322042 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.322016 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e95366-875b-4fda-8e1b-6e441df2f5b2-istio-token" pod "e4e95366-875b-4fda-8e1b-6e441df2f5b2" (UID: "e4e95366-875b-4fda-8e1b-6e441df2f5b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:17.322303 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.322280 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4e95366-875b-4fda-8e1b-6e441df2f5b2-istio-podinfo" pod "e4e95366-875b-4fda-8e1b-6e441df2f5b2" (UID: "e4e95366-875b-4fda-8e1b-6e441df2f5b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:44:17.322490 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.322465 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e95366-875b-4fda-8e1b-6e441df2f5b2-kube-api-access-n8t5z" pod "e4e95366-875b-4fda-8e1b-6e441df2f5b2" (UID: "e4e95366-875b-4fda-8e1b-6e441df2f5b2"). InnerVolumeSpecName "kube-api-access-n8t5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:17.322571 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.322469 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e95366-875b-4fda-8e1b-6e441df2f5b2-istio-envoy" pod "e4e95366-875b-4fda-8e1b-6e441df2f5b2" (UID: "e4e95366-875b-4fda-8e1b-6e441df2f5b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:17.421778 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.421745 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4e95366-875b-4fda-8e1b-6e441df2f5b2-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:17.421778 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.421774 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8t5z\" (UniqueName: \"kubernetes.io/projected/e4e95366-875b-4fda-8e1b-6e441df2f5b2-kube-api-access-n8t5z\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:17.421778 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.421785 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4e95366-875b-4fda-8e1b-6e441df2f5b2-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:17.422278 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.421794 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4e95366-875b-4fda-8e1b-6e441df2f5b2-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:17.422278 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.421803 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4e95366-875b-4fda-8e1b-6e441df2f5b2-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:17.422278 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.421811 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4e95366-875b-4fda-8e1b-6e441df2f5b2-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:17.652082 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.652046 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:17.878285 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.878250 6841 generic.go:356] "Generic (PLEG): container finished" podID="e4e95366-875b-4fda-8e1b-6e441df2f5b2" containerID="a23137f1b1b265f1cf0325beaa79d15d14a373a04245b16622e8c6d98aec9cf1" exitCode=0 Oct 31 13:44:17.878473 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.878304 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" Oct 31 13:44:17.878473 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.878334 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" event={"ID":"e4e95366-875b-4fda-8e1b-6e441df2f5b2","Type":"ContainerDied","Data":"a23137f1b1b265f1cf0325beaa79d15d14a373a04245b16622e8c6d98aec9cf1"} Oct 31 13:44:17.878473 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.878390 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df" event={"ID":"e4e95366-875b-4fda-8e1b-6e441df2f5b2","Type":"ContainerDied","Data":"98673cac34ee61d6c099d9d31a70ac0a53159c36a9796da74a3aeabbabcc40e1"} Oct 31 13:44:17.878473 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.878413 6841 scope.go:122] "RemoveContainer" containerID="a23137f1b1b265f1cf0325beaa79d15d14a373a04245b16622e8c6d98aec9cf1" Oct 31 13:44:17.888186 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.888169 6841 scope.go:122] "RemoveContainer" containerID="292291e58a2f3d3a1b8136feffc27ea8b3fc67d29f274879803b92e63bd231b4" Oct 31 13:44:17.897424 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.897401 6841 scope.go:122] "RemoveContainer" containerID="a23137f1b1b265f1cf0325beaa79d15d14a373a04245b16622e8c6d98aec9cf1" Oct 31 13:44:17.897691 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:44:17.897672 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a23137f1b1b265f1cf0325beaa79d15d14a373a04245b16622e8c6d98aec9cf1\": container with ID starting with a23137f1b1b265f1cf0325beaa79d15d14a373a04245b16622e8c6d98aec9cf1 not found: ID does not exist" containerID="a23137f1b1b265f1cf0325beaa79d15d14a373a04245b16622e8c6d98aec9cf1" Oct 31 13:44:17.897762 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.897702 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a23137f1b1b265f1cf0325beaa79d15d14a373a04245b16622e8c6d98aec9cf1"} err="failed to get container status \"a23137f1b1b265f1cf0325beaa79d15d14a373a04245b16622e8c6d98aec9cf1\": rpc error: code = NotFound desc = could not find container \"a23137f1b1b265f1cf0325beaa79d15d14a373a04245b16622e8c6d98aec9cf1\": container with ID starting with a23137f1b1b265f1cf0325beaa79d15d14a373a04245b16622e8c6d98aec9cf1 not found: ID does not exist" Oct 31 13:44:17.897762 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.897725 6841 scope.go:122] "RemoveContainer" containerID="292291e58a2f3d3a1b8136feffc27ea8b3fc67d29f274879803b92e63bd231b4" Oct 31 13:44:17.897978 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:44:17.897961 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"292291e58a2f3d3a1b8136feffc27ea8b3fc67d29f274879803b92e63bd231b4\": container with ID starting with 292291e58a2f3d3a1b8136feffc27ea8b3fc67d29f274879803b92e63bd231b4 not found: ID does not exist" containerID="292291e58a2f3d3a1b8136feffc27ea8b3fc67d29f274879803b92e63bd231b4" Oct 31 13:44:17.897978 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.898000 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"292291e58a2f3d3a1b8136feffc27ea8b3fc67d29f274879803b92e63bd231b4"} err="failed to get container status \"292291e58a2f3d3a1b8136feffc27ea8b3fc67d29f274879803b92e63bd231b4\": rpc error: code = NotFound desc = could not find container \"292291e58a2f3d3a1b8136feffc27ea8b3fc67d29f274879803b92e63bd231b4\": container with ID starting with 292291e58a2f3d3a1b8136feffc27ea8b3fc67d29f274879803b92e63bd231b4 not found: ID does not exist" Oct 31 13:44:17.908738 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.908684 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df"] Oct 31 13:44:17.915058 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:17.915035 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-bk7df"] Oct 31 13:44:18.080288 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:18.080252 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:18.316928 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:18.316879 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:19.271335 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:19.271297 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:19.642635 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:19.642543 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4e95366-875b-4fda-8e1b-6e441df2f5b2" path="/var/lib/kubelet/pods/e4e95366-875b-4fda-8e1b-6e441df2f5b2/volumes" Oct 31 13:44:20.033911 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:44:20.033879 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:44:20.434325 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:20.434237 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:21.247031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:21.246958 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:22.179844 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:22.179811 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:23.146409 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:23.146366 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:23.246240 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:23.246202 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn"] Oct 31 13:44:23.246734 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:23.246424 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" podUID="6ac22634-1f31-475f-ae5b-7bb13eb7c6b6" containerName="istio-proxy" containerID="cri-o://c3e5a5502f6547128ea04d6f9cc100477289bd0147dbf50dc383ade051a3dac6" gracePeriod=30 Oct 31 13:44:24.024276 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:24.024235 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:24.308736 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:24.308648 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:24.760008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:24.759945 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:24.994891 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:24.994838 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:25.506665 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:25.506629 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:26.198296 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:26.198261 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:26.478929 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:26.478839 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:26.870647 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:26.870560 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:27.427323 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:27.427285 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:28.234789 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.234754 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:28.510132 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.510107 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" Oct 31 13:44:28.658714 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.658682 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-workload-certs\") pod \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\" (UID: \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\") " Oct 31 13:44:28.658929 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.658752 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-istio-token\" (UniqueName: \"kubernetes.io/projected/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-istio-token\") pod \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\" (UID: \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\") " Oct 31 13:44:28.658929 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.658809 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-istio-data\") pod \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\" (UID: \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\") " Oct 31 13:44:28.658929 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.658837 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-workload-socket\") pod \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\" (UID: \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\") " Oct 31 13:44:28.658929 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.658907 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-istio-envoy\") pod \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\" (UID: \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\") " Oct 31 13:44:28.659221 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.658946 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-istiod-ca-cert\") pod \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\" (UID: \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\") " Oct 31 13:44:28.659221 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.658978 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-workload-certs" pod "6ac22634-1f31-475f-ae5b-7bb13eb7c6b6" (UID: "6ac22634-1f31-475f-ae5b-7bb13eb7c6b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:28.659221 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.659016 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-istio-podinfo\") pod \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\" (UID: \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\") " Oct 31 13:44:28.659221 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.659195 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-credential-socket\") pod \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\" (UID: \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\") " Oct 31 13:44:28.659532 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.659239 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-workload-socket" pod "6ac22634-1f31-475f-ae5b-7bb13eb7c6b6" (UID: "6ac22634-1f31-475f-ae5b-7bb13eb7c6b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:28.659532 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.659249 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-istio-data" pod "6ac22634-1f31-475f-ae5b-7bb13eb7c6b6" (UID: "6ac22634-1f31-475f-ae5b-7bb13eb7c6b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:28.659532 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.659277 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-kube-api-access-8kww6\" (UniqueName: \"kubernetes.io/projected/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-kube-api-access-8kww6\") pod \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\" (UID: \"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6\") " Oct 31 13:44:28.659532 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.659469 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-istiod-ca-cert" pod "6ac22634-1f31-475f-ae5b-7bb13eb7c6b6" (UID: "6ac22634-1f31-475f-ae5b-7bb13eb7c6b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:28.659532 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.659484 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-credential-socket" pod "6ac22634-1f31-475f-ae5b-7bb13eb7c6b6" (UID: "6ac22634-1f31-475f-ae5b-7bb13eb7c6b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:28.659825 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.659778 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:28.659825 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.659795 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:28.659825 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.659808 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:28.659825 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.659822 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:28.660087 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.659832 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:28.661253 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.661224 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-istio-envoy" pod "6ac22634-1f31-475f-ae5b-7bb13eb7c6b6" (UID: "6ac22634-1f31-475f-ae5b-7bb13eb7c6b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:28.661367 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.661283 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-istio-podinfo" pod "6ac22634-1f31-475f-ae5b-7bb13eb7c6b6" (UID: "6ac22634-1f31-475f-ae5b-7bb13eb7c6b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:44:28.661520 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.661502 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-istio-token" pod "6ac22634-1f31-475f-ae5b-7bb13eb7c6b6" (UID: "6ac22634-1f31-475f-ae5b-7bb13eb7c6b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:28.661661 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.661639 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-kube-api-access-8kww6" pod "6ac22634-1f31-475f-ae5b-7bb13eb7c6b6" (UID: "6ac22634-1f31-475f-ae5b-7bb13eb7c6b6"). InnerVolumeSpecName "kube-api-access-8kww6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:28.761448 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.761355 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kww6\" (UniqueName: \"kubernetes.io/projected/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-kube-api-access-8kww6\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:28.761448 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.761390 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:28.761448 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.761405 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:28.761448 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.761420 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:28.778475 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.778440 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps"] Oct 31 13:44:28.778710 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.778686 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps" podUID="1faea580-200c-4d2e-8ead-ef3d6cdae95b" containerName="maas-api" containerID="cri-o://6b6f47a8c3f664651cafb448332fe42267e92a688534bd170e85fb36243195d4" gracePeriod=30 Oct 31 13:44:28.945474 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.945440 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:28.959777 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.959745 6841 generic.go:356] "Generic (PLEG): container finished" podID="1faea580-200c-4d2e-8ead-ef3d6cdae95b" containerID="6b6f47a8c3f664651cafb448332fe42267e92a688534bd170e85fb36243195d4" exitCode=0 Oct 31 13:44:28.959952 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.959821 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps" event={"ID":"1faea580-200c-4d2e-8ead-ef3d6cdae95b","Type":"ContainerDied","Data":"6b6f47a8c3f664651cafb448332fe42267e92a688534bd170e85fb36243195d4"} Oct 31 13:44:28.961474 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.961449 6841 generic.go:356] "Generic (PLEG): container finished" podID="6ac22634-1f31-475f-ae5b-7bb13eb7c6b6" containerID="c3e5a5502f6547128ea04d6f9cc100477289bd0147dbf50dc383ade051a3dac6" exitCode=0 Oct 31 13:44:28.961604 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.961508 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" Oct 31 13:44:28.961604 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.961518 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" event={"ID":"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6","Type":"ContainerDied","Data":"c3e5a5502f6547128ea04d6f9cc100477289bd0147dbf50dc383ade051a3dac6"} Oct 31 13:44:28.961604 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.961563 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn" event={"ID":"6ac22634-1f31-475f-ae5b-7bb13eb7c6b6","Type":"ContainerDied","Data":"5bc4b313e0a7f6cbad8e4480daaa215d8aec4442d154db42e2ed72e94f0f2b06"} Oct 31 13:44:28.961604 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.961601 6841 scope.go:122] "RemoveContainer" containerID="c3e5a5502f6547128ea04d6f9cc100477289bd0147dbf50dc383ade051a3dac6" Oct 31 13:44:28.976124 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.976102 6841 scope.go:122] "RemoveContainer" containerID="c3e5a5502f6547128ea04d6f9cc100477289bd0147dbf50dc383ade051a3dac6" Oct 31 13:44:28.976831 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:44:28.976764 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3e5a5502f6547128ea04d6f9cc100477289bd0147dbf50dc383ade051a3dac6\": container with ID starting with c3e5a5502f6547128ea04d6f9cc100477289bd0147dbf50dc383ade051a3dac6 not found: ID does not exist" containerID="c3e5a5502f6547128ea04d6f9cc100477289bd0147dbf50dc383ade051a3dac6" Oct 31 13:44:28.976831 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:28.976804 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3e5a5502f6547128ea04d6f9cc100477289bd0147dbf50dc383ade051a3dac6"} err="failed to get container status \"c3e5a5502f6547128ea04d6f9cc100477289bd0147dbf50dc383ade051a3dac6\": rpc error: code = NotFound desc = could not find container \"c3e5a5502f6547128ea04d6f9cc100477289bd0147dbf50dc383ade051a3dac6\": container with ID starting with c3e5a5502f6547128ea04d6f9cc100477289bd0147dbf50dc383ade051a3dac6 not found: ID does not exist" Oct 31 13:44:29.004556 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:29.004515 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn"] Oct 31 13:44:29.004825 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:29.004802 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxwdn"] Oct 31 13:44:29.077294 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:29.077261 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps" Oct 31 13:44:29.268470 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:29.268435 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1faea580-200c-4d2e-8ead-ef3d6cdae95b-kube-api-access-n7frw\" (UniqueName: \"kubernetes.io/projected/1faea580-200c-4d2e-8ead-ef3d6cdae95b-kube-api-access-n7frw\") pod \"1faea580-200c-4d2e-8ead-ef3d6cdae95b\" (UID: \"1faea580-200c-4d2e-8ead-ef3d6cdae95b\") " Oct 31 13:44:29.268972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:29.268509 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1faea580-200c-4d2e-8ead-ef3d6cdae95b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1faea580-200c-4d2e-8ead-ef3d6cdae95b-openshift-service-ca\") pod \"1faea580-200c-4d2e-8ead-ef3d6cdae95b\" (UID: \"1faea580-200c-4d2e-8ead-ef3d6cdae95b\") " Oct 31 13:44:29.268972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:29.268541 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1faea580-200c-4d2e-8ead-ef3d6cdae95b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1faea580-200c-4d2e-8ead-ef3d6cdae95b-maas-api-tls\") pod \"1faea580-200c-4d2e-8ead-ef3d6cdae95b\" (UID: \"1faea580-200c-4d2e-8ead-ef3d6cdae95b\") " Oct 31 13:44:29.268972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:29.268605 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1faea580-200c-4d2e-8ead-ef3d6cdae95b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1faea580-200c-4d2e-8ead-ef3d6cdae95b-metrics-certs\") pod \"1faea580-200c-4d2e-8ead-ef3d6cdae95b\" (UID: \"1faea580-200c-4d2e-8ead-ef3d6cdae95b\") " Oct 31 13:44:29.268972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:29.268948 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1faea580-200c-4d2e-8ead-ef3d6cdae95b-openshift-service-ca" pod "1faea580-200c-4d2e-8ead-ef3d6cdae95b" (UID: "1faea580-200c-4d2e-8ead-ef3d6cdae95b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:29.270701 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:29.270679 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1faea580-200c-4d2e-8ead-ef3d6cdae95b-maas-api-tls" pod "1faea580-200c-4d2e-8ead-ef3d6cdae95b" (UID: "1faea580-200c-4d2e-8ead-ef3d6cdae95b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:44:29.270786 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:29.270684 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1faea580-200c-4d2e-8ead-ef3d6cdae95b-kube-api-access-n7frw" pod "1faea580-200c-4d2e-8ead-ef3d6cdae95b" (UID: "1faea580-200c-4d2e-8ead-ef3d6cdae95b"). InnerVolumeSpecName "kube-api-access-n7frw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:29.270966 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:29.270948 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1faea580-200c-4d2e-8ead-ef3d6cdae95b-metrics-certs" pod "1faea580-200c-4d2e-8ead-ef3d6cdae95b" (UID: "1faea580-200c-4d2e-8ead-ef3d6cdae95b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:44:29.290435 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:29.290406 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:29.370487 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:29.370404 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7frw\" (UniqueName: \"kubernetes.io/projected/1faea580-200c-4d2e-8ead-ef3d6cdae95b-kube-api-access-n7frw\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:29.370487 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:29.370432 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1faea580-200c-4d2e-8ead-ef3d6cdae95b-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:29.370487 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:29.370444 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1faea580-200c-4d2e-8ead-ef3d6cdae95b-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:29.370487 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:29.370453 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1faea580-200c-4d2e-8ead-ef3d6cdae95b-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:29.643500 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:29.643421 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ac22634-1f31-475f-ae5b-7bb13eb7c6b6" path="/var/lib/kubelet/pods/6ac22634-1f31-475f-ae5b-7bb13eb7c6b6/volumes" Oct 31 13:44:29.696386 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:44:29.696355 6841 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1faea580_200c_4d2e_8ead_ef3d6cdae95b.slice\": RecentStats: unable to find data in memory cache]" Oct 31 13:44:29.969765 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:29.969721 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps" event={"ID":"1faea580-200c-4d2e-8ead-ef3d6cdae95b","Type":"ContainerDied","Data":"70c557c860b5a828771cbda805de72861f0be609abb485de4e927efaa84b2c45"} Oct 31 13:44:29.969765 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:29.969748 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps" Oct 31 13:44:29.970024 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:29.969831 6841 scope.go:122] "RemoveContainer" containerID="6b6f47a8c3f664651cafb448332fe42267e92a688534bd170e85fb36243195d4" Oct 31 13:44:29.989837 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:29.989807 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps"] Oct 31 13:44:29.992310 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:29.992283 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4574ec-8459578899-pdzps"] Oct 31 13:44:30.118165 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:30.118117 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:31.020960 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:31.020931 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:31.642177 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:31.642137 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1faea580-200c-4d2e-8ead-ef3d6cdae95b" path="/var/lib/kubelet/pods/1faea580-200c-4d2e-8ead-ef3d6cdae95b/volumes" Oct 31 13:44:31.876545 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:31.876510 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:32.528049 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:32.528013 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:33.799588 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:33.799555 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:34.959359 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:34.959325 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:35.182972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:35.182940 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:35.742599 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:35.742558 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:36.067757 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:36.067666 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:36.280781 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:36.280734 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:36.981433 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:36.981399 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:37.208208 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:37.208172 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:38.078746 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:38.078709 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:38.510511 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:38.510478 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:38.758599 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:38.758566 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:39.036659 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:39.036622 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:39.555845 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:39.555807 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8"] Oct 31 13:44:39.556319 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:39.556070 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8" podUID="4e342df0-b108-4369-9c0a-0a2bf7b850bf" containerName="maas-api" containerID="cri-o://2cdde0a7b67dcebd70babffc79ecbb4e02a68fb2e5db44ca56500acc2352809e" gracePeriod=30 Oct 31 13:44:39.697926 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:39.697893 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:39.855629 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:39.855605 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8" Oct 31 13:44:39.887138 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:39.887110 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e342df0-b108-4369-9c0a-0a2bf7b850bf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e342df0-b108-4369-9c0a-0a2bf7b850bf-metrics-certs\") pod \"4e342df0-b108-4369-9c0a-0a2bf7b850bf\" (UID: \"4e342df0-b108-4369-9c0a-0a2bf7b850bf\") " Oct 31 13:44:39.887294 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:39.887157 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e342df0-b108-4369-9c0a-0a2bf7b850bf-kube-api-access-c4brs\" (UniqueName: \"kubernetes.io/projected/4e342df0-b108-4369-9c0a-0a2bf7b850bf-kube-api-access-c4brs\") pod \"4e342df0-b108-4369-9c0a-0a2bf7b850bf\" (UID: \"4e342df0-b108-4369-9c0a-0a2bf7b850bf\") " Oct 31 13:44:39.887294 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:39.887263 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e342df0-b108-4369-9c0a-0a2bf7b850bf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e342df0-b108-4369-9c0a-0a2bf7b850bf-maas-api-tls\") pod \"4e342df0-b108-4369-9c0a-0a2bf7b850bf\" (UID: \"4e342df0-b108-4369-9c0a-0a2bf7b850bf\") " Oct 31 13:44:39.887429 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:39.887392 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e342df0-b108-4369-9c0a-0a2bf7b850bf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e342df0-b108-4369-9c0a-0a2bf7b850bf-openshift-service-ca\") pod \"4e342df0-b108-4369-9c0a-0a2bf7b850bf\" (UID: \"4e342df0-b108-4369-9c0a-0a2bf7b850bf\") " Oct 31 13:44:39.887832 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:39.887806 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e342df0-b108-4369-9c0a-0a2bf7b850bf-openshift-service-ca" pod "4e342df0-b108-4369-9c0a-0a2bf7b850bf" (UID: "4e342df0-b108-4369-9c0a-0a2bf7b850bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:39.889703 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:39.889674 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e342df0-b108-4369-9c0a-0a2bf7b850bf-kube-api-access-c4brs" pod "4e342df0-b108-4369-9c0a-0a2bf7b850bf" (UID: "4e342df0-b108-4369-9c0a-0a2bf7b850bf"). InnerVolumeSpecName "kube-api-access-c4brs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:39.889829 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:39.889792 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e342df0-b108-4369-9c0a-0a2bf7b850bf-maas-api-tls" pod "4e342df0-b108-4369-9c0a-0a2bf7b850bf" (UID: "4e342df0-b108-4369-9c0a-0a2bf7b850bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:44:39.889902 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:39.889824 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e342df0-b108-4369-9c0a-0a2bf7b850bf-metrics-certs" pod "4e342df0-b108-4369-9c0a-0a2bf7b850bf" (UID: "4e342df0-b108-4369-9c0a-0a2bf7b850bf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:44:39.988647 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:39.988617 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e342df0-b108-4369-9c0a-0a2bf7b850bf-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:39.988647 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:39.988644 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e342df0-b108-4369-9c0a-0a2bf7b850bf-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:39.988647 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:39.988653 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e342df0-b108-4369-9c0a-0a2bf7b850bf-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:39.988886 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:39.988661 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4brs\" (UniqueName: \"kubernetes.io/projected/4e342df0-b108-4369-9c0a-0a2bf7b850bf-kube-api-access-c4brs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:40.043648 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:40.043612 6841 generic.go:356] "Generic (PLEG): container finished" podID="4e342df0-b108-4369-9c0a-0a2bf7b850bf" containerID="2cdde0a7b67dcebd70babffc79ecbb4e02a68fb2e5db44ca56500acc2352809e" exitCode=0 Oct 31 13:44:40.043841 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:40.043655 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8" Oct 31 13:44:40.043841 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:40.043706 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8" event={"ID":"4e342df0-b108-4369-9c0a-0a2bf7b850bf","Type":"ContainerDied","Data":"2cdde0a7b67dcebd70babffc79ecbb4e02a68fb2e5db44ca56500acc2352809e"} Oct 31 13:44:40.043841 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:40.043755 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8" event={"ID":"4e342df0-b108-4369-9c0a-0a2bf7b850bf","Type":"ContainerDied","Data":"610b81ae9ebcdddaff6d6f075ceacc497f0b221509adf87eb6986a82dcb99b3e"} Oct 31 13:44:40.043841 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:40.043786 6841 scope.go:122] "RemoveContainer" containerID="2cdde0a7b67dcebd70babffc79ecbb4e02a68fb2e5db44ca56500acc2352809e" Oct 31 13:44:40.054052 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:40.054016 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:40.055029 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:40.054996 6841 scope.go:122] "RemoveContainer" containerID="2cdde0a7b67dcebd70babffc79ecbb4e02a68fb2e5db44ca56500acc2352809e" Oct 31 13:44:40.055295 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:44:40.055273 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cdde0a7b67dcebd70babffc79ecbb4e02a68fb2e5db44ca56500acc2352809e\": container with ID starting with 2cdde0a7b67dcebd70babffc79ecbb4e02a68fb2e5db44ca56500acc2352809e not found: ID does not exist" containerID="2cdde0a7b67dcebd70babffc79ecbb4e02a68fb2e5db44ca56500acc2352809e" Oct 31 13:44:40.055354 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:40.055302 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cdde0a7b67dcebd70babffc79ecbb4e02a68fb2e5db44ca56500acc2352809e"} err="failed to get container status \"2cdde0a7b67dcebd70babffc79ecbb4e02a68fb2e5db44ca56500acc2352809e\": rpc error: code = NotFound desc = could not find container \"2cdde0a7b67dcebd70babffc79ecbb4e02a68fb2e5db44ca56500acc2352809e\": container with ID starting with 2cdde0a7b67dcebd70babffc79ecbb4e02a68fb2e5db44ca56500acc2352809e not found: ID does not exist" Oct 31 13:44:40.070101 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:40.070076 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8"] Oct 31 13:44:40.072875 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:40.072842 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-3dd1e5-79c859dc77-82tz8"] Oct 31 13:44:40.830322 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:40.830288 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:41.117230 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:41.117146 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:41.557798 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:41.557762 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:41.641543 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:41.641513 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e342df0-b108-4369-9c0a-0a2bf7b850bf" path="/var/lib/kubelet/pods/4e342df0-b108-4369-9c0a-0a2bf7b850bf/volumes" Oct 31 13:44:42.361141 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:42.361106 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:42.623781 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:42.623667 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:43.335350 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:43.335315 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:43.587342 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:43.587248 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:44.495952 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:44.495918 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:45.547954 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:45.547920 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:45.799319 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:45.799207 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:46.426465 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:46.426432 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:47.303770 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:47.303730 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:48.140557 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:48.140524 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:48.950764 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:48.950723 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:49.469605 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:49.469575 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq"] Oct 31 13:44:49.469797 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:49.469773 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq" podUID="35a2efb2-e354-4258-811e-d9ec46ef48a0" containerName="maas-api" containerID="cri-o://dd3fcaccc55b4c1fffc4e855cf45156c46dbcc03ef080913d96d56117259dedc" gracePeriod=30 Oct 31 13:44:49.630086 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:49.630046 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:49.774779 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:49.774750 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq" Oct 31 13:44:49.900290 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:49.900260 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35a2efb2-e354-4258-811e-d9ec46ef48a0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a2efb2-e354-4258-811e-d9ec46ef48a0-openshift-service-ca\") pod \"35a2efb2-e354-4258-811e-d9ec46ef48a0\" (UID: \"35a2efb2-e354-4258-811e-d9ec46ef48a0\") " Oct 31 13:44:49.900465 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:49.900300 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35a2efb2-e354-4258-811e-d9ec46ef48a0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35a2efb2-e354-4258-811e-d9ec46ef48a0-maas-api-tls\") pod \"35a2efb2-e354-4258-811e-d9ec46ef48a0\" (UID: \"35a2efb2-e354-4258-811e-d9ec46ef48a0\") " Oct 31 13:44:49.900465 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:49.900347 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35a2efb2-e354-4258-811e-d9ec46ef48a0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/35a2efb2-e354-4258-811e-d9ec46ef48a0-metrics-certs\") pod \"35a2efb2-e354-4258-811e-d9ec46ef48a0\" (UID: \"35a2efb2-e354-4258-811e-d9ec46ef48a0\") " Oct 31 13:44:49.900465 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:49.900382 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35a2efb2-e354-4258-811e-d9ec46ef48a0-kube-api-access-dlmxl\" (UniqueName: \"kubernetes.io/projected/35a2efb2-e354-4258-811e-d9ec46ef48a0-kube-api-access-dlmxl\") pod \"35a2efb2-e354-4258-811e-d9ec46ef48a0\" (UID: \"35a2efb2-e354-4258-811e-d9ec46ef48a0\") " Oct 31 13:44:49.900642 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:49.900602 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35a2efb2-e354-4258-811e-d9ec46ef48a0-openshift-service-ca" pod "35a2efb2-e354-4258-811e-d9ec46ef48a0" (UID: "35a2efb2-e354-4258-811e-d9ec46ef48a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:49.900761 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:49.900740 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a2efb2-e354-4258-811e-d9ec46ef48a0-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:49.902539 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:49.902510 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a2efb2-e354-4258-811e-d9ec46ef48a0-metrics-certs" pod "35a2efb2-e354-4258-811e-d9ec46ef48a0" (UID: "35a2efb2-e354-4258-811e-d9ec46ef48a0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:44:49.902648 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:49.902620 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a2efb2-e354-4258-811e-d9ec46ef48a0-kube-api-access-dlmxl" pod "35a2efb2-e354-4258-811e-d9ec46ef48a0" (UID: "35a2efb2-e354-4258-811e-d9ec46ef48a0"). InnerVolumeSpecName "kube-api-access-dlmxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:49.902648 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:49.902622 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a2efb2-e354-4258-811e-d9ec46ef48a0-maas-api-tls" pod "35a2efb2-e354-4258-811e-d9ec46ef48a0" (UID: "35a2efb2-e354-4258-811e-d9ec46ef48a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:44:50.002015 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:50.001904 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35a2efb2-e354-4258-811e-d9ec46ef48a0-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:50.002015 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:50.001931 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35a2efb2-e354-4258-811e-d9ec46ef48a0-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:50.002015 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:50.001942 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlmxl\" (UniqueName: \"kubernetes.io/projected/35a2efb2-e354-4258-811e-d9ec46ef48a0-kube-api-access-dlmxl\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:50.034384 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:44:50.034347 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:44:50.103252 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:50.103223 6841 generic.go:356] "Generic (PLEG): container finished" podID="35a2efb2-e354-4258-811e-d9ec46ef48a0" containerID="dd3fcaccc55b4c1fffc4e855cf45156c46dbcc03ef080913d96d56117259dedc" exitCode=0 Oct 31 13:44:50.103452 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:50.103256 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq" event={"ID":"35a2efb2-e354-4258-811e-d9ec46ef48a0","Type":"ContainerDied","Data":"dd3fcaccc55b4c1fffc4e855cf45156c46dbcc03ef080913d96d56117259dedc"} Oct 31 13:44:50.103452 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:50.103282 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq" Oct 31 13:44:50.103452 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:50.103339 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq" event={"ID":"35a2efb2-e354-4258-811e-d9ec46ef48a0","Type":"ContainerDied","Data":"c5ac30b5ba5f086aea6adaa9fab32322240f153bd1f2f3642adbae3a02f6dd75"} Oct 31 13:44:50.103452 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:50.103362 6841 scope.go:122] "RemoveContainer" containerID="dd3fcaccc55b4c1fffc4e855cf45156c46dbcc03ef080913d96d56117259dedc" Oct 31 13:44:50.114489 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:50.114472 6841 scope.go:122] "RemoveContainer" containerID="dd3fcaccc55b4c1fffc4e855cf45156c46dbcc03ef080913d96d56117259dedc" Oct 31 13:44:50.114721 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:44:50.114702 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd3fcaccc55b4c1fffc4e855cf45156c46dbcc03ef080913d96d56117259dedc\": container with ID starting with dd3fcaccc55b4c1fffc4e855cf45156c46dbcc03ef080913d96d56117259dedc not found: ID does not exist" containerID="dd3fcaccc55b4c1fffc4e855cf45156c46dbcc03ef080913d96d56117259dedc" Oct 31 13:44:50.114767 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:50.114728 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd3fcaccc55b4c1fffc4e855cf45156c46dbcc03ef080913d96d56117259dedc"} err="failed to get container status \"dd3fcaccc55b4c1fffc4e855cf45156c46dbcc03ef080913d96d56117259dedc\": rpc error: code = NotFound desc = could not find container \"dd3fcaccc55b4c1fffc4e855cf45156c46dbcc03ef080913d96d56117259dedc\": container with ID starting with dd3fcaccc55b4c1fffc4e855cf45156c46dbcc03ef080913d96d56117259dedc not found: ID does not exist" Oct 31 13:44:50.127248 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:50.127224 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq"] Oct 31 13:44:50.134875 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:50.134854 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5452fc-996757c87-qqjwq"] Oct 31 13:44:50.149060 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:50.149038 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:50.949762 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:50.949728 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:51.641242 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:51.641211 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35a2efb2-e354-4258-811e-d9ec46ef48a0" path="/var/lib/kubelet/pods/35a2efb2-e354-4258-811e-d9ec46ef48a0/volumes" Oct 31 13:44:51.740136 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:51.740098 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:52.498203 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:52.498173 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:52.745767 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:52.745733 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:53.177256 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.177222 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:53.453131 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.453100 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh"] Oct 31 13:44:53.453349 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.453326 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" podUID="f7095807-f9e9-47ba-be79-262d1b097fc2" containerName="istio-proxy" containerID="cri-o://65314cc449d23fc5fda8ca78706aee00cae11f94f62b53d31d9d33193838f6cf" gracePeriod=30 Oct 31 13:44:53.528228 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528194 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6"] Oct 31 13:44:53.528661 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528648 6841 cpu_manager.go:414] "containerMap: removing container" podUID="35a2efb2-e354-4258-811e-d9ec46ef48a0" containerName="maas-api" Oct 31 13:44:53.528717 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528663 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35a2efb2-e354-4258-811e-d9ec46ef48a0" containerName="maas-api" Oct 31 13:44:53.528717 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528680 6841 cpu_manager.go:414] "containerMap: removing container" podUID="96229665-4486-4750-becf-d30f28c47dd5" containerName="payload-pre-processing" Oct 31 13:44:53.528717 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528686 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96229665-4486-4750-becf-d30f28c47dd5" containerName="payload-pre-processing" Oct 31 13:44:53.528717 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528696 6841 cpu_manager.go:414] "containerMap: removing container" podUID="4e342df0-b108-4369-9c0a-0a2bf7b850bf" containerName="maas-api" Oct 31 13:44:53.528717 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528701 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e342df0-b108-4369-9c0a-0a2bf7b850bf" containerName="maas-api" Oct 31 13:44:53.528717 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528712 6841 cpu_manager.go:414] "containerMap: removing container" podUID="1faea580-200c-4d2e-8ead-ef3d6cdae95b" containerName="maas-api" Oct 31 13:44:53.528717 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528716 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1faea580-200c-4d2e-8ead-ef3d6cdae95b" containerName="maas-api" Oct 31 13:44:53.528972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528731 6841 cpu_manager.go:414] "containerMap: removing container" podUID="c85bee9f-4267-44ec-a03f-0b97c6b95d24" containerName="istio-proxy" Oct 31 13:44:53.528972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528736 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c85bee9f-4267-44ec-a03f-0b97c6b95d24" containerName="istio-proxy" Oct 31 13:44:53.528972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528743 6841 cpu_manager.go:414] "containerMap: removing container" podUID="6e2a46f8-aab8-482f-8edb-67800fa4960e" containerName="istio-proxy" Oct 31 13:44:53.528972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528748 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e2a46f8-aab8-482f-8edb-67800fa4960e" containerName="istio-proxy" Oct 31 13:44:53.528972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528758 6841 cpu_manager.go:414] "containerMap: removing container" podUID="e4e95366-875b-4fda-8e1b-6e441df2f5b2" containerName="istio-proxy" Oct 31 13:44:53.528972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528762 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4e95366-875b-4fda-8e1b-6e441df2f5b2" containerName="istio-proxy" Oct 31 13:44:53.528972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528769 6841 cpu_manager.go:414] "containerMap: removing container" podUID="6ac22634-1f31-475f-ae5b-7bb13eb7c6b6" containerName="istio-proxy" Oct 31 13:44:53.528972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528773 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ac22634-1f31-475f-ae5b-7bb13eb7c6b6" containerName="istio-proxy" Oct 31 13:44:53.528972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528782 6841 cpu_manager.go:414] "containerMap: removing container" podUID="68db5d02-47c3-4605-8967-30a007c782a7" containerName="payload-pre-processing" Oct 31 13:44:53.528972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528787 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68db5d02-47c3-4605-8967-30a007c782a7" containerName="payload-pre-processing" Oct 31 13:44:53.528972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528799 6841 cpu_manager.go:414] "containerMap: removing container" podUID="e4e95366-875b-4fda-8e1b-6e441df2f5b2" containerName="istio-proxy" Oct 31 13:44:53.528972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528804 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4e95366-875b-4fda-8e1b-6e441df2f5b2" containerName="istio-proxy" Oct 31 13:44:53.528972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528813 6841 cpu_manager.go:414] "containerMap: removing container" podUID="aa093405-83f7-4d3e-b984-bb188e5171d7" containerName="maas-api" Oct 31 13:44:53.528972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528817 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa093405-83f7-4d3e-b984-bb188e5171d7" containerName="maas-api" Oct 31 13:44:53.528972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528825 6841 cpu_manager.go:414] "containerMap: removing container" podUID="fb6f239c-4a3d-4f98-9b5e-4d775a4c561c" containerName="payload-processing" Oct 31 13:44:53.528972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528830 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb6f239c-4a3d-4f98-9b5e-4d775a4c561c" containerName="payload-processing" Oct 31 13:44:53.528972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528837 6841 cpu_manager.go:414] "containerMap: removing container" podUID="a3a899eb-181a-4f40-a64e-2953e94e6dd4" containerName="istio-proxy" Oct 31 13:44:53.528972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528841 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3a899eb-181a-4f40-a64e-2953e94e6dd4" containerName="istio-proxy" Oct 31 13:44:53.528972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528849 6841 cpu_manager.go:414] "containerMap: removing container" podUID="cfd61135-1d8a-44a5-9771-a66f1c88cd18" containerName="istio-proxy" Oct 31 13:44:53.528972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528853 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfd61135-1d8a-44a5-9771-a66f1c88cd18" containerName="istio-proxy" Oct 31 13:44:53.528972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528860 6841 cpu_manager.go:414] "containerMap: removing container" podUID="8a0d55fc-97da-4fba-a45a-68a229f87264" containerName="istio-proxy" Oct 31 13:44:53.528972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528864 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a0d55fc-97da-4fba-a45a-68a229f87264" containerName="istio-proxy" Oct 31 13:44:53.528972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528870 6841 cpu_manager.go:414] "containerMap: removing container" podUID="cfd61135-1d8a-44a5-9771-a66f1c88cd18" containerName="istio-proxy" Oct 31 13:44:53.528972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528874 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfd61135-1d8a-44a5-9771-a66f1c88cd18" containerName="istio-proxy" Oct 31 13:44:53.528972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528881 6841 cpu_manager.go:414] "containerMap: removing container" podUID="c85bee9f-4267-44ec-a03f-0b97c6b95d24" containerName="istio-proxy" Oct 31 13:44:53.528972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528886 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c85bee9f-4267-44ec-a03f-0b97c6b95d24" containerName="istio-proxy" Oct 31 13:44:53.528972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528893 6841 cpu_manager.go:414] "containerMap: removing container" podUID="481838f4-af98-4e2e-8f4e-2db1897925c4" containerName="maas-api" Oct 31 13:44:53.528972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528897 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="481838f4-af98-4e2e-8f4e-2db1897925c4" containerName="maas-api" Oct 31 13:44:53.528972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528966 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4e95366-875b-4fda-8e1b-6e441df2f5b2" containerName="istio-proxy" Oct 31 13:44:53.528972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528974 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ac22634-1f31-475f-ae5b-7bb13eb7c6b6" containerName="istio-proxy" Oct 31 13:44:53.528972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.528997 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="68db5d02-47c3-4605-8967-30a007c782a7" containerName="payload-pre-processing" Oct 31 13:44:53.528972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.529006 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e2a46f8-aab8-482f-8edb-67800fa4960e" containerName="istio-proxy" Oct 31 13:44:53.528972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.529012 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa093405-83f7-4d3e-b984-bb188e5171d7" containerName="maas-api" Oct 31 13:44:53.528972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.529018 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb6f239c-4a3d-4f98-9b5e-4d775a4c561c" containerName="payload-processing" Oct 31 13:44:53.530157 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.529025 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3a899eb-181a-4f40-a64e-2953e94e6dd4" containerName="istio-proxy" Oct 31 13:44:53.530157 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.529032 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfd61135-1d8a-44a5-9771-a66f1c88cd18" containerName="istio-proxy" Oct 31 13:44:53.530157 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.529037 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a0d55fc-97da-4fba-a45a-68a229f87264" containerName="istio-proxy" Oct 31 13:44:53.530157 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.529042 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfd61135-1d8a-44a5-9771-a66f1c88cd18" containerName="istio-proxy" Oct 31 13:44:53.530157 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.529047 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="c85bee9f-4267-44ec-a03f-0b97c6b95d24" containerName="istio-proxy" Oct 31 13:44:53.530157 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.529053 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="481838f4-af98-4e2e-8f4e-2db1897925c4" containerName="maas-api" Oct 31 13:44:53.530157 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.529059 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="35a2efb2-e354-4258-811e-d9ec46ef48a0" containerName="maas-api" Oct 31 13:44:53.530157 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.529064 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="96229665-4486-4750-becf-d30f28c47dd5" containerName="payload-pre-processing" Oct 31 13:44:53.530157 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.529070 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e342df0-b108-4369-9c0a-0a2bf7b850bf" containerName="maas-api" Oct 31 13:44:53.530157 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.529076 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="1faea580-200c-4d2e-8ead-ef3d6cdae95b" containerName="maas-api" Oct 31 13:44:53.533711 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.533690 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" Oct 31 13:44:53.546704 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.541284 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-3dd1e5-openshift-default-dockercfg-ktcdl\"" Oct 31 13:44:53.554549 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.554094 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6"] Oct 31 13:44:53.619291 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.619230 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:53.640462 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.640432 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-credential-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6\" (UID: \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" Oct 31 13:44:53.640645 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.640493 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-istiod-ca-cert\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6\" (UID: \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" Oct 31 13:44:53.640645 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.640556 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-istio-data\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6\" (UID: \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" Oct 31 13:44:53.640645 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.640640 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-istio-token\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6\" (UID: \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" Oct 31 13:44:53.640831 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.640680 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-istio-envoy\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6\" (UID: \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" Oct 31 13:44:53.640831 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.640775 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-istio-podinfo\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6\" (UID: \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" Oct 31 13:44:53.640831 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.640819 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p685k\" (UniqueName: \"kubernetes.io/projected/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-kube-api-access-p685k\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6\" (UID: \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" Oct 31 13:44:53.640969 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.640860 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-workload-certs\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6\" (UID: \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" Oct 31 13:44:53.640969 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.640881 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-workload-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6\" (UID: \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" Oct 31 13:44:53.742136 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.742060 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-credential-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6\" (UID: \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" Oct 31 13:44:53.742136 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.742106 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-istiod-ca-cert\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6\" (UID: \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" Oct 31 13:44:53.742353 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.742140 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-istio-data\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6\" (UID: \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" Oct 31 13:44:53.742353 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.742189 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-istio-token\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6\" (UID: \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" Oct 31 13:44:53.742353 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.742218 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-istio-envoy\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6\" (UID: \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" Oct 31 13:44:53.742353 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.742251 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-istio-podinfo\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6\" (UID: \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" Oct 31 13:44:53.742353 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.742301 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p685k\" (UniqueName: \"kubernetes.io/projected/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-kube-api-access-p685k\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6\" (UID: \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" Oct 31 13:44:53.742669 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.742365 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-workload-certs\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6\" (UID: \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" Oct 31 13:44:53.742669 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.742392 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-workload-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6\" (UID: \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" Oct 31 13:44:53.742789 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.742711 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-credential-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6\" (UID: \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" Oct 31 13:44:53.742923 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.742890 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-istiod-ca-cert\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6\" (UID: \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" Oct 31 13:44:53.743085 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.742962 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-workload-certs\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6\" (UID: \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" Oct 31 13:44:53.743085 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.743060 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-istio-data\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6\" (UID: \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" Oct 31 13:44:53.743235 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.743078 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-workload-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6\" (UID: \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" Oct 31 13:44:53.744727 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.744704 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-istio-envoy\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6\" (UID: \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" Oct 31 13:44:53.744870 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.744851 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-istio-podinfo\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6\" (UID: \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" Oct 31 13:44:53.762308 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.762281 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p685k\" (UniqueName: \"kubernetes.io/projected/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-kube-api-access-p685k\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6\" (UID: \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" Oct 31 13:44:53.765694 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.765669 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-istio-token\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6\" (UID: \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" Oct 31 13:44:53.874577 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:53.874527 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" Oct 31 13:44:54.148490 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:54.148371 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:54.167956 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:54.167822 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6"] Oct 31 13:44:54.174538 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:54.174496 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:44:54.174647 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:54.174584 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:44:54.174647 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:54.174624 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:44:54.385021 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:54.384188 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:54.995206 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:54.995174 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:55.136359 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:55.136323 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" event={"ID":"c1e1f66f-b5a1-4958-a5bf-097e3ee91884","Type":"ContainerStarted","Data":"aa71596758bfa988dfa77f166a988af2349c0bf59a1a589c1944dbf817dbba41"} Oct 31 13:44:55.136514 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:55.136376 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" event={"ID":"c1e1f66f-b5a1-4958-a5bf-097e3ee91884","Type":"ContainerStarted","Data":"f9d3edd8b9de5532cd4a272c53bd5d4183db57383430ef0aa8ab9eab4b287f9c"} Oct 31 13:44:55.190079 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:55.189174 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" podStartSLOduration=2.189156694 podStartE2EDuration="2.189156694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:44:55.181608246 +0000 UTC m=+4341.664731149" watchObservedRunningTime="2026-10-31 13:44:55.189156694 +0000 UTC m=+4341.672279596" Oct 31 13:44:55.232436 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:55.232399 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6"] Oct 31 13:44:55.282931 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:55.282834 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:55.538881 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:55.538774 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:55.875544 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:55.875453 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" Oct 31 13:44:55.876969 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:55.876941 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body= Oct 31 13:44:55.877141 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:55.877022 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" podUID="c1e1f66f-b5a1-4958-a5bf-097e3ee91884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" Oct 31 13:44:56.148064 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:56.142444 6841 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" secret="" err="secret \"e2e-worker-w2-3dd1e5-openshift-default-dockercfg-ktcdl\" not found" Oct 31 13:44:56.267548 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:56.267514 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:56.875232 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:56.875190 6841 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body= Oct 31 13:44:56.875523 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:56.875264 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" podUID="c1e1f66f-b5a1-4958-a5bf-097e3ee91884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" Oct 31 13:44:57.174317 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:57.174216 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" podUID="c1e1f66f-b5a1-4958-a5bf-097e3ee91884" containerName="istio-proxy" containerID="cri-o://aa71596758bfa988dfa77f166a988af2349c0bf59a1a589c1944dbf817dbba41" gracePeriod=30 Oct 31 13:44:57.328888 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:57.328841 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:58.393758 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:58.393721 6841 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:44:58.394197 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:58.393788 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" podUID="f7095807-f9e9-47ba-be79-262d1b097fc2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:44:58.627388 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:58.627354 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:58.915480 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:58.915450 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:59.170677 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.169042 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:59.179366 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.179342 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" Oct 31 13:44:59.186180 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.186152 6841 generic.go:356] "Generic (PLEG): container finished" podID="f7095807-f9e9-47ba-be79-262d1b097fc2" containerID="65314cc449d23fc5fda8ca78706aee00cae11f94f62b53d31d9d33193838f6cf" exitCode=0 Oct 31 13:44:59.186313 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.186200 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" Oct 31 13:44:59.186313 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.186228 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" event={"ID":"f7095807-f9e9-47ba-be79-262d1b097fc2","Type":"ContainerDied","Data":"65314cc449d23fc5fda8ca78706aee00cae11f94f62b53d31d9d33193838f6cf"} Oct 31 13:44:59.186313 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.186290 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh" event={"ID":"f7095807-f9e9-47ba-be79-262d1b097fc2","Type":"ContainerDied","Data":"fec811275179fc5e8f34bfb0880bcb7b318056adcf826bc232be4c759c12910b"} Oct 31 13:44:59.186425 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.186313 6841 scope.go:122] "RemoveContainer" containerID="65314cc449d23fc5fda8ca78706aee00cae11f94f62b53d31d9d33193838f6cf" Oct 31 13:44:59.195470 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.195452 6841 scope.go:122] "RemoveContainer" containerID="65314cc449d23fc5fda8ca78706aee00cae11f94f62b53d31d9d33193838f6cf" Oct 31 13:44:59.195730 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:44:59.195712 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65314cc449d23fc5fda8ca78706aee00cae11f94f62b53d31d9d33193838f6cf\": container with ID starting with 65314cc449d23fc5fda8ca78706aee00cae11f94f62b53d31d9d33193838f6cf not found: ID does not exist" containerID="65314cc449d23fc5fda8ca78706aee00cae11f94f62b53d31d9d33193838f6cf" Oct 31 13:44:59.195798 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.195738 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65314cc449d23fc5fda8ca78706aee00cae11f94f62b53d31d9d33193838f6cf"} err="failed to get container status \"65314cc449d23fc5fda8ca78706aee00cae11f94f62b53d31d9d33193838f6cf\": rpc error: code = NotFound desc = could not find container \"65314cc449d23fc5fda8ca78706aee00cae11f94f62b53d31d9d33193838f6cf\": container with ID starting with 65314cc449d23fc5fda8ca78706aee00cae11f94f62b53d31d9d33193838f6cf not found: ID does not exist" Oct 31 13:44:59.324513 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.324482 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7095807-f9e9-47ba-be79-262d1b097fc2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7095807-f9e9-47ba-be79-262d1b097fc2-workload-certs\") pod \"f7095807-f9e9-47ba-be79-262d1b097fc2\" (UID: \"f7095807-f9e9-47ba-be79-262d1b097fc2\") " Oct 31 13:44:59.324685 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.324551 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7095807-f9e9-47ba-be79-262d1b097fc2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7095807-f9e9-47ba-be79-262d1b097fc2-istio-data\") pod \"f7095807-f9e9-47ba-be79-262d1b097fc2\" (UID: \"f7095807-f9e9-47ba-be79-262d1b097fc2\") " Oct 31 13:44:59.324685 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.324583 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7095807-f9e9-47ba-be79-262d1b097fc2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7095807-f9e9-47ba-be79-262d1b097fc2-istio-podinfo\") pod \"f7095807-f9e9-47ba-be79-262d1b097fc2\" (UID: \"f7095807-f9e9-47ba-be79-262d1b097fc2\") " Oct 31 13:44:59.324685 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.324639 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7095807-f9e9-47ba-be79-262d1b097fc2-istio-token\" (UniqueName: \"kubernetes.io/projected/f7095807-f9e9-47ba-be79-262d1b097fc2-istio-token\") pod \"f7095807-f9e9-47ba-be79-262d1b097fc2\" (UID: \"f7095807-f9e9-47ba-be79-262d1b097fc2\") " Oct 31 13:44:59.324685 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.324673 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7095807-f9e9-47ba-be79-262d1b097fc2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7095807-f9e9-47ba-be79-262d1b097fc2-workload-socket\") pod \"f7095807-f9e9-47ba-be79-262d1b097fc2\" (UID: \"f7095807-f9e9-47ba-be79-262d1b097fc2\") " Oct 31 13:44:59.324937 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.324789 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7095807-f9e9-47ba-be79-262d1b097fc2-workload-certs" pod "f7095807-f9e9-47ba-be79-262d1b097fc2" (UID: "f7095807-f9e9-47ba-be79-262d1b097fc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:59.324937 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.324832 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7095807-f9e9-47ba-be79-262d1b097fc2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7095807-f9e9-47ba-be79-262d1b097fc2-credential-socket\") pod \"f7095807-f9e9-47ba-be79-262d1b097fc2\" (UID: \"f7095807-f9e9-47ba-be79-262d1b097fc2\") " Oct 31 13:44:59.324937 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.324865 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7095807-f9e9-47ba-be79-262d1b097fc2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7095807-f9e9-47ba-be79-262d1b097fc2-istiod-ca-cert\") pod \"f7095807-f9e9-47ba-be79-262d1b097fc2\" (UID: \"f7095807-f9e9-47ba-be79-262d1b097fc2\") " Oct 31 13:44:59.324937 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.324926 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7095807-f9e9-47ba-be79-262d1b097fc2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7095807-f9e9-47ba-be79-262d1b097fc2-istio-envoy\") pod \"f7095807-f9e9-47ba-be79-262d1b097fc2\" (UID: \"f7095807-f9e9-47ba-be79-262d1b097fc2\") " Oct 31 13:44:59.325218 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.324854 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7095807-f9e9-47ba-be79-262d1b097fc2-istio-data" pod "f7095807-f9e9-47ba-be79-262d1b097fc2" (UID: "f7095807-f9e9-47ba-be79-262d1b097fc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:59.325218 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.324960 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7095807-f9e9-47ba-be79-262d1b097fc2-kube-api-access-br2fh\" (UniqueName: \"kubernetes.io/projected/f7095807-f9e9-47ba-be79-262d1b097fc2-kube-api-access-br2fh\") pod \"f7095807-f9e9-47ba-be79-262d1b097fc2\" (UID: \"f7095807-f9e9-47ba-be79-262d1b097fc2\") " Oct 31 13:44:59.325218 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.325085 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7095807-f9e9-47ba-be79-262d1b097fc2-workload-socket" pod "f7095807-f9e9-47ba-be79-262d1b097fc2" (UID: "f7095807-f9e9-47ba-be79-262d1b097fc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:59.325218 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.325107 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7095807-f9e9-47ba-be79-262d1b097fc2-credential-socket" pod "f7095807-f9e9-47ba-be79-262d1b097fc2" (UID: "f7095807-f9e9-47ba-be79-262d1b097fc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:59.325531 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.325241 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7095807-f9e9-47ba-be79-262d1b097fc2-istiod-ca-cert" pod "f7095807-f9e9-47ba-be79-262d1b097fc2" (UID: "f7095807-f9e9-47ba-be79-262d1b097fc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:59.325531 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.325410 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7095807-f9e9-47ba-be79-262d1b097fc2-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:59.325531 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.325429 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7095807-f9e9-47ba-be79-262d1b097fc2-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:59.325531 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.325445 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7095807-f9e9-47ba-be79-262d1b097fc2-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:59.325531 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.325459 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7095807-f9e9-47ba-be79-262d1b097fc2-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:59.325531 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.325472 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7095807-f9e9-47ba-be79-262d1b097fc2-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:59.327018 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.326993 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7095807-f9e9-47ba-be79-262d1b097fc2-istio-podinfo" pod "f7095807-f9e9-47ba-be79-262d1b097fc2" (UID: "f7095807-f9e9-47ba-be79-262d1b097fc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:44:59.327135 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.327072 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7095807-f9e9-47ba-be79-262d1b097fc2-istio-token" pod "f7095807-f9e9-47ba-be79-262d1b097fc2" (UID: "f7095807-f9e9-47ba-be79-262d1b097fc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:59.327264 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.327239 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7095807-f9e9-47ba-be79-262d1b097fc2-kube-api-access-br2fh" pod "f7095807-f9e9-47ba-be79-262d1b097fc2" (UID: "f7095807-f9e9-47ba-be79-262d1b097fc2"). InnerVolumeSpecName "kube-api-access-br2fh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:59.327386 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.327271 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7095807-f9e9-47ba-be79-262d1b097fc2-istio-envoy" pod "f7095807-f9e9-47ba-be79-262d1b097fc2" (UID: "f7095807-f9e9-47ba-be79-262d1b097fc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:59.398878 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.398843 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:44:59.426630 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.426567 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7095807-f9e9-47ba-be79-262d1b097fc2-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:59.426630 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.426591 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7095807-f9e9-47ba-be79-262d1b097fc2-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:59.426630 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.426600 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7095807-f9e9-47ba-be79-262d1b097fc2-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:59.426630 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.426610 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br2fh\" (UniqueName: \"kubernetes.io/projected/f7095807-f9e9-47ba-be79-262d1b097fc2-kube-api-access-br2fh\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:59.529370 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.529336 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh"] Oct 31 13:44:59.537203 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.537173 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-nh6gh"] Oct 31 13:44:59.645006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.642354 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7095807-f9e9-47ba-be79-262d1b097fc2" path="/var/lib/kubelet/pods/f7095807-f9e9-47ba-be79-262d1b097fc2/volumes" Oct 31 13:44:59.792313 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:44:59.792281 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:00.022356 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.022323 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:00.173180 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.173107 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890905-jwdgm"] Oct 31 13:45:00.174611 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.174581 6841 cpu_manager.go:414] "containerMap: removing container" podUID="6e2a46f8-aab8-482f-8edb-67800fa4960e" containerName="istio-proxy" Oct 31 13:45:00.174758 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.174620 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e2a46f8-aab8-482f-8edb-67800fa4960e" containerName="istio-proxy" Oct 31 13:45:00.174758 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.174665 6841 cpu_manager.go:414] "containerMap: removing container" podUID="f7095807-f9e9-47ba-be79-262d1b097fc2" containerName="istio-proxy" Oct 31 13:45:00.174758 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.174674 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7095807-f9e9-47ba-be79-262d1b097fc2" containerName="istio-proxy" Oct 31 13:45:00.175172 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.175153 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="c85bee9f-4267-44ec-a03f-0b97c6b95d24" containerName="istio-proxy" Oct 31 13:45:00.175252 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.175197 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4e95366-875b-4fda-8e1b-6e441df2f5b2" containerName="istio-proxy" Oct 31 13:45:00.175252 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.175219 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7095807-f9e9-47ba-be79-262d1b097fc2" containerName="istio-proxy" Oct 31 13:45:00.175252 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.175233 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e2a46f8-aab8-482f-8edb-67800fa4960e" containerName="istio-proxy" Oct 31 13:45:00.187151 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.187127 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890905-jwdgm" Oct 31 13:45:00.208832 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.208800 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890905-96nlx"] Oct 31 13:45:00.216446 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.216420 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890905-jwdgm"] Oct 31 13:45:00.216639 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.216421 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:45:00.216639 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.216504 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890905-96nlx" Oct 31 13:45:00.271498 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.271463 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890905-96nlx"] Oct 31 13:45:00.290248 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.290208 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890905-9cfdl"] Oct 31 13:45:00.295201 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.295164 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-9cfdl" Oct 31 13:45:00.317129 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.317023 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890905-9cfdl"] Oct 31 13:45:00.336959 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.336929 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhglp\" (UniqueName: \"kubernetes.io/projected/8cc7973a-3561-4e4c-8817-9febedcd7ef7-kube-api-access-lhglp\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890905-96nlx\" (UID: \"8cc7973a-3561-4e4c-8817-9febedcd7ef7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890905-96nlx" Oct 31 13:45:00.337123 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.337092 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb89104-d09a-4dd1-9f92-cc97d2866b18-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-842ac9-29890905-jwdgm\" (UID: \"cfb89104-d09a-4dd1-9f92-cc97d2866b18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890905-jwdgm" Oct 31 13:45:00.337123 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.337121 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgq7s\" (UniqueName: \"kubernetes.io/projected/cfb89104-d09a-4dd1-9f92-cc97d2866b18-kube-api-access-rgq7s\") pod \"maas-api-key-cleanup-e2e-extmod-a-842ac9-29890905-jwdgm\" (UID: \"cfb89104-d09a-4dd1-9f92-cc97d2866b18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890905-jwdgm" Oct 31 13:45:00.337309 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.337282 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cc7973a-3561-4e4c-8817-9febedcd7ef7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890905-96nlx\" (UID: \"8cc7973a-3561-4e4c-8817-9febedcd7ef7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890905-96nlx" Oct 31 13:45:00.395093 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.395053 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:00.438088 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.437995 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frwj5\" (UniqueName: \"kubernetes.io/projected/a753b639-2816-4910-b9e1-3bddbf22acd7-kube-api-access-frwj5\") pod \"maas-api-key-cleanup-29890905-9cfdl\" (UID: \"a753b639-2816-4910-b9e1-3bddbf22acd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-9cfdl" Oct 31 13:45:00.438088 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.438069 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cc7973a-3561-4e4c-8817-9febedcd7ef7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890905-96nlx\" (UID: \"8cc7973a-3561-4e4c-8817-9febedcd7ef7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890905-96nlx" Oct 31 13:45:00.438640 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.438162 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhglp\" (UniqueName: \"kubernetes.io/projected/8cc7973a-3561-4e4c-8817-9febedcd7ef7-kube-api-access-lhglp\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890905-96nlx\" (UID: \"8cc7973a-3561-4e4c-8817-9febedcd7ef7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890905-96nlx" Oct 31 13:45:00.438640 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.438318 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a753b639-2816-4910-b9e1-3bddbf22acd7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890905-9cfdl\" (UID: \"a753b639-2816-4910-b9e1-3bddbf22acd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-9cfdl" Oct 31 13:45:00.438640 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.438379 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb89104-d09a-4dd1-9f92-cc97d2866b18-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-842ac9-29890905-jwdgm\" (UID: \"cfb89104-d09a-4dd1-9f92-cc97d2866b18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890905-jwdgm" Oct 31 13:45:00.438640 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.438445 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgq7s\" (UniqueName: \"kubernetes.io/projected/cfb89104-d09a-4dd1-9f92-cc97d2866b18-kube-api-access-rgq7s\") pod \"maas-api-key-cleanup-e2e-extmod-a-842ac9-29890905-jwdgm\" (UID: \"cfb89104-d09a-4dd1-9f92-cc97d2866b18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890905-jwdgm" Oct 31 13:45:00.438970 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.438941 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cc7973a-3561-4e4c-8817-9febedcd7ef7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890905-96nlx\" (UID: \"8cc7973a-3561-4e4c-8817-9febedcd7ef7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890905-96nlx" Oct 31 13:45:00.439112 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.439092 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb89104-d09a-4dd1-9f92-cc97d2866b18-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-842ac9-29890905-jwdgm\" (UID: \"cfb89104-d09a-4dd1-9f92-cc97d2866b18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890905-jwdgm" Oct 31 13:45:00.453508 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.453482 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgq7s\" (UniqueName: \"kubernetes.io/projected/cfb89104-d09a-4dd1-9f92-cc97d2866b18-kube-api-access-rgq7s\") pod \"maas-api-key-cleanup-e2e-extmod-a-842ac9-29890905-jwdgm\" (UID: \"cfb89104-d09a-4dd1-9f92-cc97d2866b18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890905-jwdgm" Oct 31 13:45:00.454602 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.454574 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhglp\" (UniqueName: \"kubernetes.io/projected/8cc7973a-3561-4e4c-8817-9febedcd7ef7-kube-api-access-lhglp\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890905-96nlx\" (UID: \"8cc7973a-3561-4e4c-8817-9febedcd7ef7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890905-96nlx" Oct 31 13:45:00.501640 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.501606 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890905-jwdgm" Oct 31 13:45:00.531803 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.531770 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890905-96nlx" Oct 31 13:45:00.540095 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.540058 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a753b639-2816-4910-b9e1-3bddbf22acd7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890905-9cfdl\" (UID: \"a753b639-2816-4910-b9e1-3bddbf22acd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-9cfdl" Oct 31 13:45:00.540240 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.540188 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frwj5\" (UniqueName: \"kubernetes.io/projected/a753b639-2816-4910-b9e1-3bddbf22acd7-kube-api-access-frwj5\") pod \"maas-api-key-cleanup-29890905-9cfdl\" (UID: \"a753b639-2816-4910-b9e1-3bddbf22acd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-9cfdl" Oct 31 13:45:00.540905 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.540878 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a753b639-2816-4910-b9e1-3bddbf22acd7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890905-9cfdl\" (UID: \"a753b639-2816-4910-b9e1-3bddbf22acd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-9cfdl" Oct 31 13:45:00.556604 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.556547 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frwj5\" (UniqueName: \"kubernetes.io/projected/a753b639-2816-4910-b9e1-3bddbf22acd7-kube-api-access-frwj5\") pod \"maas-api-key-cleanup-29890905-9cfdl\" (UID: \"a753b639-2816-4910-b9e1-3bddbf22acd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-9cfdl" Oct 31 13:45:00.616969 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.616925 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-9cfdl" Oct 31 13:45:00.867082 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:45:00.867048 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfb89104_d09a_4dd1_9f92_cc97d2866b18.slice/crio-5bad0a8890a5c2f9c635e6fc2b959ebd867749b0589c56de561e5e88c3028bc8 WatchSource:0}: Error finding container 5bad0a8890a5c2f9c635e6fc2b959ebd867749b0589c56de561e5e88c3028bc8: Status 404 returned error can't find the container with id 5bad0a8890a5c2f9c635e6fc2b959ebd867749b0589c56de561e5e88c3028bc8 Oct 31 13:45:00.868028 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:45:00.867907 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cc7973a_3561_4e4c_8817_9febedcd7ef7.slice/crio-83f2a4a10216242d7ea619a029efccd93d14eca020ff63b4f60170c811fdade3 WatchSource:0}: Error finding container 83f2a4a10216242d7ea619a029efccd93d14eca020ff63b4f60170c811fdade3: Status 404 returned error can't find the container with id 83f2a4a10216242d7ea619a029efccd93d14eca020ff63b4f60170c811fdade3 Oct 31 13:45:00.868399 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.868377 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890905-jwdgm"] Oct 31 13:45:00.871283 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.871260 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890905-96nlx"] Oct 31 13:45:00.935069 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:00.935032 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890905-9cfdl"] Oct 31 13:45:00.936518 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:45:00.936483 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda753b639_2816_4910_b9e1_3bddbf22acd7.slice/crio-64e00de2802db9d0da455e9818f379bb27d0ed376199a1e0ef1251e39aa6d6a7 WatchSource:0}: Error finding container 64e00de2802db9d0da455e9818f379bb27d0ed376199a1e0ef1251e39aa6d6a7: Status 404 returned error can't find the container with id 64e00de2802db9d0da455e9818f379bb27d0ed376199a1e0ef1251e39aa6d6a7 Oct 31 13:45:01.174083 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:01.174040 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:01.201487 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:01.201451 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-9cfdl" event={"ID":"a753b639-2816-4910-b9e1-3bddbf22acd7","Type":"ContainerStarted","Data":"64e00de2802db9d0da455e9818f379bb27d0ed376199a1e0ef1251e39aa6d6a7"} Oct 31 13:45:01.203048 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:01.203019 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890905-jwdgm" event={"ID":"cfb89104-d09a-4dd1-9f92-cc97d2866b18","Type":"ContainerStarted","Data":"5bad0a8890a5c2f9c635e6fc2b959ebd867749b0589c56de561e5e88c3028bc8"} Oct 31 13:45:01.204454 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:01.204430 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890905-96nlx" event={"ID":"8cc7973a-3561-4e4c-8817-9febedcd7ef7","Type":"ContainerStarted","Data":"83f2a4a10216242d7ea619a029efccd93d14eca020ff63b4f60170c811fdade3"} Oct 31 13:45:02.038192 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:02.038157 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:02.211239 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:02.211210 6841 generic.go:356] "Generic (PLEG): container finished" podID="a753b639-2816-4910-b9e1-3bddbf22acd7" containerID="68c45240ca93910f80d6e981810a9640991d8a97eb0d38a7181973a07bdf560b" exitCode=60 Oct 31 13:45:02.211440 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:02.211284 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-9cfdl" event={"ID":"a753b639-2816-4910-b9e1-3bddbf22acd7","Type":"ContainerDied","Data":"68c45240ca93910f80d6e981810a9640991d8a97eb0d38a7181973a07bdf560b"} Oct 31 13:45:02.211607 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:02.211590 6841 scope.go:122] "RemoveContainer" containerID="68c45240ca93910f80d6e981810a9640991d8a97eb0d38a7181973a07bdf560b" Oct 31 13:45:02.212884 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:02.212861 6841 generic.go:356] "Generic (PLEG): container finished" podID="cfb89104-d09a-4dd1-9f92-cc97d2866b18" containerID="b4b5c5abb683bfc8218cd20a4892255c1d61aa8d90b459b6271cf5388e3582a5" exitCode=60 Oct 31 13:45:02.213012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:02.212953 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890905-jwdgm" event={"ID":"cfb89104-d09a-4dd1-9f92-cc97d2866b18","Type":"ContainerDied","Data":"b4b5c5abb683bfc8218cd20a4892255c1d61aa8d90b459b6271cf5388e3582a5"} Oct 31 13:45:02.213178 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:02.213167 6841 scope.go:122] "RemoveContainer" containerID="b4b5c5abb683bfc8218cd20a4892255c1d61aa8d90b459b6271cf5388e3582a5" Oct 31 13:45:02.214808 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:02.214706 6841 generic.go:356] "Generic (PLEG): container finished" podID="8cc7973a-3561-4e4c-8817-9febedcd7ef7" containerID="6eb01a098cb2cc580f0a0badeafc1c1481f95137328bd3295916d77094b44cd4" exitCode=60 Oct 31 13:45:02.214871 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:02.214794 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890905-96nlx" event={"ID":"8cc7973a-3561-4e4c-8817-9febedcd7ef7","Type":"ContainerDied","Data":"6eb01a098cb2cc580f0a0badeafc1c1481f95137328bd3295916d77094b44cd4"} Oct 31 13:45:02.215011 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:02.214975 6841 scope.go:122] "RemoveContainer" containerID="6eb01a098cb2cc580f0a0badeafc1c1481f95137328bd3295916d77094b44cd4" Oct 31 13:45:02.290808 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:02.290776 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:03.168202 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:03.168170 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:03.223662 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:03.223630 6841 generic.go:356] "Generic (PLEG): container finished" podID="a753b639-2816-4910-b9e1-3bddbf22acd7" containerID="921ca2a12351ed27a49207d236b8555134f89b794b2525bfd12f8492210fba52" exitCode=60 Oct 31 13:45:03.223822 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:03.223713 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-9cfdl" event={"ID":"a753b639-2816-4910-b9e1-3bddbf22acd7","Type":"ContainerDied","Data":"921ca2a12351ed27a49207d236b8555134f89b794b2525bfd12f8492210fba52"} Oct 31 13:45:03.223822 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:03.223778 6841 scope.go:122] "RemoveContainer" containerID="68c45240ca93910f80d6e981810a9640991d8a97eb0d38a7181973a07bdf560b" Oct 31 13:45:03.224154 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:03.224127 6841 scope.go:122] "RemoveContainer" containerID="921ca2a12351ed27a49207d236b8555134f89b794b2525bfd12f8492210fba52" Oct 31 13:45:03.224503 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:45:03.224467 6841 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890905-9cfdl_odh-ai-gateway-infra(a753b639-2816-4910-b9e1-3bddbf22acd7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-9cfdl" podUID="a753b639-2816-4910-b9e1-3bddbf22acd7" Oct 31 13:45:03.225872 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:03.225850 6841 generic.go:356] "Generic (PLEG): container finished" podID="cfb89104-d09a-4dd1-9f92-cc97d2866b18" containerID="e848616b12009e4f82928281979daf4453282c081d5b9d33272dc4bafd54f454" exitCode=60 Oct 31 13:45:03.225969 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:03.225928 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890905-jwdgm" event={"ID":"cfb89104-d09a-4dd1-9f92-cc97d2866b18","Type":"ContainerDied","Data":"e848616b12009e4f82928281979daf4453282c081d5b9d33272dc4bafd54f454"} Oct 31 13:45:03.226203 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:03.226187 6841 scope.go:122] "RemoveContainer" containerID="e848616b12009e4f82928281979daf4453282c081d5b9d33272dc4bafd54f454" Oct 31 13:45:03.226458 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:45:03.226436 6841 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-842ac9-29890905-jwdgm_odh-ai-gateway-infra(cfb89104-d09a-4dd1-9f92-cc97d2866b18)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890905-jwdgm" podUID="cfb89104-d09a-4dd1-9f92-cc97d2866b18" Oct 31 13:45:03.228109 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:03.228070 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890905-96nlx" event={"ID":"8cc7973a-3561-4e4c-8817-9febedcd7ef7","Type":"ContainerStarted","Data":"ec2a7c2f2a169d429508078041bd893b658230ee9b6dbf842b933b34302c5208"} Oct 31 13:45:03.236071 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:03.236044 6841 scope.go:122] "RemoveContainer" containerID="b4b5c5abb683bfc8218cd20a4892255c1d61aa8d90b459b6271cf5388e3582a5" Oct 31 13:45:03.280607 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:03.280556 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890905-96nlx" podStartSLOduration=3.280542145 podStartE2EDuration="3.280542145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:03.279258404 +0000 UTC m=+4349.762381307" watchObservedRunningTime="2026-10-31 13:45:03.280542145 +0000 UTC m=+4349.763665054" Oct 31 13:45:04.236475 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:04.236447 6841 scope.go:122] "RemoveContainer" containerID="921ca2a12351ed27a49207d236b8555134f89b794b2525bfd12f8492210fba52" Oct 31 13:45:04.236936 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:45:04.236794 6841 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890905-9cfdl_odh-ai-gateway-infra(a753b639-2816-4910-b9e1-3bddbf22acd7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-9cfdl" podUID="a753b639-2816-4910-b9e1-3bddbf22acd7" Oct 31 13:45:04.238122 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:04.238098 6841 scope.go:122] "RemoveContainer" containerID="e848616b12009e4f82928281979daf4453282c081d5b9d33272dc4bafd54f454" Oct 31 13:45:04.238416 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:45:04.238367 6841 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-842ac9-29890905-jwdgm_odh-ai-gateway-infra(cfb89104-d09a-4dd1-9f92-cc97d2866b18)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890905-jwdgm" podUID="cfb89104-d09a-4dd1-9f92-cc97d2866b18" Oct 31 13:45:04.239613 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:04.239595 6841 generic.go:356] "Generic (PLEG): container finished" podID="8cc7973a-3561-4e4c-8817-9febedcd7ef7" containerID="ec2a7c2f2a169d429508078041bd893b658230ee9b6dbf842b933b34302c5208" exitCode=60 Oct 31 13:45:04.239753 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:04.239646 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890905-96nlx" event={"ID":"8cc7973a-3561-4e4c-8817-9febedcd7ef7","Type":"ContainerDied","Data":"ec2a7c2f2a169d429508078041bd893b658230ee9b6dbf842b933b34302c5208"} Oct 31 13:45:04.239753 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:04.239697 6841 scope.go:122] "RemoveContainer" containerID="6eb01a098cb2cc580f0a0badeafc1c1481f95137328bd3295916d77094b44cd4" Oct 31 13:45:04.239927 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:04.239910 6841 scope.go:122] "RemoveContainer" containerID="ec2a7c2f2a169d429508078041bd893b658230ee9b6dbf842b933b34302c5208" Oct 31 13:45:04.240248 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:45:04.240222 6841 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890905-96nlx_odh-ai-gateway-infra(8cc7973a-3561-4e4c-8817-9febedcd7ef7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890905-96nlx" podUID="8cc7973a-3561-4e4c-8817-9febedcd7ef7" Oct 31 13:45:04.263425 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:04.263396 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:05.247524 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:05.247497 6841 scope.go:122] "RemoveContainer" containerID="ec2a7c2f2a169d429508078041bd893b658230ee9b6dbf842b933b34302c5208" Oct 31 13:45:05.247896 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:45:05.247746 6841 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890905-96nlx_odh-ai-gateway-infra(8cc7973a-3561-4e4c-8817-9febedcd7ef7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890905-96nlx" podUID="8cc7973a-3561-4e4c-8817-9febedcd7ef7" Oct 31 13:45:05.275847 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:05.275804 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:06.220211 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:06.220179 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:06.501443 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:06.501412 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" Oct 31 13:45:06.619165 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:06.619132 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-istio-data\") pod \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\" (UID: \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\") " Oct 31 13:45:06.619364 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:06.619175 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-istio-token\" (UniqueName: \"kubernetes.io/projected/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-istio-token\") pod \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\" (UID: \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\") " Oct 31 13:45:06.619364 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:06.619197 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-istio-envoy\") pod \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\" (UID: \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\") " Oct 31 13:45:06.619364 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:06.619225 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-credential-socket\") pod \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\" (UID: \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\") " Oct 31 13:45:06.619364 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:06.619295 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-kube-api-access-p685k\" (UniqueName: \"kubernetes.io/projected/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-kube-api-access-p685k\") pod \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\" (UID: \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\") " Oct 31 13:45:06.619613 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:06.619392 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-workload-certs\") pod \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\" (UID: \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\") " Oct 31 13:45:06.619613 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:06.619429 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-istiod-ca-cert\") pod \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\" (UID: \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\") " Oct 31 13:45:06.619613 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:06.619450 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-istio-data" pod "c1e1f66f-b5a1-4958-a5bf-097e3ee91884" (UID: "c1e1f66f-b5a1-4958-a5bf-097e3ee91884"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:06.619613 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:06.619457 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-workload-socket\") pod \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\" (UID: \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\") " Oct 31 13:45:06.619846 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:06.619634 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-istio-podinfo\") pod \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\" (UID: \"c1e1f66f-b5a1-4958-a5bf-097e3ee91884\") " Oct 31 13:45:06.619846 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:06.619634 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-credential-socket" pod "c1e1f66f-b5a1-4958-a5bf-097e3ee91884" (UID: "c1e1f66f-b5a1-4958-a5bf-097e3ee91884"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:06.619846 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:06.619644 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-workload-socket" pod "c1e1f66f-b5a1-4958-a5bf-097e3ee91884" (UID: "c1e1f66f-b5a1-4958-a5bf-097e3ee91884"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:06.620048 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:06.619954 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-istiod-ca-cert" pod "c1e1f66f-b5a1-4958-a5bf-097e3ee91884" (UID: "c1e1f66f-b5a1-4958-a5bf-097e3ee91884"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:06.620048 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:06.619954 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-workload-certs" pod "c1e1f66f-b5a1-4958-a5bf-097e3ee91884" (UID: "c1e1f66f-b5a1-4958-a5bf-097e3ee91884"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:06.620176 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:06.620122 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:06.620176 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:06.620138 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:06.620176 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:06.620148 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:06.620176 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:06.620157 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:06.620176 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:06.620165 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:06.621428 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:06.621404 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-istio-envoy" pod "c1e1f66f-b5a1-4958-a5bf-097e3ee91884" (UID: "c1e1f66f-b5a1-4958-a5bf-097e3ee91884"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:06.621566 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:06.621507 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-istio-token" pod "c1e1f66f-b5a1-4958-a5bf-097e3ee91884" (UID: "c1e1f66f-b5a1-4958-a5bf-097e3ee91884"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:06.622017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:06.621975 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-istio-podinfo" pod "c1e1f66f-b5a1-4958-a5bf-097e3ee91884" (UID: "c1e1f66f-b5a1-4958-a5bf-097e3ee91884"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:45:06.622115 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:06.622010 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-kube-api-access-p685k" pod "c1e1f66f-b5a1-4958-a5bf-097e3ee91884" (UID: "c1e1f66f-b5a1-4958-a5bf-097e3ee91884"). InnerVolumeSpecName "kube-api-access-p685k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:06.720724 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:06.720690 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:06.720724 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:06.720720 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:06.720724 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:06.720730 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:06.721116 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:06.720739 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p685k\" (UniqueName: \"kubernetes.io/projected/c1e1f66f-b5a1-4958-a5bf-097e3ee91884-kube-api-access-p685k\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:07.166917 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:07.166874 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:07.260420 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:07.260382 6841 generic.go:356] "Generic (PLEG): container finished" podID="c1e1f66f-b5a1-4958-a5bf-097e3ee91884" containerID="aa71596758bfa988dfa77f166a988af2349c0bf59a1a589c1944dbf817dbba41" exitCode=0 Oct 31 13:45:07.260584 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:07.260434 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" Oct 31 13:45:07.260584 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:07.260468 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" event={"ID":"c1e1f66f-b5a1-4958-a5bf-097e3ee91884","Type":"ContainerDied","Data":"aa71596758bfa988dfa77f166a988af2349c0bf59a1a589c1944dbf817dbba41"} Oct 31 13:45:07.260584 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:07.260523 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6" event={"ID":"c1e1f66f-b5a1-4958-a5bf-097e3ee91884","Type":"ContainerDied","Data":"f9d3edd8b9de5532cd4a272c53bd5d4183db57383430ef0aa8ab9eab4b287f9c"} Oct 31 13:45:07.260584 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:07.260548 6841 scope.go:122] "RemoveContainer" containerID="aa71596758bfa988dfa77f166a988af2349c0bf59a1a589c1944dbf817dbba41" Oct 31 13:45:07.275675 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:07.275651 6841 scope.go:122] "RemoveContainer" containerID="aa71596758bfa988dfa77f166a988af2349c0bf59a1a589c1944dbf817dbba41" Oct 31 13:45:07.275963 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:45:07.275945 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa71596758bfa988dfa77f166a988af2349c0bf59a1a589c1944dbf817dbba41\": container with ID starting with aa71596758bfa988dfa77f166a988af2349c0bf59a1a589c1944dbf817dbba41 not found: ID does not exist" containerID="aa71596758bfa988dfa77f166a988af2349c0bf59a1a589c1944dbf817dbba41" Oct 31 13:45:07.276062 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:07.275972 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa71596758bfa988dfa77f166a988af2349c0bf59a1a589c1944dbf817dbba41"} err="failed to get container status \"aa71596758bfa988dfa77f166a988af2349c0bf59a1a589c1944dbf817dbba41\": rpc error: code = NotFound desc = could not find container \"aa71596758bfa988dfa77f166a988af2349c0bf59a1a589c1944dbf817dbba41\": container with ID starting with aa71596758bfa988dfa77f166a988af2349c0bf59a1a589c1944dbf817dbba41 not found: ID does not exist" Oct 31 13:45:07.313079 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:07.312969 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6"] Oct 31 13:45:07.316539 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:07.316511 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-vtdk6"] Oct 31 13:45:07.439772 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:07.439686 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:07.646096 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:07.646055 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1e1f66f-b5a1-4958-a5bf-097e3ee91884" path="/var/lib/kubelet/pods/c1e1f66f-b5a1-4958-a5bf-097e3ee91884/volumes" Oct 31 13:45:08.525161 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:08.525127 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:08.874958 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:08.874868 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:09.130936 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:09.130857 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:09.619311 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:09.619274 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:10.830212 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:10.830176 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:11.043702 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:11.043669 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:11.648131 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:11.648090 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:12.421544 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:12.421511 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:13.245048 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:13.245011 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:13.638865 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:13.638796 6841 scope.go:122] "RemoveContainer" containerID="921ca2a12351ed27a49207d236b8555134f89b794b2525bfd12f8492210fba52" Oct 31 13:45:14.213663 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:14.213579 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:14.307062 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:14.307026 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-9cfdl" event={"ID":"a753b639-2816-4910-b9e1-3bddbf22acd7","Type":"ContainerStarted","Data":"507a383087af1186c3a16e723f8968f269bcdc17a3585242584969f13e1bb7d4"} Oct 31 13:45:14.330185 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:14.330124 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-9cfdl" podStartSLOduration=14.330106669 podStartE2EDuration="14.330106669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:14.329966173 +0000 UTC m=+4360.813089075" watchObservedRunningTime="2026-10-31 13:45:14.330106669 +0000 UTC m=+4360.813229572" Oct 31 13:45:14.635564 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:14.635485 6841 scope.go:122] "RemoveContainer" containerID="e848616b12009e4f82928281979daf4453282c081d5b9d33272dc4bafd54f454" Oct 31 13:45:15.007631 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:15.007574 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:15.315767 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:15.315688 6841 generic.go:356] "Generic (PLEG): container finished" podID="a753b639-2816-4910-b9e1-3bddbf22acd7" containerID="507a383087af1186c3a16e723f8968f269bcdc17a3585242584969f13e1bb7d4" exitCode=60 Oct 31 13:45:15.315952 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:15.315761 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-9cfdl" event={"ID":"a753b639-2816-4910-b9e1-3bddbf22acd7","Type":"ContainerDied","Data":"507a383087af1186c3a16e723f8968f269bcdc17a3585242584969f13e1bb7d4"} Oct 31 13:45:15.315952 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:15.315826 6841 scope.go:122] "RemoveContainer" containerID="921ca2a12351ed27a49207d236b8555134f89b794b2525bfd12f8492210fba52" Oct 31 13:45:15.316277 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:15.316258 6841 scope.go:122] "RemoveContainer" containerID="507a383087af1186c3a16e723f8968f269bcdc17a3585242584969f13e1bb7d4" Oct 31 13:45:15.316538 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:45:15.316519 6841 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890905-9cfdl_odh-ai-gateway-infra(a753b639-2816-4910-b9e1-3bddbf22acd7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-9cfdl" podUID="a753b639-2816-4910-b9e1-3bddbf22acd7" Oct 31 13:45:15.317811 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:15.317790 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890905-jwdgm" event={"ID":"cfb89104-d09a-4dd1-9f92-cc97d2866b18","Type":"ContainerStarted","Data":"ba7e88ac2d35ae9da6b92892de7c7ccc1ada9913a0dea7e0e35cf3dc2ec9a79f"} Oct 31 13:45:15.363323 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:15.363294 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890905-9cfdl"] Oct 31 13:45:15.373563 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:15.373508 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890905-jwdgm" podStartSLOduration=15.373491413 podStartE2EDuration="15.373491413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:15.372874304 +0000 UTC m=+4361.855997205" watchObservedRunningTime="2026-10-31 13:45:15.373491413 +0000 UTC m=+4361.856614315" Oct 31 13:45:16.073174 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:16.073135 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:16.327863 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:16.327835 6841 generic.go:356] "Generic (PLEG): container finished" podID="cfb89104-d09a-4dd1-9f92-cc97d2866b18" containerID="ba7e88ac2d35ae9da6b92892de7c7ccc1ada9913a0dea7e0e35cf3dc2ec9a79f" exitCode=60 Oct 31 13:45:16.328022 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:16.327915 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890905-jwdgm" event={"ID":"cfb89104-d09a-4dd1-9f92-cc97d2866b18","Type":"ContainerDied","Data":"ba7e88ac2d35ae9da6b92892de7c7ccc1ada9913a0dea7e0e35cf3dc2ec9a79f"} Oct 31 13:45:16.328075 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:16.328024 6841 scope.go:122] "RemoveContainer" containerID="e848616b12009e4f82928281979daf4453282c081d5b9d33272dc4bafd54f454" Oct 31 13:45:16.328349 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:16.328335 6841 scope.go:122] "RemoveContainer" containerID="ba7e88ac2d35ae9da6b92892de7c7ccc1ada9913a0dea7e0e35cf3dc2ec9a79f" Oct 31 13:45:16.328567 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:45:16.328546 6841 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-842ac9-29890905-jwdgm_odh-ai-gateway-infra(cfb89104-d09a-4dd1-9f92-cc97d2866b18)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890905-jwdgm" podUID="cfb89104-d09a-4dd1-9f92-cc97d2866b18" Oct 31 13:45:16.384177 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:16.384143 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890905-jwdgm"] Oct 31 13:45:16.476855 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:16.476828 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-9cfdl" Oct 31 13:45:16.529278 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:16.529242 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a753b639-2816-4910-b9e1-3bddbf22acd7-kube-api-access-frwj5\" (UniqueName: \"kubernetes.io/projected/a753b639-2816-4910-b9e1-3bddbf22acd7-kube-api-access-frwj5\") pod \"a753b639-2816-4910-b9e1-3bddbf22acd7\" (UID: \"a753b639-2816-4910-b9e1-3bddbf22acd7\") " Oct 31 13:45:16.529436 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:16.529312 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a753b639-2816-4910-b9e1-3bddbf22acd7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a753b639-2816-4910-b9e1-3bddbf22acd7-maas-api-service-ca\") pod \"a753b639-2816-4910-b9e1-3bddbf22acd7\" (UID: \"a753b639-2816-4910-b9e1-3bddbf22acd7\") " Oct 31 13:45:16.529699 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:16.529670 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a753b639-2816-4910-b9e1-3bddbf22acd7-maas-api-service-ca" pod "a753b639-2816-4910-b9e1-3bddbf22acd7" (UID: "a753b639-2816-4910-b9e1-3bddbf22acd7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:16.529826 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:16.529805 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a753b639-2816-4910-b9e1-3bddbf22acd7-maas-api-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:16.531442 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:16.531418 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a753b639-2816-4910-b9e1-3bddbf22acd7-kube-api-access-frwj5" pod "a753b639-2816-4910-b9e1-3bddbf22acd7" (UID: "a753b639-2816-4910-b9e1-3bddbf22acd7"). InnerVolumeSpecName "kube-api-access-frwj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:16.630710 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:16.630638 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frwj5\" (UniqueName: \"kubernetes.io/projected/a753b639-2816-4910-b9e1-3bddbf22acd7-kube-api-access-frwj5\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:16.634832 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:16.634814 6841 scope.go:122] "RemoveContainer" containerID="ec2a7c2f2a169d429508078041bd893b658230ee9b6dbf842b933b34302c5208" Oct 31 13:45:16.848654 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:16.848617 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:17.268336 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:17.268244 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:17.335097 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:17.335066 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-9cfdl" Oct 31 13:45:17.335296 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:17.335060 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-9cfdl" event={"ID":"a753b639-2816-4910-b9e1-3bddbf22acd7","Type":"ContainerDied","Data":"64e00de2802db9d0da455e9818f379bb27d0ed376199a1e0ef1251e39aa6d6a7"} Oct 31 13:45:17.335296 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:17.335172 6841 scope.go:122] "RemoveContainer" containerID="507a383087af1186c3a16e723f8968f269bcdc17a3585242584969f13e1bb7d4" Oct 31 13:45:17.339679 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:17.339650 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890905-96nlx" event={"ID":"8cc7973a-3561-4e4c-8817-9febedcd7ef7","Type":"ContainerStarted","Data":"25d9b521f56af14ff430cba619b2463efa48d735364caeca97344103b5bf5873"} Oct 31 13:45:17.421057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:17.421026 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890905-9cfdl"] Oct 31 13:45:17.427363 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:17.427332 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890905-9cfdl"] Oct 31 13:45:17.542398 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:17.542370 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890905-jwdgm" Oct 31 13:45:17.640569 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:17.640536 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfb89104-d09a-4dd1-9f92-cc97d2866b18-kube-api-access-rgq7s\" (UniqueName: \"kubernetes.io/projected/cfb89104-d09a-4dd1-9f92-cc97d2866b18-kube-api-access-rgq7s\") pod \"cfb89104-d09a-4dd1-9f92-cc97d2866b18\" (UID: \"cfb89104-d09a-4dd1-9f92-cc97d2866b18\") " Oct 31 13:45:17.640782 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:17.640622 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfb89104-d09a-4dd1-9f92-cc97d2866b18-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb89104-d09a-4dd1-9f92-cc97d2866b18-maas-api-service-ca\") pod \"cfb89104-d09a-4dd1-9f92-cc97d2866b18\" (UID: \"cfb89104-d09a-4dd1-9f92-cc97d2866b18\") " Oct 31 13:45:17.641096 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:17.641062 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfb89104-d09a-4dd1-9f92-cc97d2866b18-maas-api-service-ca" pod "cfb89104-d09a-4dd1-9f92-cc97d2866b18" (UID: "cfb89104-d09a-4dd1-9f92-cc97d2866b18"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:17.642938 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:17.642908 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a753b639-2816-4910-b9e1-3bddbf22acd7" path="/var/lib/kubelet/pods/a753b639-2816-4910-b9e1-3bddbf22acd7/volumes" Oct 31 13:45:17.643119 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:17.642930 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb89104-d09a-4dd1-9f92-cc97d2866b18-kube-api-access-rgq7s" pod "cfb89104-d09a-4dd1-9f92-cc97d2866b18" (UID: "cfb89104-d09a-4dd1-9f92-cc97d2866b18"). InnerVolumeSpecName "kube-api-access-rgq7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:17.742464 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:17.742425 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb89104-d09a-4dd1-9f92-cc97d2866b18-maas-api-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:17.742464 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:17.742460 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgq7s\" (UniqueName: \"kubernetes.io/projected/cfb89104-d09a-4dd1-9f92-cc97d2866b18-kube-api-access-rgq7s\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:18.051416 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.051378 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:18.347238 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.347159 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890905-jwdgm" Oct 31 13:45:18.347749 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.347158 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890905-jwdgm" event={"ID":"cfb89104-d09a-4dd1-9f92-cc97d2866b18","Type":"ContainerDied","Data":"5bad0a8890a5c2f9c635e6fc2b959ebd867749b0589c56de561e5e88c3028bc8"} Oct 31 13:45:18.347749 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.347313 6841 scope.go:122] "RemoveContainer" containerID="ba7e88ac2d35ae9da6b92892de7c7ccc1ada9913a0dea7e0e35cf3dc2ec9a79f" Oct 31 13:45:18.349365 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.349319 6841 generic.go:356] "Generic (PLEG): container finished" podID="8cc7973a-3561-4e4c-8817-9febedcd7ef7" containerID="25d9b521f56af14ff430cba619b2463efa48d735364caeca97344103b5bf5873" exitCode=60 Oct 31 13:45:18.349707 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.349385 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890905-96nlx" event={"ID":"8cc7973a-3561-4e4c-8817-9febedcd7ef7","Type":"ContainerDied","Data":"25d9b521f56af14ff430cba619b2463efa48d735364caeca97344103b5bf5873"} Oct 31 13:45:18.349801 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.349744 6841 scope.go:122] "RemoveContainer" containerID="25d9b521f56af14ff430cba619b2463efa48d735364caeca97344103b5bf5873" Oct 31 13:45:18.349966 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:45:18.349947 6841 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890905-96nlx_odh-ai-gateway-infra(8cc7973a-3561-4e4c-8817-9febedcd7ef7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890905-96nlx" podUID="8cc7973a-3561-4e4c-8817-9febedcd7ef7" Oct 31 13:45:18.356686 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.356668 6841 scope.go:122] "RemoveContainer" containerID="ec2a7c2f2a169d429508078041bd893b658230ee9b6dbf842b933b34302c5208" Oct 31 13:45:18.371203 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.371178 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890905-jwdgm"] Oct 31 13:45:18.373033 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.373013 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890905-jwdgm"] Oct 31 13:45:18.438396 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.438366 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890905-96nlx"] Oct 31 13:45:18.562827 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.562787 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-214e2225-fl86v"] Oct 31 13:45:18.563361 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.563340 6841 cpu_manager.go:414] "containerMap: removing container" podUID="a753b639-2816-4910-b9e1-3bddbf22acd7" containerName="cleanup" Oct 31 13:45:18.563480 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.563362 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a753b639-2816-4910-b9e1-3bddbf22acd7" containerName="cleanup" Oct 31 13:45:18.563480 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.563390 6841 cpu_manager.go:414] "containerMap: removing container" podUID="cfb89104-d09a-4dd1-9f92-cc97d2866b18" containerName="cleanup" Oct 31 13:45:18.563480 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.563399 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfb89104-d09a-4dd1-9f92-cc97d2866b18" containerName="cleanup" Oct 31 13:45:18.563480 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.563422 6841 cpu_manager.go:414] "containerMap: removing container" podUID="cfb89104-d09a-4dd1-9f92-cc97d2866b18" containerName="cleanup" Oct 31 13:45:18.563480 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.563429 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfb89104-d09a-4dd1-9f92-cc97d2866b18" containerName="cleanup" Oct 31 13:45:18.563480 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.563442 6841 cpu_manager.go:414] "containerMap: removing container" podUID="a753b639-2816-4910-b9e1-3bddbf22acd7" containerName="cleanup" Oct 31 13:45:18.563480 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.563449 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a753b639-2816-4910-b9e1-3bddbf22acd7" containerName="cleanup" Oct 31 13:45:18.563480 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.563469 6841 cpu_manager.go:414] "containerMap: removing container" podUID="c1e1f66f-b5a1-4958-a5bf-097e3ee91884" containerName="istio-proxy" Oct 31 13:45:18.563480 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.563476 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1e1f66f-b5a1-4958-a5bf-097e3ee91884" containerName="istio-proxy" Oct 31 13:45:18.563939 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.563599 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="a753b639-2816-4910-b9e1-3bddbf22acd7" containerName="cleanup" Oct 31 13:45:18.563939 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.563615 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfb89104-d09a-4dd1-9f92-cc97d2866b18" containerName="cleanup" Oct 31 13:45:18.563939 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.563627 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="a753b639-2816-4910-b9e1-3bddbf22acd7" containerName="cleanup" Oct 31 13:45:18.563939 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.563638 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1e1f66f-b5a1-4958-a5bf-097e3ee91884" containerName="istio-proxy" Oct 31 13:45:18.563939 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.563647 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfb89104-d09a-4dd1-9f92-cc97d2866b18" containerName="cleanup" Oct 31 13:45:18.567375 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.567352 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-214e2225-fl86v" Oct 31 13:45:18.594114 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.594091 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:45:18.616635 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.616576 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-214e2225-fl86v"] Oct 31 13:45:18.650117 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.650089 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jwnt\" (UniqueName: \"kubernetes.io/projected/51feb884-049e-45fb-a054-b2f8022622a9-kube-api-access-4jwnt\") pod \"maas-api-revoke-sub-214e2225-fl86v\" (UID: \"51feb884-049e-45fb-a054-b2f8022622a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-214e2225-fl86v" Oct 31 13:45:18.650304 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.650146 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51feb884-049e-45fb-a054-b2f8022622a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-214e2225-fl86v\" (UID: \"51feb884-049e-45fb-a054-b2f8022622a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-214e2225-fl86v" Oct 31 13:45:18.650304 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.650212 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51feb884-049e-45fb-a054-b2f8022622a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-214e2225-fl86v\" (UID: \"51feb884-049e-45fb-a054-b2f8022622a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-214e2225-fl86v" Oct 31 13:45:18.751103 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.751069 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51feb884-049e-45fb-a054-b2f8022622a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-214e2225-fl86v\" (UID: \"51feb884-049e-45fb-a054-b2f8022622a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-214e2225-fl86v" Oct 31 13:45:18.751287 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.751125 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51feb884-049e-45fb-a054-b2f8022622a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-214e2225-fl86v\" (UID: \"51feb884-049e-45fb-a054-b2f8022622a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-214e2225-fl86v" Oct 31 13:45:18.751287 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.751229 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jwnt\" (UniqueName: \"kubernetes.io/projected/51feb884-049e-45fb-a054-b2f8022622a9-kube-api-access-4jwnt\") pod \"maas-api-revoke-sub-214e2225-fl86v\" (UID: \"51feb884-049e-45fb-a054-b2f8022622a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-214e2225-fl86v" Oct 31 13:45:18.751834 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.751806 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51feb884-049e-45fb-a054-b2f8022622a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-214e2225-fl86v\" (UID: \"51feb884-049e-45fb-a054-b2f8022622a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-214e2225-fl86v" Oct 31 13:45:18.753582 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.753555 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51feb884-049e-45fb-a054-b2f8022622a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-214e2225-fl86v\" (UID: \"51feb884-049e-45fb-a054-b2f8022622a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-214e2225-fl86v" Oct 31 13:45:18.769876 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.769848 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jwnt\" (UniqueName: \"kubernetes.io/projected/51feb884-049e-45fb-a054-b2f8022622a9-kube-api-access-4jwnt\") pod \"maas-api-revoke-sub-214e2225-fl86v\" (UID: \"51feb884-049e-45fb-a054-b2f8022622a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-214e2225-fl86v" Oct 31 13:45:18.881290 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:18.881215 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-214e2225-fl86v" Oct 31 13:45:19.127173 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:19.127126 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:19.170247 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:19.170219 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-214e2225-fl86v"] Oct 31 13:45:19.172484 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:45:19.172458 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51feb884_049e_45fb_a054_b2f8022622a9.slice/crio-16abff4515230681895f115bb5bb329cc74bf29c8131f45a97ab22d2f550d594 WatchSource:0}: Error finding container 16abff4515230681895f115bb5bb329cc74bf29c8131f45a97ab22d2f550d594: Status 404 returned error can't find the container with id 16abff4515230681895f115bb5bb329cc74bf29c8131f45a97ab22d2f550d594 Oct 31 13:45:19.356666 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:19.356633 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-214e2225-fl86v" event={"ID":"51feb884-049e-45fb-a054-b2f8022622a9","Type":"ContainerStarted","Data":"16abff4515230681895f115bb5bb329cc74bf29c8131f45a97ab22d2f550d594"} Oct 31 13:45:19.492837 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:19.492809 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890905-96nlx" Oct 31 13:45:19.558841 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:19.558814 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cc7973a-3561-4e4c-8817-9febedcd7ef7-kube-api-access-lhglp\" (UniqueName: \"kubernetes.io/projected/8cc7973a-3561-4e4c-8817-9febedcd7ef7-kube-api-access-lhglp\") pod \"8cc7973a-3561-4e4c-8817-9febedcd7ef7\" (UID: \"8cc7973a-3561-4e4c-8817-9febedcd7ef7\") " Oct 31 13:45:19.559040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:19.559019 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cc7973a-3561-4e4c-8817-9febedcd7ef7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cc7973a-3561-4e4c-8817-9febedcd7ef7-maas-api-service-ca\") pod \"8cc7973a-3561-4e4c-8817-9febedcd7ef7\" (UID: \"8cc7973a-3561-4e4c-8817-9febedcd7ef7\") " Oct 31 13:45:19.559329 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:19.559303 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cc7973a-3561-4e4c-8817-9febedcd7ef7-maas-api-service-ca" pod "8cc7973a-3561-4e4c-8817-9febedcd7ef7" (UID: "8cc7973a-3561-4e4c-8817-9febedcd7ef7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:19.559474 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:19.559453 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cc7973a-3561-4e4c-8817-9febedcd7ef7-maas-api-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:19.560760 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:19.560741 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cc7973a-3561-4e4c-8817-9febedcd7ef7-kube-api-access-lhglp" pod "8cc7973a-3561-4e4c-8817-9febedcd7ef7" (UID: "8cc7973a-3561-4e4c-8817-9febedcd7ef7"). InnerVolumeSpecName "kube-api-access-lhglp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:19.641612 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:19.641583 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfb89104-d09a-4dd1-9f92-cc97d2866b18" path="/var/lib/kubelet/pods/cfb89104-d09a-4dd1-9f92-cc97d2866b18/volumes" Oct 31 13:45:19.660424 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:19.660399 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhglp\" (UniqueName: \"kubernetes.io/projected/8cc7973a-3561-4e4c-8817-9febedcd7ef7-kube-api-access-lhglp\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:19.939933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:19.939867 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:20.034285 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:45:20.034200 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:45:20.224107 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:20.224074 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:20.365945 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:20.365859 6841 generic.go:356] "Generic (PLEG): container finished" podID="51feb884-049e-45fb-a054-b2f8022622a9" containerID="9e04b0f2e2ec19f00cba7c60836957ecef0bbc92a5639cc2099d2091ba4d16b7" exitCode=0 Oct 31 13:45:20.366418 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:20.365942 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-214e2225-fl86v" event={"ID":"51feb884-049e-45fb-a054-b2f8022622a9","Type":"ContainerDied","Data":"9e04b0f2e2ec19f00cba7c60836957ecef0bbc92a5639cc2099d2091ba4d16b7"} Oct 31 13:45:20.367321 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:20.367299 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890905-96nlx" event={"ID":"8cc7973a-3561-4e4c-8817-9febedcd7ef7","Type":"ContainerDied","Data":"83f2a4a10216242d7ea619a029efccd93d14eca020ff63b4f60170c811fdade3"} Oct 31 13:45:20.367447 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:20.367345 6841 scope.go:122] "RemoveContainer" containerID="25d9b521f56af14ff430cba619b2463efa48d735364caeca97344103b5bf5873" Oct 31 13:45:20.367447 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:20.367309 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890905-96nlx" Oct 31 13:45:20.399785 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:20.399758 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890905-96nlx"] Oct 31 13:45:20.404430 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:20.404408 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890905-96nlx"] Oct 31 13:45:20.716368 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:20.716330 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:21.350263 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:21.350230 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:21.550906 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:21.550875 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-214e2225-fl86v" Oct 31 13:45:21.661301 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:21.661207 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cc7973a-3561-4e4c-8817-9febedcd7ef7" path="/var/lib/kubelet/pods/8cc7973a-3561-4e4c-8817-9febedcd7ef7/volumes" Oct 31 13:45:21.684353 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:21.684321 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51feb884-049e-45fb-a054-b2f8022622a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51feb884-049e-45fb-a054-b2f8022622a9-maas-api-openshift-ca\") pod \"51feb884-049e-45fb-a054-b2f8022622a9\" (UID: \"51feb884-049e-45fb-a054-b2f8022622a9\") " Oct 31 13:45:21.684510 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:21.684428 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51feb884-049e-45fb-a054-b2f8022622a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51feb884-049e-45fb-a054-b2f8022622a9-maas-api-xks-ca\") pod \"51feb884-049e-45fb-a054-b2f8022622a9\" (UID: \"51feb884-049e-45fb-a054-b2f8022622a9\") " Oct 31 13:45:21.684510 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:21.684470 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51feb884-049e-45fb-a054-b2f8022622a9-kube-api-access-4jwnt\" (UniqueName: \"kubernetes.io/projected/51feb884-049e-45fb-a054-b2f8022622a9-kube-api-access-4jwnt\") pod \"51feb884-049e-45fb-a054-b2f8022622a9\" (UID: \"51feb884-049e-45fb-a054-b2f8022622a9\") " Oct 31 13:45:21.684731 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:21.684709 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51feb884-049e-45fb-a054-b2f8022622a9-maas-api-openshift-ca" pod "51feb884-049e-45fb-a054-b2f8022622a9" (UID: "51feb884-049e-45fb-a054-b2f8022622a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:21.684880 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:21.684856 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51feb884-049e-45fb-a054-b2f8022622a9-maas-api-openshift-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:21.686523 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:21.686501 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51feb884-049e-45fb-a054-b2f8022622a9-maas-api-xks-ca" pod "51feb884-049e-45fb-a054-b2f8022622a9" (UID: "51feb884-049e-45fb-a054-b2f8022622a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:45:21.686667 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:21.686641 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51feb884-049e-45fb-a054-b2f8022622a9-kube-api-access-4jwnt" pod "51feb884-049e-45fb-a054-b2f8022622a9" (UID: "51feb884-049e-45fb-a054-b2f8022622a9"). InnerVolumeSpecName "kube-api-access-4jwnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:21.705681 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:21.705650 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:21.786046 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:21.786009 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51feb884-049e-45fb-a054-b2f8022622a9-maas-api-xks-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:21.786046 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:21.786038 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jwnt\" (UniqueName: \"kubernetes.io/projected/51feb884-049e-45fb-a054-b2f8022622a9-kube-api-access-4jwnt\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:22.380702 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:22.380671 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-214e2225-fl86v" event={"ID":"51feb884-049e-45fb-a054-b2f8022622a9","Type":"ContainerDied","Data":"16abff4515230681895f115bb5bb329cc74bf29c8131f45a97ab22d2f550d594"} Oct 31 13:45:22.380886 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:22.380718 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16abff4515230681895f115bb5bb329cc74bf29c8131f45a97ab22d2f550d594" Oct 31 13:45:22.380886 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:22.380683 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-214e2225-fl86v" Oct 31 13:45:22.421260 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:22.421210 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:23.196537 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:23.196499 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:24.122613 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:24.122571 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:24.952529 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:24.952490 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:25.817298 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:25.817265 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:26.606127 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:26.606094 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:26.948469 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:26.948428 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:27.175355 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:27.175320 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:27.483731 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:27.483696 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:28.227563 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:28.227523 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:28.922431 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:28.922369 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:29.105730 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:29.105697 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn"] Oct 31 13:45:29.264720 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:29.264682 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:29.720284 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:29.720249 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:30.617150 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:30.617118 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:31.243798 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:31.243759 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:31.538319 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:31.538243 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:31.833939 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:31.833846 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:32.260331 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:32.260293 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:33.025710 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:33.025676 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:33.924946 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:33.924910 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:34.511487 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.511448 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp"] Oct 31 13:45:34.511919 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.511905 6841 cpu_manager.go:414] "containerMap: removing container" podUID="8cc7973a-3561-4e4c-8817-9febedcd7ef7" containerName="cleanup" Oct 31 13:45:34.511977 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.511921 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cc7973a-3561-4e4c-8817-9febedcd7ef7" containerName="cleanup" Oct 31 13:45:34.511977 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.511936 6841 cpu_manager.go:414] "containerMap: removing container" podUID="8cc7973a-3561-4e4c-8817-9febedcd7ef7" containerName="cleanup" Oct 31 13:45:34.511977 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.511942 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cc7973a-3561-4e4c-8817-9febedcd7ef7" containerName="cleanup" Oct 31 13:45:34.511977 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.511954 6841 cpu_manager.go:414] "containerMap: removing container" podUID="51feb884-049e-45fb-a054-b2f8022622a9" containerName="revoke-keys" Oct 31 13:45:34.511977 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.511959 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51feb884-049e-45fb-a054-b2f8022622a9" containerName="revoke-keys" Oct 31 13:45:34.511977 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.511973 6841 cpu_manager.go:414] "containerMap: removing container" podUID="a753b639-2816-4910-b9e1-3bddbf22acd7" containerName="cleanup" Oct 31 13:45:34.511977 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.511994 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a753b639-2816-4910-b9e1-3bddbf22acd7" containerName="cleanup" Oct 31 13:45:34.512237 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.512019 6841 cpu_manager.go:414] "containerMap: removing container" podUID="cfb89104-d09a-4dd1-9f92-cc97d2866b18" containerName="cleanup" Oct 31 13:45:34.512237 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.512024 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfb89104-d09a-4dd1-9f92-cc97d2866b18" containerName="cleanup" Oct 31 13:45:34.512237 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.512088 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cc7973a-3561-4e4c-8817-9febedcd7ef7" containerName="cleanup" Oct 31 13:45:34.512237 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.512096 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cc7973a-3561-4e4c-8817-9febedcd7ef7" containerName="cleanup" Oct 31 13:45:34.512237 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.512103 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="51feb884-049e-45fb-a054-b2f8022622a9" containerName="revoke-keys" Oct 31 13:45:34.512237 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.512114 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="a753b639-2816-4910-b9e1-3bddbf22acd7" containerName="cleanup" Oct 31 13:45:34.512237 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.512121 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfb89104-d09a-4dd1-9f92-cc97d2866b18" containerName="cleanup" Oct 31 13:45:34.516676 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.516654 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" Oct 31 13:45:34.521216 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.521194 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-47bcd9-openshift-default-dockercfg-dwvfp\"" Oct 31 13:45:34.533617 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.533595 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv"] Oct 31 13:45:34.534068 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.534054 6841 cpu_manager.go:414] "containerMap: removing container" podUID="8cc7973a-3561-4e4c-8817-9febedcd7ef7" containerName="cleanup" Oct 31 13:45:34.534118 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.534070 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cc7973a-3561-4e4c-8817-9febedcd7ef7" containerName="cleanup" Oct 31 13:45:34.534169 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.534160 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cc7973a-3561-4e4c-8817-9febedcd7ef7" containerName="cleanup" Oct 31 13:45:34.537238 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.537222 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" Oct 31 13:45:34.543312 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.543289 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp"] Oct 31 13:45:34.559725 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.559700 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv"] Oct 31 13:45:34.617295 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.617264 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/375ff5f6-486e-434d-bf92-7b173e9f2eb3-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp\" (UID: \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" Oct 31 13:45:34.617470 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.617303 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1997f236-6ad0-4054-8193-ca7b9477fafb-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv\" (UID: \"1997f236-6ad0-4054-8193-ca7b9477fafb\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" Oct 31 13:45:34.617470 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.617335 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1997f236-6ad0-4054-8193-ca7b9477fafb-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv\" (UID: \"1997f236-6ad0-4054-8193-ca7b9477fafb\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" Oct 31 13:45:34.617470 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.617430 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/375ff5f6-486e-434d-bf92-7b173e9f2eb3-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp\" (UID: \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" Oct 31 13:45:34.617598 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.617517 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/375ff5f6-486e-434d-bf92-7b173e9f2eb3-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp\" (UID: \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" Oct 31 13:45:34.617598 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.617561 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1997f236-6ad0-4054-8193-ca7b9477fafb-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv\" (UID: \"1997f236-6ad0-4054-8193-ca7b9477fafb\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" Oct 31 13:45:34.617598 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.617588 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/375ff5f6-486e-434d-bf92-7b173e9f2eb3-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp\" (UID: \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" Oct 31 13:45:34.617713 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.617642 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1997f236-6ad0-4054-8193-ca7b9477fafb-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv\" (UID: \"1997f236-6ad0-4054-8193-ca7b9477fafb\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" Oct 31 13:45:34.617713 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.617669 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1997f236-6ad0-4054-8193-ca7b9477fafb-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv\" (UID: \"1997f236-6ad0-4054-8193-ca7b9477fafb\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" Oct 31 13:45:34.617788 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.617731 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1997f236-6ad0-4054-8193-ca7b9477fafb-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv\" (UID: \"1997f236-6ad0-4054-8193-ca7b9477fafb\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" Oct 31 13:45:34.617788 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.617751 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/375ff5f6-486e-434d-bf92-7b173e9f2eb3-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp\" (UID: \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" Oct 31 13:45:34.617788 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.617776 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/375ff5f6-486e-434d-bf92-7b173e9f2eb3-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp\" (UID: \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" Oct 31 13:45:34.617888 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.617848 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/375ff5f6-486e-434d-bf92-7b173e9f2eb3-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp\" (UID: \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" Oct 31 13:45:34.617888 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.617877 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1997f236-6ad0-4054-8193-ca7b9477fafb-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv\" (UID: \"1997f236-6ad0-4054-8193-ca7b9477fafb\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" Oct 31 13:45:34.617969 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.617897 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1997f236-6ad0-4054-8193-ca7b9477fafb-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv\" (UID: \"1997f236-6ad0-4054-8193-ca7b9477fafb\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" Oct 31 13:45:34.617969 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.617931 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/375ff5f6-486e-434d-bf92-7b173e9f2eb3-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp\" (UID: \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" Oct 31 13:45:34.618064 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.618021 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwz45\" (UniqueName: \"kubernetes.io/projected/1997f236-6ad0-4054-8193-ca7b9477fafb-kube-api-access-wwz45\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv\" (UID: \"1997f236-6ad0-4054-8193-ca7b9477fafb\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" Oct 31 13:45:34.618064 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.618050 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r54x2\" (UniqueName: \"kubernetes.io/projected/375ff5f6-486e-434d-bf92-7b173e9f2eb3-kube-api-access-r54x2\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp\" (UID: \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" Oct 31 13:45:34.719472 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.719430 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1997f236-6ad0-4054-8193-ca7b9477fafb-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv\" (UID: \"1997f236-6ad0-4054-8193-ca7b9477fafb\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" Oct 31 13:45:34.719472 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.719471 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1997f236-6ad0-4054-8193-ca7b9477fafb-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv\" (UID: \"1997f236-6ad0-4054-8193-ca7b9477fafb\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" Oct 31 13:45:34.719750 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.719489 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/375ff5f6-486e-434d-bf92-7b173e9f2eb3-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp\" (UID: \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" Oct 31 13:45:34.719750 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.719515 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/375ff5f6-486e-434d-bf92-7b173e9f2eb3-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp\" (UID: \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" Oct 31 13:45:34.719750 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.719560 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/375ff5f6-486e-434d-bf92-7b173e9f2eb3-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp\" (UID: \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" Oct 31 13:45:34.719750 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.719584 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1997f236-6ad0-4054-8193-ca7b9477fafb-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv\" (UID: \"1997f236-6ad0-4054-8193-ca7b9477fafb\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" Oct 31 13:45:34.719750 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.719612 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1997f236-6ad0-4054-8193-ca7b9477fafb-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv\" (UID: \"1997f236-6ad0-4054-8193-ca7b9477fafb\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" Oct 31 13:45:34.719750 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.719652 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/375ff5f6-486e-434d-bf92-7b173e9f2eb3-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp\" (UID: \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" Oct 31 13:45:34.719750 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.719697 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwz45\" (UniqueName: \"kubernetes.io/projected/1997f236-6ad0-4054-8193-ca7b9477fafb-kube-api-access-wwz45\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv\" (UID: \"1997f236-6ad0-4054-8193-ca7b9477fafb\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" Oct 31 13:45:34.720230 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.719962 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/375ff5f6-486e-434d-bf92-7b173e9f2eb3-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp\" (UID: \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" Oct 31 13:45:34.720230 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.720055 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r54x2\" (UniqueName: \"kubernetes.io/projected/375ff5f6-486e-434d-bf92-7b173e9f2eb3-kube-api-access-r54x2\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp\" (UID: \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" Oct 31 13:45:34.720230 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.720078 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1997f236-6ad0-4054-8193-ca7b9477fafb-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv\" (UID: \"1997f236-6ad0-4054-8193-ca7b9477fafb\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" Oct 31 13:45:34.720230 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.720115 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/375ff5f6-486e-434d-bf92-7b173e9f2eb3-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp\" (UID: \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" Oct 31 13:45:34.720230 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.720149 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1997f236-6ad0-4054-8193-ca7b9477fafb-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv\" (UID: \"1997f236-6ad0-4054-8193-ca7b9477fafb\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" Oct 31 13:45:34.720230 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.720182 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1997f236-6ad0-4054-8193-ca7b9477fafb-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv\" (UID: \"1997f236-6ad0-4054-8193-ca7b9477fafb\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" Oct 31 13:45:34.720230 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.720195 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/375ff5f6-486e-434d-bf92-7b173e9f2eb3-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp\" (UID: \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" Oct 31 13:45:34.720230 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.720209 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/375ff5f6-486e-434d-bf92-7b173e9f2eb3-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp\" (UID: \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" Oct 31 13:45:34.720679 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.720250 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/375ff5f6-486e-434d-bf92-7b173e9f2eb3-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp\" (UID: \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" Oct 31 13:45:34.720679 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.720283 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/375ff5f6-486e-434d-bf92-7b173e9f2eb3-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp\" (UID: \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" Oct 31 13:45:34.720679 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.720330 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1997f236-6ad0-4054-8193-ca7b9477fafb-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv\" (UID: \"1997f236-6ad0-4054-8193-ca7b9477fafb\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" Oct 31 13:45:34.720679 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.720374 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/375ff5f6-486e-434d-bf92-7b173e9f2eb3-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp\" (UID: \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" Oct 31 13:45:34.720679 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.720420 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/375ff5f6-486e-434d-bf92-7b173e9f2eb3-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp\" (UID: \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" Oct 31 13:45:34.720679 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.720431 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1997f236-6ad0-4054-8193-ca7b9477fafb-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv\" (UID: \"1997f236-6ad0-4054-8193-ca7b9477fafb\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" Oct 31 13:45:34.721059 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.720887 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1997f236-6ad0-4054-8193-ca7b9477fafb-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv\" (UID: \"1997f236-6ad0-4054-8193-ca7b9477fafb\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" Oct 31 13:45:34.721059 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.720897 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1997f236-6ad0-4054-8193-ca7b9477fafb-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv\" (UID: \"1997f236-6ad0-4054-8193-ca7b9477fafb\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" Oct 31 13:45:34.721059 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.721045 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1997f236-6ad0-4054-8193-ca7b9477fafb-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv\" (UID: \"1997f236-6ad0-4054-8193-ca7b9477fafb\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" Oct 31 13:45:34.721375 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.721353 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1997f236-6ad0-4054-8193-ca7b9477fafb-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv\" (UID: \"1997f236-6ad0-4054-8193-ca7b9477fafb\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" Oct 31 13:45:34.721460 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.721396 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/375ff5f6-486e-434d-bf92-7b173e9f2eb3-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp\" (UID: \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" Oct 31 13:45:34.722675 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.722647 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/375ff5f6-486e-434d-bf92-7b173e9f2eb3-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp\" (UID: \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" Oct 31 13:45:34.722943 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.722920 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/375ff5f6-486e-434d-bf92-7b173e9f2eb3-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp\" (UID: \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" Oct 31 13:45:34.723097 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.723028 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1997f236-6ad0-4054-8193-ca7b9477fafb-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv\" (UID: \"1997f236-6ad0-4054-8193-ca7b9477fafb\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" Oct 31 13:45:34.723640 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.723618 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1997f236-6ad0-4054-8193-ca7b9477fafb-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv\" (UID: \"1997f236-6ad0-4054-8193-ca7b9477fafb\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" Oct 31 13:45:34.732960 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.732934 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/375ff5f6-486e-434d-bf92-7b173e9f2eb3-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp\" (UID: \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" Oct 31 13:45:34.733144 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.733118 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r54x2\" (UniqueName: \"kubernetes.io/projected/375ff5f6-486e-434d-bf92-7b173e9f2eb3-kube-api-access-r54x2\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp\" (UID: \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" Oct 31 13:45:34.736912 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.736882 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwz45\" (UniqueName: \"kubernetes.io/projected/1997f236-6ad0-4054-8193-ca7b9477fafb-kube-api-access-wwz45\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv\" (UID: \"1997f236-6ad0-4054-8193-ca7b9477fafb\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" Oct 31 13:45:34.737236 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.737214 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1997f236-6ad0-4054-8193-ca7b9477fafb-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv\" (UID: \"1997f236-6ad0-4054-8193-ca7b9477fafb\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" Oct 31 13:45:34.738687 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.738662 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:34.831443 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.831367 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" Oct 31 13:45:34.852870 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:34.852842 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" Oct 31 13:45:35.122083 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:35.122021 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp"] Oct 31 13:45:35.122771 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:45:35.122719 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod375ff5f6_486e_434d_bf92_7b173e9f2eb3.slice/crio-7134d57363e5ec91c734a060a318b6a52465bac12490ceaf09ebef8ed46a48e9 WatchSource:0}: Error finding container 7134d57363e5ec91c734a060a318b6a52465bac12490ceaf09ebef8ed46a48e9: Status 404 returned error can't find the container with id 7134d57363e5ec91c734a060a318b6a52465bac12490ceaf09ebef8ed46a48e9 Oct 31 13:45:35.125898 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:35.125858 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:45:35.126033 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:35.125940 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:45:35.126033 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:35.125997 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:45:35.155474 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:35.155449 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv"] Oct 31 13:45:35.163235 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:45:35.163210 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1997f236_6ad0_4054_8193_ca7b9477fafb.slice/crio-565301c7066bae8051bba84c85391e11fa2e8c2a158281f525b11da8428524f0 WatchSource:0}: Error finding container 565301c7066bae8051bba84c85391e11fa2e8c2a158281f525b11da8428524f0: Status 404 returned error can't find the container with id 565301c7066bae8051bba84c85391e11fa2e8c2a158281f525b11da8428524f0 Oct 31 13:45:35.165683 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:35.165648 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:45:35.165805 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:35.165726 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:45:35.165805 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:35.165772 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:45:35.458418 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:35.458001 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" event={"ID":"1997f236-6ad0-4054-8193-ca7b9477fafb","Type":"ContainerStarted","Data":"4700ff9280cf13de5868f38024d37a4f0fc0e2b93058539038d48c9dcad837f4"} Oct 31 13:45:35.458418 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:35.458075 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" event={"ID":"1997f236-6ad0-4054-8193-ca7b9477fafb","Type":"ContainerStarted","Data":"565301c7066bae8051bba84c85391e11fa2e8c2a158281f525b11da8428524f0"} Oct 31 13:45:35.462185 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:35.461784 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" event={"ID":"375ff5f6-486e-434d-bf92-7b173e9f2eb3","Type":"ContainerStarted","Data":"867e7775ad0c52dc58141f54ad08e2958f9b3458be9e4239ba2c4bc9d134e765"} Oct 31 13:45:35.462185 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:35.461848 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" event={"ID":"375ff5f6-486e-434d-bf92-7b173e9f2eb3","Type":"ContainerStarted","Data":"7134d57363e5ec91c734a060a318b6a52465bac12490ceaf09ebef8ed46a48e9"} Oct 31 13:45:35.490740 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:35.490680 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" podStartSLOduration=1.4906613 podStartE2EDuration="1.4906613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:35.48929656 +0000 UTC m=+4381.972419460" watchObservedRunningTime="2026-10-31 13:45:35.4906613 +0000 UTC m=+4381.973784204" Oct 31 13:45:35.527528 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:35.527471 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:35.832161 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:35.832087 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" Oct 31 13:45:35.865007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:35.857079 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" Oct 31 13:45:36.184759 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:36.184667 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:36.716355 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:36.716306 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:36.837905 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:36.837871 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body= Oct 31 13:45:36.838235 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:36.838190 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" podUID="375ff5f6-486e-434d-bf92-7b173e9f2eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" Oct 31 13:45:36.864401 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:36.864365 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 31 13:45:36.864685 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:36.864650 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" podUID="1997f236-6ad0-4054-8193-ca7b9477fafb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 31 13:45:37.015410 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:37.014787 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:37.311163 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:37.311075 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:37.832487 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:37.832450 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body= Oct 31 13:45:37.832870 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:37.832511 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" podUID="375ff5f6-486e-434d-bf92-7b173e9f2eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" Oct 31 13:45:37.863623 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:37.863581 6841 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 31 13:45:37.863898 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:37.863865 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" podUID="1997f236-6ad0-4054-8193-ca7b9477fafb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 31 13:45:37.964007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:37.954320 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:38.195038 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:38.195003 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:38.490025 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:38.489923 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:38.807063 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:38.804529 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" podStartSLOduration=4.8045096 podStartE2EDuration="4.8045096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:35.512719175 +0000 UTC m=+4381.995842083" watchObservedRunningTime="2026-10-31 13:45:38.8045096 +0000 UTC m=+4385.287632506" Oct 31 13:45:38.814774 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:38.814742 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:38.815201 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:38.815175 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp"] Oct 31 13:45:38.815415 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:38.815389 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" podUID="375ff5f6-486e-434d-bf92-7b173e9f2eb3" containerName="istio-proxy" containerID="cri-o://867e7775ad0c52dc58141f54ad08e2958f9b3458be9e4239ba2c4bc9d134e765" gracePeriod=30 Oct 31 13:45:38.823471 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:38.823447 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv"] Oct 31 13:45:38.823639 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:38.823622 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" podUID="1997f236-6ad0-4054-8193-ca7b9477fafb" containerName="istio-proxy" containerID="cri-o://4700ff9280cf13de5868f38024d37a4f0fc0e2b93058539038d48c9dcad837f4" gracePeriod=30 Oct 31 13:45:39.126579 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:39.126500 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:39.337432 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:39.337404 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:40.406476 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:40.406444 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:40.684472 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:40.684384 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:40.980008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:40.972920 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:41.193297 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:41.193247 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:41.462003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:41.461950 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:41.953445 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:41.953412 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f"] Oct 31 13:45:41.953646 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:41.953624 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f" podUID="19ae4518-f78f-431b-a5a0-031805427fc8" containerName="maas-api" containerID="cri-o://faabd33c71743e2a63a646af54f7a5b616a716215c55e230c2522ba3151fa562" gracePeriod=30 Oct 31 13:45:42.090108 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:42.090073 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:42.245746 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:42.245715 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f" Oct 31 13:45:42.308964 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:42.308932 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19ae4518-f78f-431b-a5a0-031805427fc8-kube-api-access-s99x5\" (UniqueName: \"kubernetes.io/projected/19ae4518-f78f-431b-a5a0-031805427fc8-kube-api-access-s99x5\") pod \"19ae4518-f78f-431b-a5a0-031805427fc8\" (UID: \"19ae4518-f78f-431b-a5a0-031805427fc8\") " Oct 31 13:45:42.309118 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:42.308972 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19ae4518-f78f-431b-a5a0-031805427fc8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/19ae4518-f78f-431b-a5a0-031805427fc8-metrics-certs\") pod \"19ae4518-f78f-431b-a5a0-031805427fc8\" (UID: \"19ae4518-f78f-431b-a5a0-031805427fc8\") " Oct 31 13:45:42.309118 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:42.309021 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19ae4518-f78f-431b-a5a0-031805427fc8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19ae4518-f78f-431b-a5a0-031805427fc8-openshift-service-ca\") pod \"19ae4518-f78f-431b-a5a0-031805427fc8\" (UID: \"19ae4518-f78f-431b-a5a0-031805427fc8\") " Oct 31 13:45:42.309221 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:42.309204 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19ae4518-f78f-431b-a5a0-031805427fc8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19ae4518-f78f-431b-a5a0-031805427fc8-maas-api-tls\") pod \"19ae4518-f78f-431b-a5a0-031805427fc8\" (UID: \"19ae4518-f78f-431b-a5a0-031805427fc8\") " Oct 31 13:45:42.309376 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:42.309348 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19ae4518-f78f-431b-a5a0-031805427fc8-openshift-service-ca" pod "19ae4518-f78f-431b-a5a0-031805427fc8" (UID: "19ae4518-f78f-431b-a5a0-031805427fc8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:42.309656 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:42.309638 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19ae4518-f78f-431b-a5a0-031805427fc8-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:42.311170 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:42.311147 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19ae4518-f78f-431b-a5a0-031805427fc8-maas-api-tls" pod "19ae4518-f78f-431b-a5a0-031805427fc8" (UID: "19ae4518-f78f-431b-a5a0-031805427fc8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:45:42.311284 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:42.311267 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19ae4518-f78f-431b-a5a0-031805427fc8-kube-api-access-s99x5" pod "19ae4518-f78f-431b-a5a0-031805427fc8" (UID: "19ae4518-f78f-431b-a5a0-031805427fc8"). InnerVolumeSpecName "kube-api-access-s99x5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:42.311336 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:42.311281 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19ae4518-f78f-431b-a5a0-031805427fc8-metrics-certs" pod "19ae4518-f78f-431b-a5a0-031805427fc8" (UID: "19ae4518-f78f-431b-a5a0-031805427fc8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:45:42.411028 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:42.410966 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19ae4518-f78f-431b-a5a0-031805427fc8-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:42.411028 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:42.411023 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s99x5\" (UniqueName: \"kubernetes.io/projected/19ae4518-f78f-431b-a5a0-031805427fc8-kube-api-access-s99x5\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:42.411028 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:42.411035 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19ae4518-f78f-431b-a5a0-031805427fc8-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:42.531971 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:42.531885 6841 generic.go:356] "Generic (PLEG): container finished" podID="19ae4518-f78f-431b-a5a0-031805427fc8" containerID="faabd33c71743e2a63a646af54f7a5b616a716215c55e230c2522ba3151fa562" exitCode=0 Oct 31 13:45:42.531971 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:42.531938 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f" Oct 31 13:45:42.531971 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:42.531970 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f" event={"ID":"19ae4518-f78f-431b-a5a0-031805427fc8","Type":"ContainerDied","Data":"faabd33c71743e2a63a646af54f7a5b616a716215c55e230c2522ba3151fa562"} Oct 31 13:45:42.532489 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:42.532064 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f" event={"ID":"19ae4518-f78f-431b-a5a0-031805427fc8","Type":"ContainerDied","Data":"46885a66c156eb54f857570411315156c34d04a1e3564d190ffdb50e18e651b5"} Oct 31 13:45:42.532489 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:42.532088 6841 scope.go:122] "RemoveContainer" containerID="faabd33c71743e2a63a646af54f7a5b616a716215c55e230c2522ba3151fa562" Oct 31 13:45:42.542483 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:42.542464 6841 scope.go:122] "RemoveContainer" containerID="faabd33c71743e2a63a646af54f7a5b616a716215c55e230c2522ba3151fa562" Oct 31 13:45:42.542738 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:45:42.542720 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faabd33c71743e2a63a646af54f7a5b616a716215c55e230c2522ba3151fa562\": container with ID starting with faabd33c71743e2a63a646af54f7a5b616a716215c55e230c2522ba3151fa562 not found: ID does not exist" containerID="faabd33c71743e2a63a646af54f7a5b616a716215c55e230c2522ba3151fa562" Oct 31 13:45:42.542793 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:42.542748 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faabd33c71743e2a63a646af54f7a5b616a716215c55e230c2522ba3151fa562"} err="failed to get container status \"faabd33c71743e2a63a646af54f7a5b616a716215c55e230c2522ba3151fa562\": rpc error: code = NotFound desc = could not find container \"faabd33c71743e2a63a646af54f7a5b616a716215c55e230c2522ba3151fa562\": container with ID starting with faabd33c71743e2a63a646af54f7a5b616a716215c55e230c2522ba3151fa562 not found: ID does not exist" Oct 31 13:45:42.558781 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:42.558746 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f"] Oct 31 13:45:42.560928 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:42.560907 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-842ac9-5c7f6bd669-vmd9f"] Oct 31 13:45:42.572618 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:42.572594 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:43.529137 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:43.529090 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:43.643009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:43.642954 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19ae4518-f78f-431b-a5a0-031805427fc8" path="/var/lib/kubelet/pods/19ae4518-f78f-431b-a5a0-031805427fc8/volumes" Oct 31 13:45:44.126227 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:44.126181 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn" podUID="1dc69d96-bfd1-4ec9-8414-75553fde6d87" containerName="main" containerID="cri-o://b28d1b9a1243058fe374bbd228d42beba74afb4eea3ed115779ef0e227601da3" gracePeriod=45 Oct 31 13:45:44.390928 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:44.390843 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn" Oct 31 13:45:44.430739 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:44.430700 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dc69d96-bfd1-4ec9-8414-75553fde6d87-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dc69d96-bfd1-4ec9-8414-75553fde6d87-model-cache\") pod \"1dc69d96-bfd1-4ec9-8414-75553fde6d87\" (UID: \"1dc69d96-bfd1-4ec9-8414-75553fde6d87\") " Oct 31 13:45:44.430915 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:44.430818 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dc69d96-bfd1-4ec9-8414-75553fde6d87-tls-certs\" (UniqueName: \"kubernetes.io/secret/1dc69d96-bfd1-4ec9-8414-75553fde6d87-tls-certs\") pod \"1dc69d96-bfd1-4ec9-8414-75553fde6d87\" (UID: \"1dc69d96-bfd1-4ec9-8414-75553fde6d87\") " Oct 31 13:45:44.430915 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:44.430864 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dc69d96-bfd1-4ec9-8414-75553fde6d87-kube-api-access-g9qcv\" (UniqueName: \"kubernetes.io/projected/1dc69d96-bfd1-4ec9-8414-75553fde6d87-kube-api-access-g9qcv\") pod \"1dc69d96-bfd1-4ec9-8414-75553fde6d87\" (UID: \"1dc69d96-bfd1-4ec9-8414-75553fde6d87\") " Oct 31 13:45:44.431054 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:44.430921 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dc69d96-bfd1-4ec9-8414-75553fde6d87-home\" (UniqueName: \"kubernetes.io/empty-dir/1dc69d96-bfd1-4ec9-8414-75553fde6d87-home\") pod \"1dc69d96-bfd1-4ec9-8414-75553fde6d87\" (UID: \"1dc69d96-bfd1-4ec9-8414-75553fde6d87\") " Oct 31 13:45:44.431054 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:44.431035 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dc69d96-bfd1-4ec9-8414-75553fde6d87-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dc69d96-bfd1-4ec9-8414-75553fde6d87-tmp-dir\") pod \"1dc69d96-bfd1-4ec9-8414-75553fde6d87\" (UID: \"1dc69d96-bfd1-4ec9-8414-75553fde6d87\") " Oct 31 13:45:44.431174 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:44.431048 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc69d96-bfd1-4ec9-8414-75553fde6d87-model-cache" pod "1dc69d96-bfd1-4ec9-8414-75553fde6d87" (UID: "1dc69d96-bfd1-4ec9-8414-75553fde6d87"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:44.431174 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:44.431074 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dc69d96-bfd1-4ec9-8414-75553fde6d87-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dc69d96-bfd1-4ec9-8414-75553fde6d87-dshm\") pod \"1dc69d96-bfd1-4ec9-8414-75553fde6d87\" (UID: \"1dc69d96-bfd1-4ec9-8414-75553fde6d87\") " Oct 31 13:45:44.431296 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:44.431240 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc69d96-bfd1-4ec9-8414-75553fde6d87-home" pod "1dc69d96-bfd1-4ec9-8414-75553fde6d87" (UID: "1dc69d96-bfd1-4ec9-8414-75553fde6d87"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:44.431296 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:44.431278 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc69d96-bfd1-4ec9-8414-75553fde6d87-tmp-dir" pod "1dc69d96-bfd1-4ec9-8414-75553fde6d87" (UID: "1dc69d96-bfd1-4ec9-8414-75553fde6d87"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:44.431566 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:44.431540 6841 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dc69d96-bfd1-4ec9-8414-75553fde6d87-model-cache\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:44.431566 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:44.431567 6841 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dc69d96-bfd1-4ec9-8414-75553fde6d87-home\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:44.431769 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:44.431582 6841 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dc69d96-bfd1-4ec9-8414-75553fde6d87-tmp-dir\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:44.433440 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:44.433414 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dc69d96-bfd1-4ec9-8414-75553fde6d87-tls-certs" pod "1dc69d96-bfd1-4ec9-8414-75553fde6d87" (UID: "1dc69d96-bfd1-4ec9-8414-75553fde6d87"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:45:44.433570 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:44.433551 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc69d96-bfd1-4ec9-8414-75553fde6d87-dshm" pod "1dc69d96-bfd1-4ec9-8414-75553fde6d87" (UID: "1dc69d96-bfd1-4ec9-8414-75553fde6d87"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:44.433630 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:44.433564 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc69d96-bfd1-4ec9-8414-75553fde6d87-kube-api-access-g9qcv" pod "1dc69d96-bfd1-4ec9-8414-75553fde6d87" (UID: "1dc69d96-bfd1-4ec9-8414-75553fde6d87"). InnerVolumeSpecName "kube-api-access-g9qcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:44.434347 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:44.434311 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:44.532951 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:44.532921 6841 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dc69d96-bfd1-4ec9-8414-75553fde6d87-dshm\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:44.532951 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:44.532948 6841 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dc69d96-bfd1-4ec9-8414-75553fde6d87-tls-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:44.533408 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:44.532959 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9qcv\" (UniqueName: \"kubernetes.io/projected/1dc69d96-bfd1-4ec9-8414-75553fde6d87-kube-api-access-g9qcv\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:44.544435 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:44.544407 6841 generic.go:356] "Generic (PLEG): container finished" podID="1dc69d96-bfd1-4ec9-8414-75553fde6d87" containerID="b28d1b9a1243058fe374bbd228d42beba74afb4eea3ed115779ef0e227601da3" exitCode=0 Oct 31 13:45:44.544571 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:44.544452 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn" Oct 31 13:45:44.544571 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:44.544487 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn" event={"ID":"1dc69d96-bfd1-4ec9-8414-75553fde6d87","Type":"ContainerDied","Data":"b28d1b9a1243058fe374bbd228d42beba74afb4eea3ed115779ef0e227601da3"} Oct 31 13:45:44.544571 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:44.544549 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn" event={"ID":"1dc69d96-bfd1-4ec9-8414-75553fde6d87","Type":"ContainerDied","Data":"6401f6ffe868da5a402f87ab87860636bf3ee64620a54f18714823bfc447ccda"} Oct 31 13:45:44.544571 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:44.544569 6841 scope.go:122] "RemoveContainer" containerID="b28d1b9a1243058fe374bbd228d42beba74afb4eea3ed115779ef0e227601da3" Oct 31 13:45:44.554990 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:44.554963 6841 scope.go:122] "RemoveContainer" containerID="b28d1b9a1243058fe374bbd228d42beba74afb4eea3ed115779ef0e227601da3" Oct 31 13:45:44.555255 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:45:44.555236 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b28d1b9a1243058fe374bbd228d42beba74afb4eea3ed115779ef0e227601da3\": container with ID starting with b28d1b9a1243058fe374bbd228d42beba74afb4eea3ed115779ef0e227601da3 not found: ID does not exist" containerID="b28d1b9a1243058fe374bbd228d42beba74afb4eea3ed115779ef0e227601da3" Oct 31 13:45:44.555323 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:44.555268 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b28d1b9a1243058fe374bbd228d42beba74afb4eea3ed115779ef0e227601da3"} err="failed to get container status \"b28d1b9a1243058fe374bbd228d42beba74afb4eea3ed115779ef0e227601da3\": rpc error: code = NotFound desc = could not find container \"b28d1b9a1243058fe374bbd228d42beba74afb4eea3ed115779ef0e227601da3\": container with ID starting with b28d1b9a1243058fe374bbd228d42beba74afb4eea3ed115779ef0e227601da3 not found: ID does not exist" Oct 31 13:45:44.567596 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:44.567571 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn"] Oct 31 13:45:44.572041 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:44.572017 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-6c0612/sub-test-model-6c0612-kserve-5d8dcdc7cc-9mqnn"] Oct 31 13:45:45.443927 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:45.443880 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:45.641499 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:45.641467 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dc69d96-bfd1-4ec9-8414-75553fde6d87" path="/var/lib/kubelet/pods/1dc69d96-bfd1-4ec9-8414-75553fde6d87/volumes" Oct 31 13:45:46.230042 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:46.230008 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:46.927941 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:46.927831 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:47.000774 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.000751 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" Oct 31 13:45:47.029269 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.029242 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" Oct 31 13:45:47.058388 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.058356 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1997f236-6ad0-4054-8193-ca7b9477fafb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1997f236-6ad0-4054-8193-ca7b9477fafb-credential-socket\") pod \"1997f236-6ad0-4054-8193-ca7b9477fafb\" (UID: \"1997f236-6ad0-4054-8193-ca7b9477fafb\") " Oct 31 13:45:47.058541 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.058410 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/375ff5f6-486e-434d-bf92-7b173e9f2eb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/375ff5f6-486e-434d-bf92-7b173e9f2eb3-istiod-ca-cert\") pod \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\" (UID: \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\") " Oct 31 13:45:47.058541 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.058446 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/375ff5f6-486e-434d-bf92-7b173e9f2eb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/375ff5f6-486e-434d-bf92-7b173e9f2eb3-istio-podinfo\") pod \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\" (UID: \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\") " Oct 31 13:45:47.058541 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.058489 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/375ff5f6-486e-434d-bf92-7b173e9f2eb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/375ff5f6-486e-434d-bf92-7b173e9f2eb3-workload-socket\") pod \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\" (UID: \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\") " Oct 31 13:45:47.058541 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.058524 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1997f236-6ad0-4054-8193-ca7b9477fafb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1997f236-6ad0-4054-8193-ca7b9477fafb-workload-certs\") pod \"1997f236-6ad0-4054-8193-ca7b9477fafb\" (UID: \"1997f236-6ad0-4054-8193-ca7b9477fafb\") " Oct 31 13:45:47.058797 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.058568 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/375ff5f6-486e-434d-bf92-7b173e9f2eb3-kube-api-access-r54x2\" (UniqueName: \"kubernetes.io/projected/375ff5f6-486e-434d-bf92-7b173e9f2eb3-kube-api-access-r54x2\") pod \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\" (UID: \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\") " Oct 31 13:45:47.058797 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.058601 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1997f236-6ad0-4054-8193-ca7b9477fafb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1997f236-6ad0-4054-8193-ca7b9477fafb-istiod-ca-cert\") pod \"1997f236-6ad0-4054-8193-ca7b9477fafb\" (UID: \"1997f236-6ad0-4054-8193-ca7b9477fafb\") " Oct 31 13:45:47.058797 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.058629 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/375ff5f6-486e-434d-bf92-7b173e9f2eb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/375ff5f6-486e-434d-bf92-7b173e9f2eb3-credential-socket\") pod \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\" (UID: \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\") " Oct 31 13:45:47.058797 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.058656 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1997f236-6ad0-4054-8193-ca7b9477fafb-kube-api-access-wwz45\" (UniqueName: \"kubernetes.io/projected/1997f236-6ad0-4054-8193-ca7b9477fafb-kube-api-access-wwz45\") pod \"1997f236-6ad0-4054-8193-ca7b9477fafb\" (UID: \"1997f236-6ad0-4054-8193-ca7b9477fafb\") " Oct 31 13:45:47.058797 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.058688 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/375ff5f6-486e-434d-bf92-7b173e9f2eb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/375ff5f6-486e-434d-bf92-7b173e9f2eb3-workload-certs\") pod \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\" (UID: \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\") " Oct 31 13:45:47.058797 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.058742 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1997f236-6ad0-4054-8193-ca7b9477fafb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1997f236-6ad0-4054-8193-ca7b9477fafb-istio-podinfo\") pod \"1997f236-6ad0-4054-8193-ca7b9477fafb\" (UID: \"1997f236-6ad0-4054-8193-ca7b9477fafb\") " Oct 31 13:45:47.058797 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.058769 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/375ff5f6-486e-434d-bf92-7b173e9f2eb3-istio-token\" (UniqueName: \"kubernetes.io/projected/375ff5f6-486e-434d-bf92-7b173e9f2eb3-istio-token\") pod \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\" (UID: \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\") " Oct 31 13:45:47.059222 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.058765 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1997f236-6ad0-4054-8193-ca7b9477fafb-credential-socket" pod "1997f236-6ad0-4054-8193-ca7b9477fafb" (UID: "1997f236-6ad0-4054-8193-ca7b9477fafb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:47.059222 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.058828 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1997f236-6ad0-4054-8193-ca7b9477fafb-istio-token\" (UniqueName: \"kubernetes.io/projected/1997f236-6ad0-4054-8193-ca7b9477fafb-istio-token\") pod \"1997f236-6ad0-4054-8193-ca7b9477fafb\" (UID: \"1997f236-6ad0-4054-8193-ca7b9477fafb\") " Oct 31 13:45:47.059222 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.058856 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1997f236-6ad0-4054-8193-ca7b9477fafb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1997f236-6ad0-4054-8193-ca7b9477fafb-istio-data\") pod \"1997f236-6ad0-4054-8193-ca7b9477fafb\" (UID: \"1997f236-6ad0-4054-8193-ca7b9477fafb\") " Oct 31 13:45:47.059222 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.058829 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1997f236-6ad0-4054-8193-ca7b9477fafb-workload-certs" pod "1997f236-6ad0-4054-8193-ca7b9477fafb" (UID: "1997f236-6ad0-4054-8193-ca7b9477fafb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:47.059222 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.058859 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/375ff5f6-486e-434d-bf92-7b173e9f2eb3-istiod-ca-cert" pod "375ff5f6-486e-434d-bf92-7b173e9f2eb3" (UID: "375ff5f6-486e-434d-bf92-7b173e9f2eb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:47.059222 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.058924 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1997f236-6ad0-4054-8193-ca7b9477fafb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1997f236-6ad0-4054-8193-ca7b9477fafb-istio-envoy\") pod \"1997f236-6ad0-4054-8193-ca7b9477fafb\" (UID: \"1997f236-6ad0-4054-8193-ca7b9477fafb\") " Oct 31 13:45:47.059222 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.059026 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/375ff5f6-486e-434d-bf92-7b173e9f2eb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/375ff5f6-486e-434d-bf92-7b173e9f2eb3-istio-envoy\") pod \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\" (UID: \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\") " Oct 31 13:45:47.059222 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.059068 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1997f236-6ad0-4054-8193-ca7b9477fafb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1997f236-6ad0-4054-8193-ca7b9477fafb-workload-socket\") pod \"1997f236-6ad0-4054-8193-ca7b9477fafb\" (UID: \"1997f236-6ad0-4054-8193-ca7b9477fafb\") " Oct 31 13:45:47.059222 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.059150 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/375ff5f6-486e-434d-bf92-7b173e9f2eb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/375ff5f6-486e-434d-bf92-7b173e9f2eb3-istio-data\") pod \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\" (UID: \"375ff5f6-486e-434d-bf92-7b173e9f2eb3\") " Oct 31 13:45:47.059745 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.059572 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/375ff5f6-486e-434d-bf92-7b173e9f2eb3-workload-socket" pod "375ff5f6-486e-434d-bf92-7b173e9f2eb3" (UID: "375ff5f6-486e-434d-bf92-7b173e9f2eb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:47.059745 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.059651 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1997f236-6ad0-4054-8193-ca7b9477fafb-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:47.059745 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.059675 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/375ff5f6-486e-434d-bf92-7b173e9f2eb3-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:47.059745 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.059692 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1997f236-6ad0-4054-8193-ca7b9477fafb-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:47.059977 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.059774 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1997f236-6ad0-4054-8193-ca7b9477fafb-istio-data" pod "1997f236-6ad0-4054-8193-ca7b9477fafb" (UID: "1997f236-6ad0-4054-8193-ca7b9477fafb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:47.059977 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.059860 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/375ff5f6-486e-434d-bf92-7b173e9f2eb3-credential-socket" pod "375ff5f6-486e-434d-bf92-7b173e9f2eb3" (UID: "375ff5f6-486e-434d-bf92-7b173e9f2eb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:47.059977 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.059879 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1997f236-6ad0-4054-8193-ca7b9477fafb-istiod-ca-cert" pod "1997f236-6ad0-4054-8193-ca7b9477fafb" (UID: "1997f236-6ad0-4054-8193-ca7b9477fafb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:47.059977 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.059925 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1997f236-6ad0-4054-8193-ca7b9477fafb-workload-socket" pod "1997f236-6ad0-4054-8193-ca7b9477fafb" (UID: "1997f236-6ad0-4054-8193-ca7b9477fafb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:47.060604 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.060580 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/375ff5f6-486e-434d-bf92-7b173e9f2eb3-workload-certs" pod "375ff5f6-486e-434d-bf92-7b173e9f2eb3" (UID: "375ff5f6-486e-434d-bf92-7b173e9f2eb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:47.060740 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.060642 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/375ff5f6-486e-434d-bf92-7b173e9f2eb3-istio-data" pod "375ff5f6-486e-434d-bf92-7b173e9f2eb3" (UID: "375ff5f6-486e-434d-bf92-7b173e9f2eb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:47.062156 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.061833 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1997f236-6ad0-4054-8193-ca7b9477fafb-istio-token" pod "1997f236-6ad0-4054-8193-ca7b9477fafb" (UID: "1997f236-6ad0-4054-8193-ca7b9477fafb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:47.062156 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.062081 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1997f236-6ad0-4054-8193-ca7b9477fafb-istio-podinfo" pod "1997f236-6ad0-4054-8193-ca7b9477fafb" (UID: "1997f236-6ad0-4054-8193-ca7b9477fafb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:45:47.062156 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.062120 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/375ff5f6-486e-434d-bf92-7b173e9f2eb3-kube-api-access-r54x2" pod "375ff5f6-486e-434d-bf92-7b173e9f2eb3" (UID: "375ff5f6-486e-434d-bf92-7b173e9f2eb3"). InnerVolumeSpecName "kube-api-access-r54x2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:47.063015 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.062971 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1997f236-6ad0-4054-8193-ca7b9477fafb-kube-api-access-wwz45" pod "1997f236-6ad0-4054-8193-ca7b9477fafb" (UID: "1997f236-6ad0-4054-8193-ca7b9477fafb"). InnerVolumeSpecName "kube-api-access-wwz45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:47.063366 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.063344 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/375ff5f6-486e-434d-bf92-7b173e9f2eb3-istio-token" pod "375ff5f6-486e-434d-bf92-7b173e9f2eb3" (UID: "375ff5f6-486e-434d-bf92-7b173e9f2eb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:47.063533 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.063511 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/375ff5f6-486e-434d-bf92-7b173e9f2eb3-istio-podinfo" pod "375ff5f6-486e-434d-bf92-7b173e9f2eb3" (UID: "375ff5f6-486e-434d-bf92-7b173e9f2eb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:45:47.063590 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.063518 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/375ff5f6-486e-434d-bf92-7b173e9f2eb3-istio-envoy" pod "375ff5f6-486e-434d-bf92-7b173e9f2eb3" (UID: "375ff5f6-486e-434d-bf92-7b173e9f2eb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:47.063590 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.063576 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1997f236-6ad0-4054-8193-ca7b9477fafb-istio-envoy" pod "1997f236-6ad0-4054-8193-ca7b9477fafb" (UID: "1997f236-6ad0-4054-8193-ca7b9477fafb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:47.161243 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.161201 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/375ff5f6-486e-434d-bf92-7b173e9f2eb3-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:47.161243 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.161242 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/375ff5f6-486e-434d-bf92-7b173e9f2eb3-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:47.161481 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.161257 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/375ff5f6-486e-434d-bf92-7b173e9f2eb3-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:47.161481 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.161274 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r54x2\" (UniqueName: \"kubernetes.io/projected/375ff5f6-486e-434d-bf92-7b173e9f2eb3-kube-api-access-r54x2\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:47.161481 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.161292 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1997f236-6ad0-4054-8193-ca7b9477fafb-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:47.161481 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.161309 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/375ff5f6-486e-434d-bf92-7b173e9f2eb3-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:47.161481 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.161324 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwz45\" (UniqueName: \"kubernetes.io/projected/1997f236-6ad0-4054-8193-ca7b9477fafb-kube-api-access-wwz45\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:47.161481 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.161338 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/375ff5f6-486e-434d-bf92-7b173e9f2eb3-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:47.161481 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.161351 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1997f236-6ad0-4054-8193-ca7b9477fafb-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:47.161481 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.161365 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/375ff5f6-486e-434d-bf92-7b173e9f2eb3-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:47.161481 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.161379 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1997f236-6ad0-4054-8193-ca7b9477fafb-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:47.161481 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.161393 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1997f236-6ad0-4054-8193-ca7b9477fafb-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:47.161481 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.161408 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1997f236-6ad0-4054-8193-ca7b9477fafb-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:47.161481 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.161421 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/375ff5f6-486e-434d-bf92-7b173e9f2eb3-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:47.161481 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.161437 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1997f236-6ad0-4054-8193-ca7b9477fafb-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:47.564634 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.564602 6841 generic.go:356] "Generic (PLEG): container finished" podID="375ff5f6-486e-434d-bf92-7b173e9f2eb3" containerID="867e7775ad0c52dc58141f54ad08e2958f9b3458be9e4239ba2c4bc9d134e765" exitCode=0 Oct 31 13:45:47.564865 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.564659 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" Oct 31 13:45:47.564865 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.564692 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" event={"ID":"375ff5f6-486e-434d-bf92-7b173e9f2eb3","Type":"ContainerDied","Data":"867e7775ad0c52dc58141f54ad08e2958f9b3458be9e4239ba2c4bc9d134e765"} Oct 31 13:45:47.564865 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.564758 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp" event={"ID":"375ff5f6-486e-434d-bf92-7b173e9f2eb3","Type":"ContainerDied","Data":"7134d57363e5ec91c734a060a318b6a52465bac12490ceaf09ebef8ed46a48e9"} Oct 31 13:45:47.564865 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.564787 6841 scope.go:122] "RemoveContainer" containerID="867e7775ad0c52dc58141f54ad08e2958f9b3458be9e4239ba2c4bc9d134e765" Oct 31 13:45:47.566464 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.566439 6841 generic.go:356] "Generic (PLEG): container finished" podID="1997f236-6ad0-4054-8193-ca7b9477fafb" containerID="4700ff9280cf13de5868f38024d37a4f0fc0e2b93058539038d48c9dcad837f4" exitCode=0 Oct 31 13:45:47.566618 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.566528 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" Oct 31 13:45:47.566704 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.566529 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" event={"ID":"1997f236-6ad0-4054-8193-ca7b9477fafb","Type":"ContainerDied","Data":"4700ff9280cf13de5868f38024d37a4f0fc0e2b93058539038d48c9dcad837f4"} Oct 31 13:45:47.566851 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.566728 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv" event={"ID":"1997f236-6ad0-4054-8193-ca7b9477fafb","Type":"ContainerDied","Data":"565301c7066bae8051bba84c85391e11fa2e8c2a158281f525b11da8428524f0"} Oct 31 13:45:47.577649 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.577623 6841 scope.go:122] "RemoveContainer" containerID="867e7775ad0c52dc58141f54ad08e2958f9b3458be9e4239ba2c4bc9d134e765" Oct 31 13:45:47.577914 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:45:47.577891 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"867e7775ad0c52dc58141f54ad08e2958f9b3458be9e4239ba2c4bc9d134e765\": container with ID starting with 867e7775ad0c52dc58141f54ad08e2958f9b3458be9e4239ba2c4bc9d134e765 not found: ID does not exist" containerID="867e7775ad0c52dc58141f54ad08e2958f9b3458be9e4239ba2c4bc9d134e765" Oct 31 13:45:47.578045 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.577924 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"867e7775ad0c52dc58141f54ad08e2958f9b3458be9e4239ba2c4bc9d134e765"} err="failed to get container status \"867e7775ad0c52dc58141f54ad08e2958f9b3458be9e4239ba2c4bc9d134e765\": rpc error: code = NotFound desc = could not find container \"867e7775ad0c52dc58141f54ad08e2958f9b3458be9e4239ba2c4bc9d134e765\": container with ID starting with 867e7775ad0c52dc58141f54ad08e2958f9b3458be9e4239ba2c4bc9d134e765 not found: ID does not exist" Oct 31 13:45:47.578045 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.577947 6841 scope.go:122] "RemoveContainer" containerID="4700ff9280cf13de5868f38024d37a4f0fc0e2b93058539038d48c9dcad837f4" Oct 31 13:45:47.587649 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.587633 6841 scope.go:122] "RemoveContainer" containerID="4700ff9280cf13de5868f38024d37a4f0fc0e2b93058539038d48c9dcad837f4" Oct 31 13:45:47.587901 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:45:47.587878 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4700ff9280cf13de5868f38024d37a4f0fc0e2b93058539038d48c9dcad837f4\": container with ID starting with 4700ff9280cf13de5868f38024d37a4f0fc0e2b93058539038d48c9dcad837f4 not found: ID does not exist" containerID="4700ff9280cf13de5868f38024d37a4f0fc0e2b93058539038d48c9dcad837f4" Oct 31 13:45:47.588023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.587907 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4700ff9280cf13de5868f38024d37a4f0fc0e2b93058539038d48c9dcad837f4"} err="failed to get container status \"4700ff9280cf13de5868f38024d37a4f0fc0e2b93058539038d48c9dcad837f4\": rpc error: code = NotFound desc = could not find container \"4700ff9280cf13de5868f38024d37a4f0fc0e2b93058539038d48c9dcad837f4\": container with ID starting with 4700ff9280cf13de5868f38024d37a4f0fc0e2b93058539038d48c9dcad837f4 not found: ID does not exist" Oct 31 13:45:47.594019 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.593993 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp"] Oct 31 13:45:47.598239 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.597864 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:47.598374 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.598353 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-btvrp"] Oct 31 13:45:47.618958 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.618935 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv"] Oct 31 13:45:47.620934 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.620910 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-x9lzv"] Oct 31 13:45:47.641717 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.641688 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1997f236-6ad0-4054-8193-ca7b9477fafb" path="/var/lib/kubelet/pods/1997f236-6ad0-4054-8193-ca7b9477fafb/volumes" Oct 31 13:45:47.642336 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.642312 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="375ff5f6-486e-434d-bf92-7b173e9f2eb3" path="/var/lib/kubelet/pods/375ff5f6-486e-434d-bf92-7b173e9f2eb3/volumes" Oct 31 13:45:47.802795 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:47.802759 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:48.315504 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:48.315462 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:48.640610 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:48.640531 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:48.934140 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:48.934065 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:49.427010 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:49.426949 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:49.629436 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:49.629403 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:50.033943 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:45:50.033910 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:45:50.384431 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:50.384394 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:51.195526 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:51.195491 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:52.225264 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:52.225210 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:53.105523 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:53.105484 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:53.820940 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:53.820895 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:54.528796 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:54.528761 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:55.336190 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:55.336153 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:55.567280 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:55.567246 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:56.292204 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:56.292170 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:56.863353 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:56.863297 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:57.742026 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:57.741965 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:57.942748 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:57.942714 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:58.620242 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:58.620203 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:45:59.411729 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:45:59.411689 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:00.302728 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:00.302695 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:01.076654 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:01.076623 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:01.829961 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:01.829925 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:02.549209 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:02.549176 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:03.247129 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:03.247087 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:03.994267 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:03.994223 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:04.618859 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:04.618803 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:04.831713 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:04.831659 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:05.339833 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:05.339791 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:05.805756 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:05.805722 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d"] Oct 31 13:46:05.805998 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:05.805945 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" podUID="bdaa9d86-a826-4768-ae0d-471ce6e65e15" containerName="istio-proxy" containerID="cri-o://b8d8d040a18028bc816d2714d48c2058d0b268a7082d0d5c7fdf915039f8ec72" gracePeriod=30 Oct 31 13:46:05.881736 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:05.881692 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:06.200879 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:06.200844 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:06.725606 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:06.725576 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:06.941926 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:06.941893 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:07.766611 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:07.766573 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:08.531424 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:08.531360 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:09.296353 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:09.296319 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:10.088190 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:10.088157 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:10.968363 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:10.964866 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:11.063478 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.063453 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:46:11.119452 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.119375 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdaa9d86-a826-4768-ae0d-471ce6e65e15-kube-api-access-m4zsz\" (UniqueName: \"kubernetes.io/projected/bdaa9d86-a826-4768-ae0d-471ce6e65e15-kube-api-access-m4zsz\") pod \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\" (UID: \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\") " Oct 31 13:46:11.119452 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.119412 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdaa9d86-a826-4768-ae0d-471ce6e65e15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdaa9d86-a826-4768-ae0d-471ce6e65e15-credential-socket\") pod \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\" (UID: \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\") " Oct 31 13:46:11.119692 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.119461 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdaa9d86-a826-4768-ae0d-471ce6e65e15-istio-token\" (UniqueName: \"kubernetes.io/projected/bdaa9d86-a826-4768-ae0d-471ce6e65e15-istio-token\") pod \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\" (UID: \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\") " Oct 31 13:46:11.119692 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.119479 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdaa9d86-a826-4768-ae0d-471ce6e65e15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdaa9d86-a826-4768-ae0d-471ce6e65e15-workload-certs\") pod \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\" (UID: \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\") " Oct 31 13:46:11.119692 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.119519 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdaa9d86-a826-4768-ae0d-471ce6e65e15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdaa9d86-a826-4768-ae0d-471ce6e65e15-istio-envoy\") pod \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\" (UID: \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\") " Oct 31 13:46:11.119692 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.119581 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdaa9d86-a826-4768-ae0d-471ce6e65e15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdaa9d86-a826-4768-ae0d-471ce6e65e15-istio-data\") pod \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\" (UID: \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\") " Oct 31 13:46:11.119692 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.119653 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bdaa9d86-a826-4768-ae0d-471ce6e65e15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdaa9d86-a826-4768-ae0d-471ce6e65e15-istio-podinfo\") pod \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\" (UID: \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\") " Oct 31 13:46:11.119692 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.119683 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdaa9d86-a826-4768-ae0d-471ce6e65e15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdaa9d86-a826-4768-ae0d-471ce6e65e15-istiod-ca-cert\") pod \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\" (UID: \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\") " Oct 31 13:46:11.120063 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.119732 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdaa9d86-a826-4768-ae0d-471ce6e65e15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdaa9d86-a826-4768-ae0d-471ce6e65e15-workload-socket\") pod \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\" (UID: \"bdaa9d86-a826-4768-ae0d-471ce6e65e15\") " Oct 31 13:46:11.120063 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.119908 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdaa9d86-a826-4768-ae0d-471ce6e65e15-istio-data" pod "bdaa9d86-a826-4768-ae0d-471ce6e65e15" (UID: "bdaa9d86-a826-4768-ae0d-471ce6e65e15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:11.120199 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.120176 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdaa9d86-a826-4768-ae0d-471ce6e65e15-workload-socket" pod "bdaa9d86-a826-4768-ae0d-471ce6e65e15" (UID: "bdaa9d86-a826-4768-ae0d-471ce6e65e15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:11.120258 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.120216 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdaa9d86-a826-4768-ae0d-471ce6e65e15-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:11.120258 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.119919 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdaa9d86-a826-4768-ae0d-471ce6e65e15-workload-certs" pod "bdaa9d86-a826-4768-ae0d-471ce6e65e15" (UID: "bdaa9d86-a826-4768-ae0d-471ce6e65e15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:11.120377 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.120305 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdaa9d86-a826-4768-ae0d-471ce6e65e15-credential-socket" pod "bdaa9d86-a826-4768-ae0d-471ce6e65e15" (UID: "bdaa9d86-a826-4768-ae0d-471ce6e65e15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:11.120495 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.120474 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdaa9d86-a826-4768-ae0d-471ce6e65e15-istiod-ca-cert" pod "bdaa9d86-a826-4768-ae0d-471ce6e65e15" (UID: "bdaa9d86-a826-4768-ae0d-471ce6e65e15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:11.122041 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.122014 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdaa9d86-a826-4768-ae0d-471ce6e65e15-istio-token" pod "bdaa9d86-a826-4768-ae0d-471ce6e65e15" (UID: "bdaa9d86-a826-4768-ae0d-471ce6e65e15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:11.122345 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.122319 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdaa9d86-a826-4768-ae0d-471ce6e65e15-kube-api-access-m4zsz" pod "bdaa9d86-a826-4768-ae0d-471ce6e65e15" (UID: "bdaa9d86-a826-4768-ae0d-471ce6e65e15"). InnerVolumeSpecName "kube-api-access-m4zsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:11.122470 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.122355 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdaa9d86-a826-4768-ae0d-471ce6e65e15-istio-envoy" pod "bdaa9d86-a826-4768-ae0d-471ce6e65e15" (UID: "bdaa9d86-a826-4768-ae0d-471ce6e65e15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:11.122470 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.122460 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdaa9d86-a826-4768-ae0d-471ce6e65e15-istio-podinfo" pod "bdaa9d86-a826-4768-ae0d-471ce6e65e15" (UID: "bdaa9d86-a826-4768-ae0d-471ce6e65e15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:46:11.221744 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.221709 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdaa9d86-a826-4768-ae0d-471ce6e65e15-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:11.221744 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.221738 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdaa9d86-a826-4768-ae0d-471ce6e65e15-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:11.221744 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.221747 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdaa9d86-a826-4768-ae0d-471ce6e65e15-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:11.222005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.221756 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdaa9d86-a826-4768-ae0d-471ce6e65e15-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:11.222005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.221765 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4zsz\" (UniqueName: \"kubernetes.io/projected/bdaa9d86-a826-4768-ae0d-471ce6e65e15-kube-api-access-m4zsz\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:11.222005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.221776 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdaa9d86-a826-4768-ae0d-471ce6e65e15-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:11.222005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.221784 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdaa9d86-a826-4768-ae0d-471ce6e65e15-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:11.222005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.221792 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdaa9d86-a826-4768-ae0d-471ce6e65e15-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:11.720035 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.719998 6841 generic.go:356] "Generic (PLEG): container finished" podID="bdaa9d86-a826-4768-ae0d-471ce6e65e15" containerID="b8d8d040a18028bc816d2714d48c2058d0b268a7082d0d5c7fdf915039f8ec72" exitCode=0 Oct 31 13:46:11.720242 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.720049 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" Oct 31 13:46:11.720242 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.720088 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" event={"ID":"bdaa9d86-a826-4768-ae0d-471ce6e65e15","Type":"ContainerDied","Data":"b8d8d040a18028bc816d2714d48c2058d0b268a7082d0d5c7fdf915039f8ec72"} Oct 31 13:46:11.720242 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.720146 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d" event={"ID":"bdaa9d86-a826-4768-ae0d-471ce6e65e15","Type":"ContainerDied","Data":"f2072ebde24061bb88ed6424649003e54210db00862a65744a90c90ae026ad6d"} Oct 31 13:46:11.720242 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.720168 6841 scope.go:122] "RemoveContainer" containerID="b8d8d040a18028bc816d2714d48c2058d0b268a7082d0d5c7fdf915039f8ec72" Oct 31 13:46:11.729872 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.729848 6841 scope.go:122] "RemoveContainer" containerID="0b855bab5c92549530dbfaddf0071e0362ee969a7fc687a03a902ac4b17cabcd" Oct 31 13:46:11.738299 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.738281 6841 scope.go:122] "RemoveContainer" containerID="b8d8d040a18028bc816d2714d48c2058d0b268a7082d0d5c7fdf915039f8ec72" Oct 31 13:46:11.738554 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:46:11.738537 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8d8d040a18028bc816d2714d48c2058d0b268a7082d0d5c7fdf915039f8ec72\": container with ID starting with b8d8d040a18028bc816d2714d48c2058d0b268a7082d0d5c7fdf915039f8ec72 not found: ID does not exist" containerID="b8d8d040a18028bc816d2714d48c2058d0b268a7082d0d5c7fdf915039f8ec72" Oct 31 13:46:11.738605 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.738563 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8d8d040a18028bc816d2714d48c2058d0b268a7082d0d5c7fdf915039f8ec72"} err="failed to get container status \"b8d8d040a18028bc816d2714d48c2058d0b268a7082d0d5c7fdf915039f8ec72\": rpc error: code = NotFound desc = could not find container \"b8d8d040a18028bc816d2714d48c2058d0b268a7082d0d5c7fdf915039f8ec72\": container with ID starting with b8d8d040a18028bc816d2714d48c2058d0b268a7082d0d5c7fdf915039f8ec72 not found: ID does not exist" Oct 31 13:46:11.738605 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.738580 6841 scope.go:122] "RemoveContainer" containerID="0b855bab5c92549530dbfaddf0071e0362ee969a7fc687a03a902ac4b17cabcd" Oct 31 13:46:11.738814 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:46:11.738791 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b855bab5c92549530dbfaddf0071e0362ee969a7fc687a03a902ac4b17cabcd\": container with ID starting with 0b855bab5c92549530dbfaddf0071e0362ee969a7fc687a03a902ac4b17cabcd not found: ID does not exist" containerID="0b855bab5c92549530dbfaddf0071e0362ee969a7fc687a03a902ac4b17cabcd" Oct 31 13:46:11.738856 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.738821 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b855bab5c92549530dbfaddf0071e0362ee969a7fc687a03a902ac4b17cabcd"} err="failed to get container status \"0b855bab5c92549530dbfaddf0071e0362ee969a7fc687a03a902ac4b17cabcd\": rpc error: code = NotFound desc = could not find container \"0b855bab5c92549530dbfaddf0071e0362ee969a7fc687a03a902ac4b17cabcd\": container with ID starting with 0b855bab5c92549530dbfaddf0071e0362ee969a7fc687a03a902ac4b17cabcd not found: ID does not exist" Oct 31 13:46:11.750490 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.750469 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d"] Oct 31 13:46:11.760511 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.760489 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-79n7d"] Oct 31 13:46:11.762132 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:11.762090 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:12.616610 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:12.616573 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:13.488049 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:13.488011 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:13.641761 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:13.641731 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdaa9d86-a826-4768-ae0d-471ce6e65e15" path="/var/lib/kubelet/pods/bdaa9d86-a826-4768-ae0d-471ce6e65e15/volumes" Oct 31 13:46:13.855228 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:13.855143 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:14.297631 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:14.297596 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt"] Oct 31 13:46:14.297823 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:14.297800 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt" podUID="909de922-16af-4ee1-b6cf-375ab5fb6053" containerName="maas-api" containerID="cri-o://0135250c8edf94c11553b4e7bc62b863749593952676bac1268674ee970f0730" gracePeriod=30 Oct 31 13:46:14.492542 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:14.492447 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:14.566362 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:14.566337 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt" Oct 31 13:46:14.654082 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:14.654047 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/909de922-16af-4ee1-b6cf-375ab5fb6053-metrics-certs\" (UniqueName: \"kubernetes.io/secret/909de922-16af-4ee1-b6cf-375ab5fb6053-metrics-certs\") pod \"909de922-16af-4ee1-b6cf-375ab5fb6053\" (UID: \"909de922-16af-4ee1-b6cf-375ab5fb6053\") " Oct 31 13:46:14.654489 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:14.654113 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/909de922-16af-4ee1-b6cf-375ab5fb6053-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/909de922-16af-4ee1-b6cf-375ab5fb6053-maas-api-tls\") pod \"909de922-16af-4ee1-b6cf-375ab5fb6053\" (UID: \"909de922-16af-4ee1-b6cf-375ab5fb6053\") " Oct 31 13:46:14.654489 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:14.654195 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/909de922-16af-4ee1-b6cf-375ab5fb6053-kube-api-access-drx7d\" (UniqueName: \"kubernetes.io/projected/909de922-16af-4ee1-b6cf-375ab5fb6053-kube-api-access-drx7d\") pod \"909de922-16af-4ee1-b6cf-375ab5fb6053\" (UID: \"909de922-16af-4ee1-b6cf-375ab5fb6053\") " Oct 31 13:46:14.654489 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:14.654247 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/909de922-16af-4ee1-b6cf-375ab5fb6053-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/909de922-16af-4ee1-b6cf-375ab5fb6053-openshift-service-ca\") pod \"909de922-16af-4ee1-b6cf-375ab5fb6053\" (UID: \"909de922-16af-4ee1-b6cf-375ab5fb6053\") " Oct 31 13:46:14.654676 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:14.654628 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/909de922-16af-4ee1-b6cf-375ab5fb6053-openshift-service-ca" pod "909de922-16af-4ee1-b6cf-375ab5fb6053" (UID: "909de922-16af-4ee1-b6cf-375ab5fb6053"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:14.656333 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:14.656308 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/909de922-16af-4ee1-b6cf-375ab5fb6053-metrics-certs" pod "909de922-16af-4ee1-b6cf-375ab5fb6053" (UID: "909de922-16af-4ee1-b6cf-375ab5fb6053"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:46:14.656333 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:14.656322 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/909de922-16af-4ee1-b6cf-375ab5fb6053-maas-api-tls" pod "909de922-16af-4ee1-b6cf-375ab5fb6053" (UID: "909de922-16af-4ee1-b6cf-375ab5fb6053"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:46:14.656489 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:14.656421 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/909de922-16af-4ee1-b6cf-375ab5fb6053-kube-api-access-drx7d" pod "909de922-16af-4ee1-b6cf-375ab5fb6053" (UID: "909de922-16af-4ee1-b6cf-375ab5fb6053"). InnerVolumeSpecName "kube-api-access-drx7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:14.741520 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:14.741482 6841 generic.go:356] "Generic (PLEG): container finished" podID="909de922-16af-4ee1-b6cf-375ab5fb6053" containerID="0135250c8edf94c11553b4e7bc62b863749593952676bac1268674ee970f0730" exitCode=0 Oct 31 13:46:14.741753 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:14.741557 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt" event={"ID":"909de922-16af-4ee1-b6cf-375ab5fb6053","Type":"ContainerDied","Data":"0135250c8edf94c11553b4e7bc62b863749593952676bac1268674ee970f0730"} Oct 31 13:46:14.741833 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:14.741817 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt" Oct 31 13:46:14.743660 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:14.743631 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt" event={"ID":"909de922-16af-4ee1-b6cf-375ab5fb6053","Type":"ContainerDied","Data":"ebfb8aade2604a876881254f7df85154cf8394348aee56919e64a9c1f6f7ab1d"} Oct 31 13:46:14.743824 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:14.743679 6841 scope.go:122] "RemoveContainer" containerID="0135250c8edf94c11553b4e7bc62b863749593952676bac1268674ee970f0730" Oct 31 13:46:14.754721 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:14.754694 6841 scope.go:122] "RemoveContainer" containerID="0135250c8edf94c11553b4e7bc62b863749593952676bac1268674ee970f0730" Oct 31 13:46:14.755024 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:46:14.755004 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0135250c8edf94c11553b4e7bc62b863749593952676bac1268674ee970f0730\": container with ID starting with 0135250c8edf94c11553b4e7bc62b863749593952676bac1268674ee970f0730 not found: ID does not exist" containerID="0135250c8edf94c11553b4e7bc62b863749593952676bac1268674ee970f0730" Oct 31 13:46:14.755102 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:14.755032 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0135250c8edf94c11553b4e7bc62b863749593952676bac1268674ee970f0730"} err="failed to get container status \"0135250c8edf94c11553b4e7bc62b863749593952676bac1268674ee970f0730\": rpc error: code = NotFound desc = could not find container \"0135250c8edf94c11553b4e7bc62b863749593952676bac1268674ee970f0730\": container with ID starting with 0135250c8edf94c11553b4e7bc62b863749593952676bac1268674ee970f0730 not found: ID does not exist" Oct 31 13:46:14.755243 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:14.755223 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drx7d\" (UniqueName: \"kubernetes.io/projected/909de922-16af-4ee1-b6cf-375ab5fb6053-kube-api-access-drx7d\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:14.755286 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:14.755247 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/909de922-16af-4ee1-b6cf-375ab5fb6053-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:14.755286 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:14.755259 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/909de922-16af-4ee1-b6cf-375ab5fb6053-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:14.755286 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:14.755268 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/909de922-16af-4ee1-b6cf-375ab5fb6053-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:14.774958 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:14.774925 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt"] Oct 31 13:46:14.784109 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:14.784083 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0612-78f9cf6cb6-865jt"] Oct 31 13:46:14.958110 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:14.958075 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:15.476509 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:15.476472 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:15.644188 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:15.641954 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="909de922-16af-4ee1-b6cf-375ab5fb6053" path="/var/lib/kubelet/pods/909de922-16af-4ee1-b6cf-375ab5fb6053/volumes" Oct 31 13:46:16.134430 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:16.134391 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:16.995877 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:16.995837 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:17.969407 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:17.969364 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:18.850918 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:18.850881 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:19.641271 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:19.641242 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:20.034397 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:46:20.034365 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:46:20.550768 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:20.550701 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:21.617782 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:21.617738 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:22.343849 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:22.343818 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:23.150855 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:23.150813 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:23.374371 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:23.374337 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:23.815620 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:23.815584 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:24.065082 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:24.065049 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44"] Oct 31 13:46:24.065301 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:24.065277 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" podUID="fbae99f9-22a1-49d4-a08d-3bc7a0f69bde" containerName="istio-proxy" containerID="cri-o://829b0011aedad82c53d858697c31defb37c628f815a56da8935a7c227206d6f4" gracePeriod=30 Oct 31 13:46:24.139374 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:24.139340 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:24.414277 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:24.414179 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:25.102199 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:25.102160 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:25.451153 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:25.451106 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:25.529077 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:25.529041 6841 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:46:25.529253 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:25.529097 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" podUID="fbae99f9-22a1-49d4-a08d-3bc7a0f69bde" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:46:26.279605 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:26.279569 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:27.243090 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:27.243051 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:28.085516 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:28.085480 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:28.923105 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:28.923072 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:29.316478 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.316451 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:46:29.411421 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.411386 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-istio-podinfo\") pod \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\" (UID: \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\") " Oct 31 13:46:29.411609 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.411434 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-credential-socket\") pod \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\" (UID: \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\") " Oct 31 13:46:29.411609 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.411479 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-istio-data\") pod \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\" (UID: \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\") " Oct 31 13:46:29.411609 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.411536 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-istio-envoy\") pod \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\" (UID: \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\") " Oct 31 13:46:29.411609 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.411584 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-kube-api-access-hcgzd\" (UniqueName: \"kubernetes.io/projected/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-kube-api-access-hcgzd\") pod \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\" (UID: \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\") " Oct 31 13:46:29.411855 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.411611 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-workload-socket\") pod \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\" (UID: \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\") " Oct 31 13:46:29.411855 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.411661 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-istio-token\" (UniqueName: \"kubernetes.io/projected/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-istio-token\") pod \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\" (UID: \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\") " Oct 31 13:46:29.411855 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.411689 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-workload-certs\") pod \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\" (UID: \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\") " Oct 31 13:46:29.411855 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.411722 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-istiod-ca-cert\") pod \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\" (UID: \"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde\") " Oct 31 13:46:29.411855 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.411820 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-credential-socket" pod "fbae99f9-22a1-49d4-a08d-3bc7a0f69bde" (UID: "fbae99f9-22a1-49d4-a08d-3bc7a0f69bde"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:29.411855 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.411833 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-istio-data" pod "fbae99f9-22a1-49d4-a08d-3bc7a0f69bde" (UID: "fbae99f9-22a1-49d4-a08d-3bc7a0f69bde"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:29.412205 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.412080 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-workload-socket" pod "fbae99f9-22a1-49d4-a08d-3bc7a0f69bde" (UID: "fbae99f9-22a1-49d4-a08d-3bc7a0f69bde"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:29.412278 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.412209 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-workload-certs" pod "fbae99f9-22a1-49d4-a08d-3bc7a0f69bde" (UID: "fbae99f9-22a1-49d4-a08d-3bc7a0f69bde"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:29.412340 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.412277 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:29.412340 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.412297 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:29.412340 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.412312 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:29.412340 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.412318 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-istiod-ca-cert" pod "fbae99f9-22a1-49d4-a08d-3bc7a0f69bde" (UID: "fbae99f9-22a1-49d4-a08d-3bc7a0f69bde"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:29.413955 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.413930 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-istio-envoy" pod "fbae99f9-22a1-49d4-a08d-3bc7a0f69bde" (UID: "fbae99f9-22a1-49d4-a08d-3bc7a0f69bde"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:29.414262 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.414244 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-istio-token" pod "fbae99f9-22a1-49d4-a08d-3bc7a0f69bde" (UID: "fbae99f9-22a1-49d4-a08d-3bc7a0f69bde"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:29.414349 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.414284 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-istio-podinfo" pod "fbae99f9-22a1-49d4-a08d-3bc7a0f69bde" (UID: "fbae99f9-22a1-49d4-a08d-3bc7a0f69bde"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:46:29.414947 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.414934 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-kube-api-access-hcgzd" pod "fbae99f9-22a1-49d4-a08d-3bc7a0f69bde" (UID: "fbae99f9-22a1-49d4-a08d-3bc7a0f69bde"). InnerVolumeSpecName "kube-api-access-hcgzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:29.513651 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.513615 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:29.513651 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.513645 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:29.513651 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.513656 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcgzd\" (UniqueName: \"kubernetes.io/projected/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-kube-api-access-hcgzd\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:29.513900 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.513665 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:29.513900 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.513675 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:29.513900 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.513684 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:29.627399 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.627359 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:29.831061 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.828525 6841 generic.go:356] "Generic (PLEG): container finished" podID="fbae99f9-22a1-49d4-a08d-3bc7a0f69bde" containerID="829b0011aedad82c53d858697c31defb37c628f815a56da8935a7c227206d6f4" exitCode=0 Oct 31 13:46:29.831061 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.828775 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" event={"ID":"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde","Type":"ContainerDied","Data":"829b0011aedad82c53d858697c31defb37c628f815a56da8935a7c227206d6f4"} Oct 31 13:46:29.831061 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.828878 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" event={"ID":"fbae99f9-22a1-49d4-a08d-3bc7a0f69bde","Type":"ContainerDied","Data":"9a5d55f7a4a026e1819333294bc267627934a02af99e913c505255e55e99c62a"} Oct 31 13:46:29.831061 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.828928 6841 scope.go:122] "RemoveContainer" containerID="829b0011aedad82c53d858697c31defb37c628f815a56da8935a7c227206d6f4" Oct 31 13:46:29.831061 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.829234 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44" Oct 31 13:46:29.845910 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.845877 6841 scope.go:122] "RemoveContainer" containerID="48ec3c10d81300a560938fa16eecf19be41d253c74ae0a5c468d9456574975d4" Oct 31 13:46:29.854729 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.854706 6841 scope.go:122] "RemoveContainer" containerID="829b0011aedad82c53d858697c31defb37c628f815a56da8935a7c227206d6f4" Oct 31 13:46:29.855031 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:46:29.855008 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"829b0011aedad82c53d858697c31defb37c628f815a56da8935a7c227206d6f4\": container with ID starting with 829b0011aedad82c53d858697c31defb37c628f815a56da8935a7c227206d6f4 not found: ID does not exist" containerID="829b0011aedad82c53d858697c31defb37c628f815a56da8935a7c227206d6f4" Oct 31 13:46:29.855134 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.855037 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"829b0011aedad82c53d858697c31defb37c628f815a56da8935a7c227206d6f4"} err="failed to get container status \"829b0011aedad82c53d858697c31defb37c628f815a56da8935a7c227206d6f4\": rpc error: code = NotFound desc = could not find container \"829b0011aedad82c53d858697c31defb37c628f815a56da8935a7c227206d6f4\": container with ID starting with 829b0011aedad82c53d858697c31defb37c628f815a56da8935a7c227206d6f4 not found: ID does not exist" Oct 31 13:46:29.855134 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.855055 6841 scope.go:122] "RemoveContainer" containerID="48ec3c10d81300a560938fa16eecf19be41d253c74ae0a5c468d9456574975d4" Oct 31 13:46:29.855319 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:46:29.855302 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48ec3c10d81300a560938fa16eecf19be41d253c74ae0a5c468d9456574975d4\": container with ID starting with 48ec3c10d81300a560938fa16eecf19be41d253c74ae0a5c468d9456574975d4 not found: ID does not exist" containerID="48ec3c10d81300a560938fa16eecf19be41d253c74ae0a5c468d9456574975d4" Oct 31 13:46:29.855361 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.855325 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48ec3c10d81300a560938fa16eecf19be41d253c74ae0a5c468d9456574975d4"} err="failed to get container status \"48ec3c10d81300a560938fa16eecf19be41d253c74ae0a5c468d9456574975d4\": rpc error: code = NotFound desc = could not find container \"48ec3c10d81300a560938fa16eecf19be41d253c74ae0a5c468d9456574975d4\": container with ID starting with 48ec3c10d81300a560938fa16eecf19be41d253c74ae0a5c468d9456574975d4 not found: ID does not exist" Oct 31 13:46:29.883648 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.883624 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44"] Oct 31 13:46:29.904009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:29.903971 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-lhh44"] Oct 31 13:46:30.550796 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:30.550757 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:31.353140 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:31.353104 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:31.555484 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:31.555452 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:31.641907 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:31.641829 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbae99f9-22a1-49d4-a08d-3bc7a0f69bde" path="/var/lib/kubelet/pods/fbae99f9-22a1-49d4-a08d-3bc7a0f69bde/volumes" Oct 31 13:46:31.763287 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:31.763253 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:32.024881 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:32.024839 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc"] Oct 31 13:46:32.025144 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:32.025116 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc" podUID="444d54fd-04f0-4bca-9cb0-ec5cce618593" containerName="maas-api" containerID="cri-o://4fc78dcd62305ffc52201e24ffba76dcb821175fcc28fa65ef79dac33573aabe" gracePeriod=30 Oct 31 13:46:32.175399 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:32.175370 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:32.284399 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:32.284335 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc" Oct 31 13:46:32.377180 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:32.377142 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:32.449236 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:32.449201 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/444d54fd-04f0-4bca-9cb0-ec5cce618593-metrics-certs\" (UniqueName: \"kubernetes.io/secret/444d54fd-04f0-4bca-9cb0-ec5cce618593-metrics-certs\") pod \"444d54fd-04f0-4bca-9cb0-ec5cce618593\" (UID: \"444d54fd-04f0-4bca-9cb0-ec5cce618593\") " Oct 31 13:46:32.449403 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:32.449289 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/444d54fd-04f0-4bca-9cb0-ec5cce618593-kube-api-access-5whbb\" (UniqueName: \"kubernetes.io/projected/444d54fd-04f0-4bca-9cb0-ec5cce618593-kube-api-access-5whbb\") pod \"444d54fd-04f0-4bca-9cb0-ec5cce618593\" (UID: \"444d54fd-04f0-4bca-9cb0-ec5cce618593\") " Oct 31 13:46:32.449403 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:32.449330 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/444d54fd-04f0-4bca-9cb0-ec5cce618593-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/444d54fd-04f0-4bca-9cb0-ec5cce618593-maas-api-tls\") pod \"444d54fd-04f0-4bca-9cb0-ec5cce618593\" (UID: \"444d54fd-04f0-4bca-9cb0-ec5cce618593\") " Oct 31 13:46:32.449497 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:32.449434 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/444d54fd-04f0-4bca-9cb0-ec5cce618593-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/444d54fd-04f0-4bca-9cb0-ec5cce618593-openshift-service-ca\") pod \"444d54fd-04f0-4bca-9cb0-ec5cce618593\" (UID: \"444d54fd-04f0-4bca-9cb0-ec5cce618593\") " Oct 31 13:46:32.449822 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:32.449799 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/444d54fd-04f0-4bca-9cb0-ec5cce618593-openshift-service-ca" pod "444d54fd-04f0-4bca-9cb0-ec5cce618593" (UID: "444d54fd-04f0-4bca-9cb0-ec5cce618593"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:32.451458 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:32.451427 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/444d54fd-04f0-4bca-9cb0-ec5cce618593-metrics-certs" pod "444d54fd-04f0-4bca-9cb0-ec5cce618593" (UID: "444d54fd-04f0-4bca-9cb0-ec5cce618593"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:46:32.451555 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:32.451463 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/444d54fd-04f0-4bca-9cb0-ec5cce618593-maas-api-tls" pod "444d54fd-04f0-4bca-9cb0-ec5cce618593" (UID: "444d54fd-04f0-4bca-9cb0-ec5cce618593"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:46:32.451555 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:32.451496 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/444d54fd-04f0-4bca-9cb0-ec5cce618593-kube-api-access-5whbb" pod "444d54fd-04f0-4bca-9cb0-ec5cce618593" (UID: "444d54fd-04f0-4bca-9cb0-ec5cce618593"). InnerVolumeSpecName "kube-api-access-5whbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:32.550446 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:32.550369 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5whbb\" (UniqueName: \"kubernetes.io/projected/444d54fd-04f0-4bca-9cb0-ec5cce618593-kube-api-access-5whbb\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:32.550446 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:32.550398 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/444d54fd-04f0-4bca-9cb0-ec5cce618593-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:32.550446 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:32.550408 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/444d54fd-04f0-4bca-9cb0-ec5cce618593-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:32.550446 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:32.550417 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/444d54fd-04f0-4bca-9cb0-ec5cce618593-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:32.579410 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:32.579361 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:32.848484 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:32.848401 6841 generic.go:356] "Generic (PLEG): container finished" podID="444d54fd-04f0-4bca-9cb0-ec5cce618593" containerID="4fc78dcd62305ffc52201e24ffba76dcb821175fcc28fa65ef79dac33573aabe" exitCode=0 Oct 31 13:46:32.848484 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:32.848450 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc" Oct 31 13:46:32.848766 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:32.848492 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc" event={"ID":"444d54fd-04f0-4bca-9cb0-ec5cce618593","Type":"ContainerDied","Data":"4fc78dcd62305ffc52201e24ffba76dcb821175fcc28fa65ef79dac33573aabe"} Oct 31 13:46:32.848766 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:32.848549 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc" event={"ID":"444d54fd-04f0-4bca-9cb0-ec5cce618593","Type":"ContainerDied","Data":"5113e9ff2b3dc481ca6719eeb5da9c56b9be2407aeb50ca49e3b7c3b4f318525"} Oct 31 13:46:32.848766 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:32.848583 6841 scope.go:122] "RemoveContainer" containerID="4fc78dcd62305ffc52201e24ffba76dcb821175fcc28fa65ef79dac33573aabe" Oct 31 13:46:32.861084 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:32.861067 6841 scope.go:122] "RemoveContainer" containerID="4fc78dcd62305ffc52201e24ffba76dcb821175fcc28fa65ef79dac33573aabe" Oct 31 13:46:32.861378 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:46:32.861360 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fc78dcd62305ffc52201e24ffba76dcb821175fcc28fa65ef79dac33573aabe\": container with ID starting with 4fc78dcd62305ffc52201e24ffba76dcb821175fcc28fa65ef79dac33573aabe not found: ID does not exist" containerID="4fc78dcd62305ffc52201e24ffba76dcb821175fcc28fa65ef79dac33573aabe" Oct 31 13:46:32.861450 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:32.861387 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fc78dcd62305ffc52201e24ffba76dcb821175fcc28fa65ef79dac33573aabe"} err="failed to get container status \"4fc78dcd62305ffc52201e24ffba76dcb821175fcc28fa65ef79dac33573aabe\": rpc error: code = NotFound desc = could not find container \"4fc78dcd62305ffc52201e24ffba76dcb821175fcc28fa65ef79dac33573aabe\": container with ID starting with 4fc78dcd62305ffc52201e24ffba76dcb821175fcc28fa65ef79dac33573aabe not found: ID does not exist" Oct 31 13:46:32.875185 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:32.875159 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc"] Oct 31 13:46:32.882465 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:32.882446 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1e72f6-7b5d975694-t7qkc"] Oct 31 13:46:33.207213 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:33.207180 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:33.642383 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:33.642307 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="444d54fd-04f0-4bca-9cb0-ec5cce618593" path="/var/lib/kubelet/pods/444d54fd-04f0-4bca-9cb0-ec5cce618593/volumes" Oct 31 13:46:33.933096 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:33.933018 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:34.663949 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:34.663910 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:35.625121 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:35.625080 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:36.390204 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:36.390166 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:37.148246 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:37.148204 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:37.868345 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:37.868314 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:38.625246 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:38.625209 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:39.434538 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:39.434505 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:39.650866 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:39.650824 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:40.006321 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:40.006289 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:40.187531 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:40.187497 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp"] Oct 31 13:46:40.187743 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:40.187717 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" podUID="01218eaf-82da-49c5-9195-1f8af3775573" containerName="istio-proxy" containerID="cri-o://ced144ffcd84f45b4052bc3e30650879dcc8dec4cfe6822d8b93a8175d529a48" gracePeriod=30 Oct 31 13:46:40.339358 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:40.339252 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:40.551835 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:40.551796 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:40.822647 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:40.822617 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:41.172189 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.172106 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79aa7668-bl9km"] Oct 31 13:46:41.172578 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.172565 6841 cpu_manager.go:414] "containerMap: removing container" podUID="fbae99f9-22a1-49d4-a08d-3bc7a0f69bde" containerName="istio-proxy" Oct 31 13:46:41.172635 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.172579 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbae99f9-22a1-49d4-a08d-3bc7a0f69bde" containerName="istio-proxy" Oct 31 13:46:41.172635 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.172590 6841 cpu_manager.go:414] "containerMap: removing container" podUID="bdaa9d86-a826-4768-ae0d-471ce6e65e15" containerName="istio-proxy" Oct 31 13:46:41.172635 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.172595 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdaa9d86-a826-4768-ae0d-471ce6e65e15" containerName="istio-proxy" Oct 31 13:46:41.172635 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.172613 6841 cpu_manager.go:414] "containerMap: removing container" podUID="375ff5f6-486e-434d-bf92-7b173e9f2eb3" containerName="istio-proxy" Oct 31 13:46:41.172635 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.172618 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="375ff5f6-486e-434d-bf92-7b173e9f2eb3" containerName="istio-proxy" Oct 31 13:46:41.172635 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.172625 6841 cpu_manager.go:414] "containerMap: removing container" podUID="1997f236-6ad0-4054-8193-ca7b9477fafb" containerName="istio-proxy" Oct 31 13:46:41.172635 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.172630 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1997f236-6ad0-4054-8193-ca7b9477fafb" containerName="istio-proxy" Oct 31 13:46:41.172857 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.172641 6841 cpu_manager.go:414] "containerMap: removing container" podUID="444d54fd-04f0-4bca-9cb0-ec5cce618593" containerName="maas-api" Oct 31 13:46:41.172857 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.172646 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="444d54fd-04f0-4bca-9cb0-ec5cce618593" containerName="maas-api" Oct 31 13:46:41.172857 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.172658 6841 cpu_manager.go:414] "containerMap: removing container" podUID="fbae99f9-22a1-49d4-a08d-3bc7a0f69bde" containerName="istio-proxy" Oct 31 13:46:41.172857 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.172662 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbae99f9-22a1-49d4-a08d-3bc7a0f69bde" containerName="istio-proxy" Oct 31 13:46:41.172857 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.172671 6841 cpu_manager.go:414] "containerMap: removing container" podUID="1dc69d96-bfd1-4ec9-8414-75553fde6d87" containerName="main" Oct 31 13:46:41.172857 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.172676 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dc69d96-bfd1-4ec9-8414-75553fde6d87" containerName="main" Oct 31 13:46:41.172857 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.172682 6841 cpu_manager.go:414] "containerMap: removing container" podUID="909de922-16af-4ee1-b6cf-375ab5fb6053" containerName="maas-api" Oct 31 13:46:41.172857 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.172687 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="909de922-16af-4ee1-b6cf-375ab5fb6053" containerName="maas-api" Oct 31 13:46:41.172857 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.172693 6841 cpu_manager.go:414] "containerMap: removing container" podUID="bdaa9d86-a826-4768-ae0d-471ce6e65e15" containerName="istio-proxy" Oct 31 13:46:41.172857 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.172697 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdaa9d86-a826-4768-ae0d-471ce6e65e15" containerName="istio-proxy" Oct 31 13:46:41.172857 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.172711 6841 cpu_manager.go:414] "containerMap: removing container" podUID="19ae4518-f78f-431b-a5a0-031805427fc8" containerName="maas-api" Oct 31 13:46:41.172857 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.172715 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19ae4518-f78f-431b-a5a0-031805427fc8" containerName="maas-api" Oct 31 13:46:41.172857 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.172769 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="19ae4518-f78f-431b-a5a0-031805427fc8" containerName="maas-api" Oct 31 13:46:41.172857 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.172779 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbae99f9-22a1-49d4-a08d-3bc7a0f69bde" containerName="istio-proxy" Oct 31 13:46:41.172857 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.172784 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdaa9d86-a826-4768-ae0d-471ce6e65e15" containerName="istio-proxy" Oct 31 13:46:41.172857 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.172792 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="375ff5f6-486e-434d-bf92-7b173e9f2eb3" containerName="istio-proxy" Oct 31 13:46:41.172857 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.172799 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="1997f236-6ad0-4054-8193-ca7b9477fafb" containerName="istio-proxy" Oct 31 13:46:41.172857 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.172806 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="444d54fd-04f0-4bca-9cb0-ec5cce618593" containerName="maas-api" Oct 31 13:46:41.172857 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.172814 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dc69d96-bfd1-4ec9-8414-75553fde6d87" containerName="main" Oct 31 13:46:41.172857 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.172819 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="909de922-16af-4ee1-b6cf-375ab5fb6053" containerName="maas-api" Oct 31 13:46:41.177663 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.177646 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79aa7668-bl9km" Oct 31 13:46:41.186240 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.186212 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79aa7668-bl9km"] Oct 31 13:46:41.188698 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.188674 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:46:41.188821 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.188786 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:46:41.188821 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.188793 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:46:41.191775 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.191751 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:46:41.225079 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.225047 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:41.337076 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.337036 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e701b9b8-f494-4a8f-ae55-87dda0640089-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79aa7668-bl9km\" (UID: \"e701b9b8-f494-4a8f-ae55-87dda0640089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79aa7668-bl9km" Oct 31 13:46:41.337261 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.337097 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s59dd\" (UniqueName: \"kubernetes.io/projected/e701b9b8-f494-4a8f-ae55-87dda0640089-kube-api-access-s59dd\") pod \"maas-api-revoke-sub-79aa7668-bl9km\" (UID: \"e701b9b8-f494-4a8f-ae55-87dda0640089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79aa7668-bl9km" Oct 31 13:46:41.337261 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.337194 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e701b9b8-f494-4a8f-ae55-87dda0640089-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79aa7668-bl9km\" (UID: \"e701b9b8-f494-4a8f-ae55-87dda0640089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79aa7668-bl9km" Oct 31 13:46:41.437767 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.437692 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s59dd\" (UniqueName: \"kubernetes.io/projected/e701b9b8-f494-4a8f-ae55-87dda0640089-kube-api-access-s59dd\") pod \"maas-api-revoke-sub-79aa7668-bl9km\" (UID: \"e701b9b8-f494-4a8f-ae55-87dda0640089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79aa7668-bl9km" Oct 31 13:46:41.437767 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.437737 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e701b9b8-f494-4a8f-ae55-87dda0640089-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79aa7668-bl9km\" (UID: \"e701b9b8-f494-4a8f-ae55-87dda0640089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79aa7668-bl9km" Oct 31 13:46:41.437952 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.437822 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e701b9b8-f494-4a8f-ae55-87dda0640089-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79aa7668-bl9km\" (UID: \"e701b9b8-f494-4a8f-ae55-87dda0640089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79aa7668-bl9km" Oct 31 13:46:41.438573 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.438546 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e701b9b8-f494-4a8f-ae55-87dda0640089-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79aa7668-bl9km\" (UID: \"e701b9b8-f494-4a8f-ae55-87dda0640089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79aa7668-bl9km" Oct 31 13:46:41.440339 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.440315 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e701b9b8-f494-4a8f-ae55-87dda0640089-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79aa7668-bl9km\" (UID: \"e701b9b8-f494-4a8f-ae55-87dda0640089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79aa7668-bl9km" Oct 31 13:46:41.450908 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.450883 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s59dd\" (UniqueName: \"kubernetes.io/projected/e701b9b8-f494-4a8f-ae55-87dda0640089-kube-api-access-s59dd\") pod \"maas-api-revoke-sub-79aa7668-bl9km\" (UID: \"e701b9b8-f494-4a8f-ae55-87dda0640089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79aa7668-bl9km" Oct 31 13:46:41.490729 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.490688 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:41.493503 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.493480 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79aa7668-bl9km" Oct 31 13:46:41.773610 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.773579 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79aa7668-bl9km"] Oct 31 13:46:41.775356 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:46:41.775328 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode701b9b8_f494_4a8f_ae55_87dda0640089.slice/crio-ae8fc18f5d4e42ecaad5312c95de29f659d3218c07bba97b3c36bc0a99b8474d WatchSource:0}: Error finding container ae8fc18f5d4e42ecaad5312c95de29f659d3218c07bba97b3c36bc0a99b8474d: Status 404 returned error can't find the container with id ae8fc18f5d4e42ecaad5312c95de29f659d3218c07bba97b3c36bc0a99b8474d Oct 31 13:46:41.905933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:41.905904 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79aa7668-bl9km" event={"ID":"e701b9b8-f494-4a8f-ae55-87dda0640089","Type":"ContainerStarted","Data":"ae8fc18f5d4e42ecaad5312c95de29f659d3218c07bba97b3c36bc0a99b8474d"} Oct 31 13:46:42.134388 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:42.134342 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:42.349442 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:42.349363 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:42.913438 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:42.913354 6841 generic.go:356] "Generic (PLEG): container finished" podID="e701b9b8-f494-4a8f-ae55-87dda0640089" containerID="0927e7fad0afff766a5b5d0d531b8a68b17e6c944a71a9b94c2d5b1248ed5f33" exitCode=0 Oct 31 13:46:42.913812 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:42.913436 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79aa7668-bl9km" event={"ID":"e701b9b8-f494-4a8f-ae55-87dda0640089","Type":"ContainerDied","Data":"0927e7fad0afff766a5b5d0d531b8a68b17e6c944a71a9b94c2d5b1248ed5f33"} Oct 31 13:46:43.170489 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:43.170404 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:43.699998 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:43.699944 6841 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:46:43.700151 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:43.700027 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" podUID="01218eaf-82da-49c5-9195-1f8af3775573" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:46:43.930774 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:43.930736 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:44.055151 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:44.055126 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79aa7668-bl9km" Oct 31 13:46:44.059365 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:44.059342 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e701b9b8-f494-4a8f-ae55-87dda0640089-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e701b9b8-f494-4a8f-ae55-87dda0640089-maas-api-xks-ca\") pod \"e701b9b8-f494-4a8f-ae55-87dda0640089\" (UID: \"e701b9b8-f494-4a8f-ae55-87dda0640089\") " Oct 31 13:46:44.059475 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:44.059435 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e701b9b8-f494-4a8f-ae55-87dda0640089-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e701b9b8-f494-4a8f-ae55-87dda0640089-maas-api-openshift-ca\") pod \"e701b9b8-f494-4a8f-ae55-87dda0640089\" (UID: \"e701b9b8-f494-4a8f-ae55-87dda0640089\") " Oct 31 13:46:44.059529 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:44.059488 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e701b9b8-f494-4a8f-ae55-87dda0640089-kube-api-access-s59dd\" (UniqueName: \"kubernetes.io/projected/e701b9b8-f494-4a8f-ae55-87dda0640089-kube-api-access-s59dd\") pod \"e701b9b8-f494-4a8f-ae55-87dda0640089\" (UID: \"e701b9b8-f494-4a8f-ae55-87dda0640089\") " Oct 31 13:46:44.059808 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:44.059785 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e701b9b8-f494-4a8f-ae55-87dda0640089-maas-api-openshift-ca" pod "e701b9b8-f494-4a8f-ae55-87dda0640089" (UID: "e701b9b8-f494-4a8f-ae55-87dda0640089"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:44.061302 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:44.061284 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e701b9b8-f494-4a8f-ae55-87dda0640089-maas-api-xks-ca" pod "e701b9b8-f494-4a8f-ae55-87dda0640089" (UID: "e701b9b8-f494-4a8f-ae55-87dda0640089"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:46:44.061438 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:44.061417 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e701b9b8-f494-4a8f-ae55-87dda0640089-kube-api-access-s59dd" pod "e701b9b8-f494-4a8f-ae55-87dda0640089" (UID: "e701b9b8-f494-4a8f-ae55-87dda0640089"). InnerVolumeSpecName "kube-api-access-s59dd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:44.160421 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:44.160387 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s59dd\" (UniqueName: \"kubernetes.io/projected/e701b9b8-f494-4a8f-ae55-87dda0640089-kube-api-access-s59dd\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:44.160421 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:44.160417 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e701b9b8-f494-4a8f-ae55-87dda0640089-maas-api-xks-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:44.160421 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:44.160426 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e701b9b8-f494-4a8f-ae55-87dda0640089-maas-api-openshift-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:44.851275 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:44.851235 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:44.925668 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:44.925642 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79aa7668-bl9km" Oct 31 13:46:44.925848 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:44.925673 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79aa7668-bl9km" event={"ID":"e701b9b8-f494-4a8f-ae55-87dda0640089","Type":"ContainerDied","Data":"ae8fc18f5d4e42ecaad5312c95de29f659d3218c07bba97b3c36bc0a99b8474d"} Oct 31 13:46:44.925848 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:44.925723 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae8fc18f5d4e42ecaad5312c95de29f659d3218c07bba97b3c36bc0a99b8474d" Oct 31 13:46:45.444467 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.444442 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:46:45.473231 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.473152 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01218eaf-82da-49c5-9195-1f8af3775573-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01218eaf-82da-49c5-9195-1f8af3775573-istiod-ca-cert\") pod \"01218eaf-82da-49c5-9195-1f8af3775573\" (UID: \"01218eaf-82da-49c5-9195-1f8af3775573\") " Oct 31 13:46:45.473231 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.473203 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01218eaf-82da-49c5-9195-1f8af3775573-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01218eaf-82da-49c5-9195-1f8af3775573-credential-socket\") pod \"01218eaf-82da-49c5-9195-1f8af3775573\" (UID: \"01218eaf-82da-49c5-9195-1f8af3775573\") " Oct 31 13:46:45.473420 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.473233 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01218eaf-82da-49c5-9195-1f8af3775573-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01218eaf-82da-49c5-9195-1f8af3775573-istio-envoy\") pod \"01218eaf-82da-49c5-9195-1f8af3775573\" (UID: \"01218eaf-82da-49c5-9195-1f8af3775573\") " Oct 31 13:46:45.473420 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.473267 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01218eaf-82da-49c5-9195-1f8af3775573-kube-api-access-q6wxw\" (UniqueName: \"kubernetes.io/projected/01218eaf-82da-49c5-9195-1f8af3775573-kube-api-access-q6wxw\") pod \"01218eaf-82da-49c5-9195-1f8af3775573\" (UID: \"01218eaf-82da-49c5-9195-1f8af3775573\") " Oct 31 13:46:45.473420 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.473296 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01218eaf-82da-49c5-9195-1f8af3775573-istio-token\" (UniqueName: \"kubernetes.io/projected/01218eaf-82da-49c5-9195-1f8af3775573-istio-token\") pod \"01218eaf-82da-49c5-9195-1f8af3775573\" (UID: \"01218eaf-82da-49c5-9195-1f8af3775573\") " Oct 31 13:46:45.473420 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.473332 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01218eaf-82da-49c5-9195-1f8af3775573-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01218eaf-82da-49c5-9195-1f8af3775573-istio-data\") pod \"01218eaf-82da-49c5-9195-1f8af3775573\" (UID: \"01218eaf-82da-49c5-9195-1f8af3775573\") " Oct 31 13:46:45.473420 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.473364 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01218eaf-82da-49c5-9195-1f8af3775573-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01218eaf-82da-49c5-9195-1f8af3775573-workload-certs\") pod \"01218eaf-82da-49c5-9195-1f8af3775573\" (UID: \"01218eaf-82da-49c5-9195-1f8af3775573\") " Oct 31 13:46:45.473420 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.473414 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01218eaf-82da-49c5-9195-1f8af3775573-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01218eaf-82da-49c5-9195-1f8af3775573-workload-socket\") pod \"01218eaf-82da-49c5-9195-1f8af3775573\" (UID: \"01218eaf-82da-49c5-9195-1f8af3775573\") " Oct 31 13:46:45.473641 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.473485 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01218eaf-82da-49c5-9195-1f8af3775573-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01218eaf-82da-49c5-9195-1f8af3775573-istio-podinfo\") pod \"01218eaf-82da-49c5-9195-1f8af3775573\" (UID: \"01218eaf-82da-49c5-9195-1f8af3775573\") " Oct 31 13:46:45.474117 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.474024 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01218eaf-82da-49c5-9195-1f8af3775573-istio-data" pod "01218eaf-82da-49c5-9195-1f8af3775573" (UID: "01218eaf-82da-49c5-9195-1f8af3775573"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:45.474819 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.474681 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01218eaf-82da-49c5-9195-1f8af3775573-istiod-ca-cert" pod "01218eaf-82da-49c5-9195-1f8af3775573" (UID: "01218eaf-82da-49c5-9195-1f8af3775573"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:45.475036 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.474997 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01218eaf-82da-49c5-9195-1f8af3775573-workload-certs" pod "01218eaf-82da-49c5-9195-1f8af3775573" (UID: "01218eaf-82da-49c5-9195-1f8af3775573"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:45.475157 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.475132 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01218eaf-82da-49c5-9195-1f8af3775573-workload-socket" pod "01218eaf-82da-49c5-9195-1f8af3775573" (UID: "01218eaf-82da-49c5-9195-1f8af3775573"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:45.475234 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.475187 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01218eaf-82da-49c5-9195-1f8af3775573-credential-socket" pod "01218eaf-82da-49c5-9195-1f8af3775573" (UID: "01218eaf-82da-49c5-9195-1f8af3775573"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:45.476367 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.476344 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01218eaf-82da-49c5-9195-1f8af3775573-istio-envoy" pod "01218eaf-82da-49c5-9195-1f8af3775573" (UID: "01218eaf-82da-49c5-9195-1f8af3775573"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:45.476530 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.476483 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01218eaf-82da-49c5-9195-1f8af3775573-istio-podinfo" pod "01218eaf-82da-49c5-9195-1f8af3775573" (UID: "01218eaf-82da-49c5-9195-1f8af3775573"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:46:45.487020 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.476946 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01218eaf-82da-49c5-9195-1f8af3775573-kube-api-access-q6wxw" pod "01218eaf-82da-49c5-9195-1f8af3775573" (UID: "01218eaf-82da-49c5-9195-1f8af3775573"). InnerVolumeSpecName "kube-api-access-q6wxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:45.487020 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.477201 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01218eaf-82da-49c5-9195-1f8af3775573-istio-token" pod "01218eaf-82da-49c5-9195-1f8af3775573" (UID: "01218eaf-82da-49c5-9195-1f8af3775573"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:45.575098 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.575066 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01218eaf-82da-49c5-9195-1f8af3775573-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:45.575098 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.575092 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01218eaf-82da-49c5-9195-1f8af3775573-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:45.575098 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.575102 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01218eaf-82da-49c5-9195-1f8af3775573-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:45.575334 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.575112 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01218eaf-82da-49c5-9195-1f8af3775573-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:45.575334 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.575122 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01218eaf-82da-49c5-9195-1f8af3775573-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:45.575334 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.575131 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01218eaf-82da-49c5-9195-1f8af3775573-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:45.575334 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.575141 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01218eaf-82da-49c5-9195-1f8af3775573-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:45.575334 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.575151 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6wxw\" (UniqueName: \"kubernetes.io/projected/01218eaf-82da-49c5-9195-1f8af3775573-kube-api-access-q6wxw\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:45.575334 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.575160 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01218eaf-82da-49c5-9195-1f8af3775573-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:45.711230 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.711179 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:45.925796 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.925703 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:45.932366 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.932337 6841 generic.go:356] "Generic (PLEG): container finished" podID="01218eaf-82da-49c5-9195-1f8af3775573" containerID="ced144ffcd84f45b4052bc3e30650879dcc8dec4cfe6822d8b93a8175d529a48" exitCode=0 Oct 31 13:46:45.932557 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.932392 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" Oct 31 13:46:45.932557 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.932424 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" event={"ID":"01218eaf-82da-49c5-9195-1f8af3775573","Type":"ContainerDied","Data":"ced144ffcd84f45b4052bc3e30650879dcc8dec4cfe6822d8b93a8175d529a48"} Oct 31 13:46:45.932557 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.932497 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp" event={"ID":"01218eaf-82da-49c5-9195-1f8af3775573","Type":"ContainerDied","Data":"32e87c434306caefa99837e8ec6fd7e92c0f610a1bc2552b97de56bee7c5eca0"} Oct 31 13:46:45.932557 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.932529 6841 scope.go:122] "RemoveContainer" containerID="ced144ffcd84f45b4052bc3e30650879dcc8dec4cfe6822d8b93a8175d529a48" Oct 31 13:46:45.942706 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.942689 6841 scope.go:122] "RemoveContainer" containerID="1f3a721c728f679db20ccf78ded720a693a5e9a06bbb4ccfb1e0865506375925" Oct 31 13:46:45.952066 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.952046 6841 scope.go:122] "RemoveContainer" containerID="ced144ffcd84f45b4052bc3e30650879dcc8dec4cfe6822d8b93a8175d529a48" Oct 31 13:46:45.952367 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:46:45.952347 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ced144ffcd84f45b4052bc3e30650879dcc8dec4cfe6822d8b93a8175d529a48\": container with ID starting with ced144ffcd84f45b4052bc3e30650879dcc8dec4cfe6822d8b93a8175d529a48 not found: ID does not exist" containerID="ced144ffcd84f45b4052bc3e30650879dcc8dec4cfe6822d8b93a8175d529a48" Oct 31 13:46:45.952451 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.952379 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ced144ffcd84f45b4052bc3e30650879dcc8dec4cfe6822d8b93a8175d529a48"} err="failed to get container status \"ced144ffcd84f45b4052bc3e30650879dcc8dec4cfe6822d8b93a8175d529a48\": rpc error: code = NotFound desc = could not find container \"ced144ffcd84f45b4052bc3e30650879dcc8dec4cfe6822d8b93a8175d529a48\": container with ID starting with ced144ffcd84f45b4052bc3e30650879dcc8dec4cfe6822d8b93a8175d529a48 not found: ID does not exist" Oct 31 13:46:45.952451 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.952401 6841 scope.go:122] "RemoveContainer" containerID="1f3a721c728f679db20ccf78ded720a693a5e9a06bbb4ccfb1e0865506375925" Oct 31 13:46:45.952697 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:46:45.952664 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f3a721c728f679db20ccf78ded720a693a5e9a06bbb4ccfb1e0865506375925\": container with ID starting with 1f3a721c728f679db20ccf78ded720a693a5e9a06bbb4ccfb1e0865506375925 not found: ID does not exist" containerID="1f3a721c728f679db20ccf78ded720a693a5e9a06bbb4ccfb1e0865506375925" Oct 31 13:46:45.952750 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.952707 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f3a721c728f679db20ccf78ded720a693a5e9a06bbb4ccfb1e0865506375925"} err="failed to get container status \"1f3a721c728f679db20ccf78ded720a693a5e9a06bbb4ccfb1e0865506375925\": rpc error: code = NotFound desc = could not find container \"1f3a721c728f679db20ccf78ded720a693a5e9a06bbb4ccfb1e0865506375925\": container with ID starting with 1f3a721c728f679db20ccf78ded720a693a5e9a06bbb4ccfb1e0865506375925 not found: ID does not exist" Oct 31 13:46:45.960058 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.960035 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp"] Oct 31 13:46:45.964014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:45.963969 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-zxrjp"] Oct 31 13:46:46.402533 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:46.402502 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:46.605061 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:46.604977 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:47.342592 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:47.342545 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:47.642836 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:47.642746 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01218eaf-82da-49c5-9195-1f8af3775573" path="/var/lib/kubelet/pods/01218eaf-82da-49c5-9195-1f8af3775573/volumes" Oct 31 13:46:48.112064 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:48.112023 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:48.803430 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:48.803321 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:49.044731 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:49.044694 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:49.701463 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:49.701429 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:50.034434 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:46:50.034357 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:46:50.448005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:50.447949 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:51.388806 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:51.388770 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:52.133541 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:52.133507 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:52.734677 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:52.734643 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3efb793-45fcq"] Oct 31 13:46:52.735084 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:52.735069 6841 cpu_manager.go:414] "containerMap: removing container" podUID="01218eaf-82da-49c5-9195-1f8af3775573" containerName="istio-proxy" Oct 31 13:46:52.735084 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:52.735081 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01218eaf-82da-49c5-9195-1f8af3775573" containerName="istio-proxy" Oct 31 13:46:52.735162 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:52.735092 6841 cpu_manager.go:414] "containerMap: removing container" podUID="e701b9b8-f494-4a8f-ae55-87dda0640089" containerName="revoke-keys" Oct 31 13:46:52.735162 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:52.735097 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e701b9b8-f494-4a8f-ae55-87dda0640089" containerName="revoke-keys" Oct 31 13:46:52.735162 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:52.735129 6841 cpu_manager.go:414] "containerMap: removing container" podUID="01218eaf-82da-49c5-9195-1f8af3775573" containerName="istio-proxy" Oct 31 13:46:52.735162 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:52.735134 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01218eaf-82da-49c5-9195-1f8af3775573" containerName="istio-proxy" Oct 31 13:46:52.735293 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:52.735201 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="01218eaf-82da-49c5-9195-1f8af3775573" containerName="istio-proxy" Oct 31 13:46:52.735293 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:52.735211 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="e701b9b8-f494-4a8f-ae55-87dda0640089" containerName="revoke-keys" Oct 31 13:46:52.735293 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:52.735220 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbae99f9-22a1-49d4-a08d-3bc7a0f69bde" containerName="istio-proxy" Oct 31 13:46:52.735293 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:52.735225 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdaa9d86-a826-4768-ae0d-471ce6e65e15" containerName="istio-proxy" Oct 31 13:46:52.739966 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:52.739944 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3efb793-45fcq" Oct 31 13:46:52.748916 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:52.748878 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:46:52.749041 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:52.748923 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:46:52.749041 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:52.748957 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:46:52.749041 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:52.748963 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:46:52.759466 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:52.759439 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3efb793-45fcq"] Oct 31 13:46:52.845589 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:52.845561 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3efb793-45fcq\" (UID: \"d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3efb793-45fcq" Oct 31 13:46:52.845781 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:52.845603 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3efb793-45fcq\" (UID: \"d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3efb793-45fcq" Oct 31 13:46:52.845781 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:52.845767 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvcdw\" (UniqueName: \"kubernetes.io/projected/d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9-kube-api-access-gvcdw\") pod \"maas-api-revoke-sub-c3efb793-45fcq\" (UID: \"d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3efb793-45fcq" Oct 31 13:46:52.922280 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:52.922239 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7ae9e7-57b99fbcdc-rqs6p"] Oct 31 13:46:52.940451 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:52.940419 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7ae9e7-65f7d7db79-bsvx2"] Oct 31 13:46:52.947512 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:52.947476 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvcdw\" (UniqueName: \"kubernetes.io/projected/d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9-kube-api-access-gvcdw\") pod \"maas-api-revoke-sub-c3efb793-45fcq\" (UID: \"d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3efb793-45fcq" Oct 31 13:46:52.947692 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:52.947582 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3efb793-45fcq\" (UID: \"d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3efb793-45fcq" Oct 31 13:46:52.947692 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:52.947631 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3efb793-45fcq\" (UID: \"d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3efb793-45fcq" Oct 31 13:46:52.949083 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:52.948542 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3efb793-45fcq\" (UID: \"d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3efb793-45fcq" Oct 31 13:46:52.951751 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:52.951712 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3efb793-45fcq\" (UID: \"d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3efb793-45fcq" Oct 31 13:46:52.970657 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:52.970631 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:52.972057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:52.972027 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvcdw\" (UniqueName: \"kubernetes.io/projected/d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9-kube-api-access-gvcdw\") pod \"maas-api-revoke-sub-c3efb793-45fcq\" (UID: \"d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3efb793-45fcq" Oct 31 13:46:53.052969 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:53.052872 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3efb793-45fcq" Oct 31 13:46:53.176318 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:53.176280 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:53.333414 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:53.333392 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3efb793-45fcq"] Oct 31 13:46:53.335658 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:46:53.335629 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5bce8ca_9936_4e67_a8cd_98c9e91b9bd9.slice/crio-9351a192ef5765196a854db1f10908c23aad70436031e1f6cdb3e6589836da33 WatchSource:0}: Error finding container 9351a192ef5765196a854db1f10908c23aad70436031e1f6cdb3e6589836da33: Status 404 returned error can't find the container with id 9351a192ef5765196a854db1f10908c23aad70436031e1f6cdb3e6589836da33 Oct 31 13:46:53.931288 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:53.931192 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:46:53.977386 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:53.977349 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3efb793-45fcq" event={"ID":"d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9","Type":"ContainerStarted","Data":"45c03f04514571518ded9f50f0e079892c8b684a54916580d613b67f92f8f82c"} Oct 31 13:46:53.977573 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:53.977398 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3efb793-45fcq" event={"ID":"d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9","Type":"ContainerStarted","Data":"9351a192ef5765196a854db1f10908c23aad70436031e1f6cdb3e6589836da33"} Oct 31 13:46:53.997158 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:53.997112 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3efb793-45fcq" podStartSLOduration=1.997098102 podStartE2EDuration="1.997098102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:46:53.99553202 +0000 UTC m=+4460.478654923" watchObservedRunningTime="2026-10-31 13:46:53.997098102 +0000 UTC m=+4460.480221003" Oct 31 13:46:54.984593 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:54.984557 6841 generic.go:356] "Generic (PLEG): container finished" podID="d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9" containerID="45c03f04514571518ded9f50f0e079892c8b684a54916580d613b67f92f8f82c" exitCode=0 Oct 31 13:46:54.985058 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:54.984644 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3efb793-45fcq" event={"ID":"d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9","Type":"ContainerDied","Data":"45c03f04514571518ded9f50f0e079892c8b684a54916580d613b67f92f8f82c"} Oct 31 13:46:56.124890 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:56.124866 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3efb793-45fcq" Oct 31 13:46:56.176723 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:56.176691 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9-kube-api-access-gvcdw\" (UniqueName: \"kubernetes.io/projected/d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9-kube-api-access-gvcdw\") pod \"d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9\" (UID: \"d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9\") " Oct 31 13:46:56.176723 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:56.176731 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9-maas-api-openshift-ca\") pod \"d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9\" (UID: \"d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9\") " Oct 31 13:46:56.176951 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:56.176794 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9-maas-api-xks-ca\") pod \"d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9\" (UID: \"d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9\") " Oct 31 13:46:56.177208 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:56.177187 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9-maas-api-openshift-ca" pod "d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9" (UID: "d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:56.178949 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:56.178918 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9-maas-api-xks-ca" pod "d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9" (UID: "d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:46:56.179146 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:56.179069 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9-kube-api-access-gvcdw" pod "d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9" (UID: "d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9"). InnerVolumeSpecName "kube-api-access-gvcdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:56.277943 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:56.277866 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvcdw\" (UniqueName: \"kubernetes.io/projected/d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9-kube-api-access-gvcdw\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:56.277943 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:56.277897 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9-maas-api-openshift-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:56.277943 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:56.277908 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9-maas-api-xks-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:56.996776 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:56.996746 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3efb793-45fcq" Oct 31 13:46:56.996776 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:56.996761 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3efb793-45fcq" event={"ID":"d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9","Type":"ContainerDied","Data":"9351a192ef5765196a854db1f10908c23aad70436031e1f6cdb3e6589836da33"} Oct 31 13:46:56.997053 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:56.996803 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9351a192ef5765196a854db1f10908c23aad70436031e1f6cdb3e6589836da33" Oct 31 13:46:57.923383 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:57.923341 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ae9e7-57b99fbcdc-rqs6p" podUID="f721bc13-8ce6-4d6b-b53a-65b9c0030e08" containerName="payload-processing" containerID="cri-o://a9e2f283a5b7e8d250af5e5eb65afe802b9cbda17666c5c1309c0cf2ba635c30" gracePeriod=25 Oct 31 13:46:57.941413 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:57.941384 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ae9e7-65f7d7db79-bsvx2" podUID="716df414-c063-4bf8-b85c-fcb3c319fb70" containerName="payload-pre-processing" containerID="cri-o://0effe3ab060992c2a9fe684dea2c2681dde7e8cada2b4b031c692d8d1be60392" gracePeriod=25 Oct 31 13:46:59.782600 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:59.782556 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ae9e7-57b99fbcdc-rqs6p" podUID="f721bc13-8ce6-4d6b-b53a-65b9c0030e08" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:46:59.782600 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:59.782573 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ae9e7-65f7d7db79-bsvx2" podUID="716df414-c063-4bf8-b85c-fcb3c319fb70" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:46:59.784291 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:46:59.784271 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:03.776203 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:03.776171 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:05.848113 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:05.848069 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:06.053848 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:06.053806 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:09.782391 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:09.782346 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ae9e7-57b99fbcdc-rqs6p" podUID="f721bc13-8ce6-4d6b-b53a-65b9c0030e08" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:47:09.782391 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:09.782362 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ae9e7-65f7d7db79-bsvx2" podUID="716df414-c063-4bf8-b85c-fcb3c319fb70" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:47:11.015093 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:11.015057 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:11.227081 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:11.227042 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:12.465584 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.465549 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c"] Oct 31 13:47:12.465977 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.465966 6841 cpu_manager.go:414] "containerMap: removing container" podUID="d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9" containerName="revoke-keys" Oct 31 13:47:12.465977 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.465978 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9" containerName="revoke-keys" Oct 31 13:47:12.466102 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.466091 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9" containerName="revoke-keys" Oct 31 13:47:12.466144 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.466105 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="01218eaf-82da-49c5-9195-1f8af3775573" containerName="istio-proxy" Oct 31 13:47:12.469390 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.469369 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" Oct 31 13:47:12.480167 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.480143 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-7ae9e7-openshift-default-dockercfg-pplr5\"" Oct 31 13:47:12.485565 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.485540 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c"] Oct 31 13:47:12.639600 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.639569 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffbd5e7d-bbde-426b-a47f-687d886a8b44-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c\" (UID: \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" Oct 31 13:47:12.639783 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.639611 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffbd5e7d-bbde-426b-a47f-687d886a8b44-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c\" (UID: \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" Oct 31 13:47:12.639783 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.639636 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbd5e7d-bbde-426b-a47f-687d886a8b44-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c\" (UID: \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" Oct 31 13:47:12.639783 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.639740 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbd5e7d-bbde-426b-a47f-687d886a8b44-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c\" (UID: \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" Oct 31 13:47:12.639904 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.639782 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffbd5e7d-bbde-426b-a47f-687d886a8b44-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c\" (UID: \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" Oct 31 13:47:12.639904 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.639863 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpf7s\" (UniqueName: \"kubernetes.io/projected/ffbd5e7d-bbde-426b-a47f-687d886a8b44-kube-api-access-jpf7s\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c\" (UID: \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" Oct 31 13:47:12.639970 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.639919 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffbd5e7d-bbde-426b-a47f-687d886a8b44-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c\" (UID: \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" Oct 31 13:47:12.640053 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.640014 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffbd5e7d-bbde-426b-a47f-687d886a8b44-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c\" (UID: \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" Oct 31 13:47:12.640053 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.640039 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffbd5e7d-bbde-426b-a47f-687d886a8b44-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c\" (UID: \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" Oct 31 13:47:12.741390 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.741307 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffbd5e7d-bbde-426b-a47f-687d886a8b44-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c\" (UID: \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" Oct 31 13:47:12.741390 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.741353 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbd5e7d-bbde-426b-a47f-687d886a8b44-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c\" (UID: \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" Oct 31 13:47:12.741624 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.741468 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbd5e7d-bbde-426b-a47f-687d886a8b44-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c\" (UID: \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" Oct 31 13:47:12.741624 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.741512 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffbd5e7d-bbde-426b-a47f-687d886a8b44-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c\" (UID: \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" Oct 31 13:47:12.741624 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.741548 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpf7s\" (UniqueName: \"kubernetes.io/projected/ffbd5e7d-bbde-426b-a47f-687d886a8b44-kube-api-access-jpf7s\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c\" (UID: \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" Oct 31 13:47:12.741624 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.741613 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffbd5e7d-bbde-426b-a47f-687d886a8b44-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c\" (UID: \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" Oct 31 13:47:12.741836 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.741649 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffbd5e7d-bbde-426b-a47f-687d886a8b44-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c\" (UID: \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" Oct 31 13:47:12.741836 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.741684 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffbd5e7d-bbde-426b-a47f-687d886a8b44-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c\" (UID: \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" Oct 31 13:47:12.741836 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.741743 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffbd5e7d-bbde-426b-a47f-687d886a8b44-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c\" (UID: \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" Oct 31 13:47:12.741836 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.741757 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffbd5e7d-bbde-426b-a47f-687d886a8b44-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c\" (UID: \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" Oct 31 13:47:12.742055 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.741884 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbd5e7d-bbde-426b-a47f-687d886a8b44-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c\" (UID: \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" Oct 31 13:47:12.742055 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.741899 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbd5e7d-bbde-426b-a47f-687d886a8b44-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c\" (UID: \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" Oct 31 13:47:12.742146 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.742114 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffbd5e7d-bbde-426b-a47f-687d886a8b44-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c\" (UID: \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" Oct 31 13:47:12.742406 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.742386 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffbd5e7d-bbde-426b-a47f-687d886a8b44-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c\" (UID: \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" Oct 31 13:47:12.743936 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.743911 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffbd5e7d-bbde-426b-a47f-687d886a8b44-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c\" (UID: \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" Oct 31 13:47:12.744069 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.743999 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffbd5e7d-bbde-426b-a47f-687d886a8b44-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c\" (UID: \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" Oct 31 13:47:12.754284 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.754254 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffbd5e7d-bbde-426b-a47f-687d886a8b44-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c\" (UID: \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" Oct 31 13:47:12.754388 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.754324 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpf7s\" (UniqueName: \"kubernetes.io/projected/ffbd5e7d-bbde-426b-a47f-687d886a8b44-kube-api-access-jpf7s\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c\" (UID: \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" Oct 31 13:47:12.782280 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.782254 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" Oct 31 13:47:12.936282 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:12.936244 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:13.063446 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:13.063418 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c"] Oct 31 13:47:13.065128 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:47:13.065100 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffbd5e7d_bbde_426b_a47f_687d886a8b44.slice/crio-f3c84e8162d1046028e6a75e6df68a8b09c2c0d0f6d1e00ab113ce951cfe34d2 WatchSource:0}: Error finding container f3c84e8162d1046028e6a75e6df68a8b09c2c0d0f6d1e00ab113ce951cfe34d2: Status 404 returned error can't find the container with id f3c84e8162d1046028e6a75e6df68a8b09c2c0d0f6d1e00ab113ce951cfe34d2 Oct 31 13:47:13.067227 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:13.067193 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:47:13.067308 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:13.067257 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:47:13.067308 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:13.067287 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:47:13.085946 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:13.085920 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" event={"ID":"ffbd5e7d-bbde-426b-a47f-687d886a8b44","Type":"ContainerStarted","Data":"f3c84e8162d1046028e6a75e6df68a8b09c2c0d0f6d1e00ab113ce951cfe34d2"} Oct 31 13:47:13.254717 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:13.254684 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c"] Oct 31 13:47:13.327347 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:13.327248 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:13.634610 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:13.634517 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:14.092802 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:14.092761 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" event={"ID":"ffbd5e7d-bbde-426b-a47f-687d886a8b44","Type":"ContainerStarted","Data":"d9c8f94eecbefd1d725551fac56a9b633cbed4d1d02ef92e9098a9c508335aee"} Oct 31 13:47:14.092967 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:14.092948 6841 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" secret="" err="secret \"e2e-worker-w4-7ae9e7-openshift-default-dockercfg-pplr5\" not found" Oct 31 13:47:14.119728 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:14.119680 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" podStartSLOduration=2.1196665709999998 podStartE2EDuration="2.119666571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:47:14.118950232 +0000 UTC m=+4480.602073135" watchObservedRunningTime="2026-10-31 13:47:14.119666571 +0000 UTC m=+4480.602789472" Oct 31 13:47:14.639904 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:14.639867 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-88qp9_63f4505c-09a4-4903-b6ba-824d5052bb24/manager/0.log" Oct 31 13:47:14.782563 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:14.782532 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" Oct 31 13:47:14.783888 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:14.783860 6841 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 31 13:47:14.784027 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:14.783917 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" podUID="ffbd5e7d-bbde-426b-a47f-687d886a8b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 31 13:47:15.098180 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:15.098138 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" podUID="ffbd5e7d-bbde-426b-a47f-687d886a8b44" containerName="istio-proxy" containerID="cri-o://d9c8f94eecbefd1d725551fac56a9b633cbed4d1d02ef92e9098a9c508335aee" gracePeriod=30 Oct 31 13:47:15.690389 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:15.690343 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-x7xbq_38fff3bd-4067-4f10-b87f-b9e0fa806e3a/manager/0.log" Oct 31 13:47:16.838257 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:16.838219 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-j72kb_96de151e-d180-45de-8c17-a934e66ddc50/manager/0.log" Oct 31 13:47:17.947628 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:17.947603 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-jm7lb_ecf5ba9f-127b-45de-969f-8128243e0bff/server/0.log" Oct 31 13:47:18.119190 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:18.119159 6841 generic.go:356] "Generic (PLEG): container finished" podID="f721bc13-8ce6-4d6b-b53a-65b9c0030e08" containerID="a9e2f283a5b7e8d250af5e5eb65afe802b9cbda17666c5c1309c0cf2ba635c30" exitCode=0 Oct 31 13:47:18.119400 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:18.119376 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ae9e7-57b99fbcdc-rqs6p" event={"ID":"f721bc13-8ce6-4d6b-b53a-65b9c0030e08","Type":"ContainerDied","Data":"a9e2f283a5b7e8d250af5e5eb65afe802b9cbda17666c5c1309c0cf2ba635c30"} Oct 31 13:47:18.121843 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:18.121817 6841 generic.go:356] "Generic (PLEG): container finished" podID="716df414-c063-4bf8-b85c-fcb3c319fb70" containerID="0effe3ab060992c2a9fe684dea2c2681dde7e8cada2b4b031c692d8d1be60392" exitCode=0 Oct 31 13:47:18.121995 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:18.121878 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ae9e7-65f7d7db79-bsvx2" event={"ID":"716df414-c063-4bf8-b85c-fcb3c319fb70","Type":"ContainerDied","Data":"0effe3ab060992c2a9fe684dea2c2681dde7e8cada2b4b031c692d8d1be60392"} Oct 31 13:47:18.136281 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:18.136244 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-s2ldm_e7c757c0-2ab1-4639-9594-42ab52830a01/manager/0.log" Oct 31 13:47:18.197927 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:18.197893 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ae9e7-57b99fbcdc-rqs6p" Oct 31 13:47:18.238733 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:18.238700 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ae9e7-65f7d7db79-bsvx2" Oct 31 13:47:18.295233 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:18.295204 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/716df414-c063-4bf8-b85c-fcb3c319fb70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/716df414-c063-4bf8-b85c-fcb3c319fb70-plugins-config-volume\") pod \"716df414-c063-4bf8-b85c-fcb3c319fb70\" (UID: \"716df414-c063-4bf8-b85c-fcb3c319fb70\") " Oct 31 13:47:18.295417 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:18.295242 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f721bc13-8ce6-4d6b-b53a-65b9c0030e08-kube-api-access-gw8cc\" (UniqueName: \"kubernetes.io/projected/f721bc13-8ce6-4d6b-b53a-65b9c0030e08-kube-api-access-gw8cc\") pod \"f721bc13-8ce6-4d6b-b53a-65b9c0030e08\" (UID: \"f721bc13-8ce6-4d6b-b53a-65b9c0030e08\") " Oct 31 13:47:18.295417 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:18.295314 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/716df414-c063-4bf8-b85c-fcb3c319fb70-kube-api-access-r6xzz\" (UniqueName: \"kubernetes.io/projected/716df414-c063-4bf8-b85c-fcb3c319fb70-kube-api-access-r6xzz\") pod \"716df414-c063-4bf8-b85c-fcb3c319fb70\" (UID: \"716df414-c063-4bf8-b85c-fcb3c319fb70\") " Oct 31 13:47:18.295576 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:18.295549 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/716df414-c063-4bf8-b85c-fcb3c319fb70-plugins-config-volume" pod "716df414-c063-4bf8-b85c-fcb3c319fb70" (UID: "716df414-c063-4bf8-b85c-fcb3c319fb70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:47:18.297464 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:18.297434 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f721bc13-8ce6-4d6b-b53a-65b9c0030e08-kube-api-access-gw8cc" pod "f721bc13-8ce6-4d6b-b53a-65b9c0030e08" (UID: "f721bc13-8ce6-4d6b-b53a-65b9c0030e08"). InnerVolumeSpecName "kube-api-access-gw8cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:47:18.297464 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:18.297457 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/716df414-c063-4bf8-b85c-fcb3c319fb70-kube-api-access-r6xzz" pod "716df414-c063-4bf8-b85c-fcb3c319fb70" (UID: "716df414-c063-4bf8-b85c-fcb3c319fb70"). InnerVolumeSpecName "kube-api-access-r6xzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:47:18.396254 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:18.396219 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f721bc13-8ce6-4d6b-b53a-65b9c0030e08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f721bc13-8ce6-4d6b-b53a-65b9c0030e08-plugins-config-volume\") pod \"f721bc13-8ce6-4d6b-b53a-65b9c0030e08\" (UID: \"f721bc13-8ce6-4d6b-b53a-65b9c0030e08\") " Oct 31 13:47:18.396463 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:18.396450 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6xzz\" (UniqueName: \"kubernetes.io/projected/716df414-c063-4bf8-b85c-fcb3c319fb70-kube-api-access-r6xzz\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:18.396510 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:18.396467 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/716df414-c063-4bf8-b85c-fcb3c319fb70-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:18.396510 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:18.396478 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw8cc\" (UniqueName: \"kubernetes.io/projected/f721bc13-8ce6-4d6b-b53a-65b9c0030e08-kube-api-access-gw8cc\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:18.396604 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:18.396581 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f721bc13-8ce6-4d6b-b53a-65b9c0030e08-plugins-config-volume" pod "f721bc13-8ce6-4d6b-b53a-65b9c0030e08" (UID: "f721bc13-8ce6-4d6b-b53a-65b9c0030e08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:47:18.496885 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:18.496851 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f721bc13-8ce6-4d6b-b53a-65b9c0030e08-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:19.128242 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:19.128210 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ae9e7-65f7d7db79-bsvx2" Oct 31 13:47:19.128724 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:19.128241 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ae9e7-65f7d7db79-bsvx2" event={"ID":"716df414-c063-4bf8-b85c-fcb3c319fb70","Type":"ContainerDied","Data":"6514dc8ab28c2c35ed146bfff74cea74042ea0a49eed19dde899988479a8958f"} Oct 31 13:47:19.128724 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:19.128316 6841 scope.go:122] "RemoveContainer" containerID="0effe3ab060992c2a9fe684dea2c2681dde7e8cada2b4b031c692d8d1be60392" Oct 31 13:47:19.129768 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:19.129743 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ae9e7-57b99fbcdc-rqs6p" Oct 31 13:47:19.129859 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:19.129787 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ae9e7-57b99fbcdc-rqs6p" event={"ID":"f721bc13-8ce6-4d6b-b53a-65b9c0030e08","Type":"ContainerDied","Data":"7d46bff8eebed35cb1a3897a36367f2aaaeebb50e641fedadad1e0cc2fdb896a"} Oct 31 13:47:19.138964 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:19.138946 6841 scope.go:122] "RemoveContainer" containerID="a9e2f283a5b7e8d250af5e5eb65afe802b9cbda17666c5c1309c0cf2ba635c30" Oct 31 13:47:19.152133 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:19.152106 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7ae9e7-65f7d7db79-bsvx2"] Oct 31 13:47:19.158644 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:19.158621 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7ae9e7-65f7d7db79-bsvx2"] Oct 31 13:47:19.170365 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:19.170336 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7ae9e7-57b99fbcdc-rqs6p"] Oct 31 13:47:19.174238 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:19.174217 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7ae9e7-57b99fbcdc-rqs6p"] Oct 31 13:47:19.479468 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:19.479436 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:19.640394 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:19.640358 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="716df414-c063-4bf8-b85c-fcb3c319fb70" path="/var/lib/kubelet/pods/716df414-c063-4bf8-b85c-fcb3c319fb70/volumes" Oct 31 13:47:19.640852 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:19.640830 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f721bc13-8ce6-4d6b-b53a-65b9c0030e08" path="/var/lib/kubelet/pods/f721bc13-8ce6-4d6b-b53a-65b9c0030e08/volumes" Oct 31 13:47:19.684357 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:19.684328 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:20.033402 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:47:20.033369 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:47:20.343368 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:20.343344 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" Oct 31 13:47:20.416681 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:20.416645 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffbd5e7d-bbde-426b-a47f-687d886a8b44-istio-token\" (UniqueName: \"kubernetes.io/projected/ffbd5e7d-bbde-426b-a47f-687d886a8b44-istio-token\") pod \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\" (UID: \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\") " Oct 31 13:47:20.416869 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:20.416712 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffbd5e7d-bbde-426b-a47f-687d886a8b44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffbd5e7d-bbde-426b-a47f-687d886a8b44-istio-envoy\") pod \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\" (UID: \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\") " Oct 31 13:47:20.416869 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:20.416744 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffbd5e7d-bbde-426b-a47f-687d886a8b44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffbd5e7d-bbde-426b-a47f-687d886a8b44-workload-certs\") pod \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\" (UID: \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\") " Oct 31 13:47:20.416869 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:20.416774 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffbd5e7d-bbde-426b-a47f-687d886a8b44-kube-api-access-jpf7s\" (UniqueName: \"kubernetes.io/projected/ffbd5e7d-bbde-426b-a47f-687d886a8b44-kube-api-access-jpf7s\") pod \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\" (UID: \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\") " Oct 31 13:47:20.416869 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:20.416824 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffbd5e7d-bbde-426b-a47f-687d886a8b44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbd5e7d-bbde-426b-a47f-687d886a8b44-credential-socket\") pod \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\" (UID: \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\") " Oct 31 13:47:20.416869 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:20.416865 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffbd5e7d-bbde-426b-a47f-687d886a8b44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffbd5e7d-bbde-426b-a47f-687d886a8b44-istio-data\") pod \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\" (UID: \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\") " Oct 31 13:47:20.417215 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:20.416919 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffbd5e7d-bbde-426b-a47f-687d886a8b44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbd5e7d-bbde-426b-a47f-687d886a8b44-workload-socket\") pod \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\" (UID: \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\") " Oct 31 13:47:20.417215 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:20.417034 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffbd5e7d-bbde-426b-a47f-687d886a8b44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffbd5e7d-bbde-426b-a47f-687d886a8b44-istiod-ca-cert\") pod \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\" (UID: \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\") " Oct 31 13:47:20.417215 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:20.417065 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ffbd5e7d-bbde-426b-a47f-687d886a8b44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffbd5e7d-bbde-426b-a47f-687d886a8b44-istio-podinfo\") pod \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\" (UID: \"ffbd5e7d-bbde-426b-a47f-687d886a8b44\") " Oct 31 13:47:20.417215 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:20.417165 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffbd5e7d-bbde-426b-a47f-687d886a8b44-workload-certs" pod "ffbd5e7d-bbde-426b-a47f-687d886a8b44" (UID: "ffbd5e7d-bbde-426b-a47f-687d886a8b44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:47:20.417391 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:20.417364 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffbd5e7d-bbde-426b-a47f-687d886a8b44-credential-socket" pod "ffbd5e7d-bbde-426b-a47f-687d886a8b44" (UID: "ffbd5e7d-bbde-426b-a47f-687d886a8b44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:47:20.417564 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:20.417543 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbd5e7d-bbde-426b-a47f-687d886a8b44-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:20.417653 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:20.417569 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffbd5e7d-bbde-426b-a47f-687d886a8b44-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:20.418094 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:20.418067 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffbd5e7d-bbde-426b-a47f-687d886a8b44-istio-data" pod "ffbd5e7d-bbde-426b-a47f-687d886a8b44" (UID: "ffbd5e7d-bbde-426b-a47f-687d886a8b44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:47:20.418255 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:20.418233 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffbd5e7d-bbde-426b-a47f-687d886a8b44-istiod-ca-cert" pod "ffbd5e7d-bbde-426b-a47f-687d886a8b44" (UID: "ffbd5e7d-bbde-426b-a47f-687d886a8b44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:47:20.418332 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:20.418294 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffbd5e7d-bbde-426b-a47f-687d886a8b44-workload-socket" pod "ffbd5e7d-bbde-426b-a47f-687d886a8b44" (UID: "ffbd5e7d-bbde-426b-a47f-687d886a8b44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:47:20.419208 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:20.419188 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffbd5e7d-bbde-426b-a47f-687d886a8b44-istio-token" pod "ffbd5e7d-bbde-426b-a47f-687d886a8b44" (UID: "ffbd5e7d-bbde-426b-a47f-687d886a8b44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:47:20.419423 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:20.419408 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffbd5e7d-bbde-426b-a47f-687d886a8b44-istio-envoy" pod "ffbd5e7d-bbde-426b-a47f-687d886a8b44" (UID: "ffbd5e7d-bbde-426b-a47f-687d886a8b44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:47:20.419583 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:20.419569 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffbd5e7d-bbde-426b-a47f-687d886a8b44-istio-podinfo" pod "ffbd5e7d-bbde-426b-a47f-687d886a8b44" (UID: "ffbd5e7d-bbde-426b-a47f-687d886a8b44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:47:20.419711 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:20.419693 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffbd5e7d-bbde-426b-a47f-687d886a8b44-kube-api-access-jpf7s" pod "ffbd5e7d-bbde-426b-a47f-687d886a8b44" (UID: "ffbd5e7d-bbde-426b-a47f-687d886a8b44"). InnerVolumeSpecName "kube-api-access-jpf7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:47:20.518047 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:20.518018 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpf7s\" (UniqueName: \"kubernetes.io/projected/ffbd5e7d-bbde-426b-a47f-687d886a8b44-kube-api-access-jpf7s\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:20.518047 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:20.518046 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffbd5e7d-bbde-426b-a47f-687d886a8b44-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:20.518252 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:20.518056 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbd5e7d-bbde-426b-a47f-687d886a8b44-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:20.518252 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:20.518065 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffbd5e7d-bbde-426b-a47f-687d886a8b44-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:20.518252 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:20.518074 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffbd5e7d-bbde-426b-a47f-687d886a8b44-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:20.518252 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:20.518082 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffbd5e7d-bbde-426b-a47f-687d886a8b44-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:20.518252 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:20.518090 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffbd5e7d-bbde-426b-a47f-687d886a8b44-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:21.143977 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:21.143944 6841 generic.go:356] "Generic (PLEG): container finished" podID="ffbd5e7d-bbde-426b-a47f-687d886a8b44" containerID="d9c8f94eecbefd1d725551fac56a9b633cbed4d1d02ef92e9098a9c508335aee" exitCode=0 Oct 31 13:47:21.144174 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:21.144030 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" Oct 31 13:47:21.144174 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:21.144030 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" event={"ID":"ffbd5e7d-bbde-426b-a47f-687d886a8b44","Type":"ContainerDied","Data":"d9c8f94eecbefd1d725551fac56a9b633cbed4d1d02ef92e9098a9c508335aee"} Oct 31 13:47:21.144174 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:21.144089 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c" event={"ID":"ffbd5e7d-bbde-426b-a47f-687d886a8b44","Type":"ContainerDied","Data":"f3c84e8162d1046028e6a75e6df68a8b09c2c0d0f6d1e00ab113ce951cfe34d2"} Oct 31 13:47:21.144174 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:21.144116 6841 scope.go:122] "RemoveContainer" containerID="d9c8f94eecbefd1d725551fac56a9b633cbed4d1d02ef92e9098a9c508335aee" Oct 31 13:47:21.156425 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:21.156371 6841 scope.go:122] "RemoveContainer" containerID="d9c8f94eecbefd1d725551fac56a9b633cbed4d1d02ef92e9098a9c508335aee" Oct 31 13:47:21.156913 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:47:21.156878 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9c8f94eecbefd1d725551fac56a9b633cbed4d1d02ef92e9098a9c508335aee\": container with ID starting with d9c8f94eecbefd1d725551fac56a9b633cbed4d1d02ef92e9098a9c508335aee not found: ID does not exist" containerID="d9c8f94eecbefd1d725551fac56a9b633cbed4d1d02ef92e9098a9c508335aee" Oct 31 13:47:21.157069 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:21.156925 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9c8f94eecbefd1d725551fac56a9b633cbed4d1d02ef92e9098a9c508335aee"} err="failed to get container status \"d9c8f94eecbefd1d725551fac56a9b633cbed4d1d02ef92e9098a9c508335aee\": rpc error: code = NotFound desc = could not find container \"d9c8f94eecbefd1d725551fac56a9b633cbed4d1d02ef92e9098a9c508335aee\": container with ID starting with d9c8f94eecbefd1d725551fac56a9b633cbed4d1d02ef92e9098a9c508335aee not found: ID does not exist" Oct 31 13:47:21.168886 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:21.168854 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c"] Oct 31 13:47:21.177261 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:21.177237 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-22c4c"] Oct 31 13:47:21.640441 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:21.640404 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffbd5e7d-bbde-426b-a47f-687d886a8b44" path="/var/lib/kubelet/pods/ffbd5e7d-bbde-426b-a47f-687d886a8b44/volumes" Oct 31 13:47:22.860229 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:22.860196 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:23.166876 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.166790 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x"] Oct 31 13:47:23.167261 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.167233 6841 cpu_manager.go:414] "containerMap: removing container" podUID="ffbd5e7d-bbde-426b-a47f-687d886a8b44" containerName="istio-proxy" Oct 31 13:47:23.167261 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.167255 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffbd5e7d-bbde-426b-a47f-687d886a8b44" containerName="istio-proxy" Oct 31 13:47:23.167660 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.167311 6841 cpu_manager.go:414] "containerMap: removing container" podUID="716df414-c063-4bf8-b85c-fcb3c319fb70" containerName="payload-pre-processing" Oct 31 13:47:23.167660 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.167322 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="716df414-c063-4bf8-b85c-fcb3c319fb70" containerName="payload-pre-processing" Oct 31 13:47:23.167660 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.167333 6841 cpu_manager.go:414] "containerMap: removing container" podUID="f721bc13-8ce6-4d6b-b53a-65b9c0030e08" containerName="payload-processing" Oct 31 13:47:23.167660 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.167341 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f721bc13-8ce6-4d6b-b53a-65b9c0030e08" containerName="payload-processing" Oct 31 13:47:23.167660 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.167472 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffbd5e7d-bbde-426b-a47f-687d886a8b44" containerName="istio-proxy" Oct 31 13:47:23.167660 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.167496 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="716df414-c063-4bf8-b85c-fcb3c319fb70" containerName="payload-pre-processing" Oct 31 13:47:23.167660 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.167526 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="f721bc13-8ce6-4d6b-b53a-65b9c0030e08" containerName="payload-processing" Oct 31 13:47:23.172182 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.172162 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:23.176409 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.176387 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8504d853-openshift-default-dockercfg-9w887\"" Oct 31 13:47:23.184506 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.184480 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x"] Oct 31 13:47:23.243644 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.243615 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5674e96-0a31-4017-9b03-84eb47f22e0d-istiod-ca-cert\") pod \"e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x\" (UID: \"f5674e96-0a31-4017-9b03-84eb47f22e0d\") " pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:23.243844 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.243669 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5674e96-0a31-4017-9b03-84eb47f22e0d-istio-podinfo\") pod \"e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x\" (UID: \"f5674e96-0a31-4017-9b03-84eb47f22e0d\") " pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:23.243844 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.243691 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5674e96-0a31-4017-9b03-84eb47f22e0d-workload-socket\") pod \"e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x\" (UID: \"f5674e96-0a31-4017-9b03-84eb47f22e0d\") " pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:23.243844 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.243823 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5674e96-0a31-4017-9b03-84eb47f22e0d-istio-envoy\") pod \"e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x\" (UID: \"f5674e96-0a31-4017-9b03-84eb47f22e0d\") " pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:23.244077 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.243863 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5674e96-0a31-4017-9b03-84eb47f22e0d-istio-data\") pod \"e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x\" (UID: \"f5674e96-0a31-4017-9b03-84eb47f22e0d\") " pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:23.244077 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.243938 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5674e96-0a31-4017-9b03-84eb47f22e0d-istio-token\") pod \"e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x\" (UID: \"f5674e96-0a31-4017-9b03-84eb47f22e0d\") " pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:23.244077 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.243995 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5674e96-0a31-4017-9b03-84eb47f22e0d-credential-socket\") pod \"e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x\" (UID: \"f5674e96-0a31-4017-9b03-84eb47f22e0d\") " pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:23.244077 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.244031 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5674e96-0a31-4017-9b03-84eb47f22e0d-workload-certs\") pod \"e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x\" (UID: \"f5674e96-0a31-4017-9b03-84eb47f22e0d\") " pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:23.244077 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.244048 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv9gz\" (UniqueName: \"kubernetes.io/projected/f5674e96-0a31-4017-9b03-84eb47f22e0d-kube-api-access-cv9gz\") pod \"e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x\" (UID: \"f5674e96-0a31-4017-9b03-84eb47f22e0d\") " pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:23.315018 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.310480 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:23.345199 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.345166 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5674e96-0a31-4017-9b03-84eb47f22e0d-istio-envoy\") pod \"e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x\" (UID: \"f5674e96-0a31-4017-9b03-84eb47f22e0d\") " pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:23.345199 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.345202 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5674e96-0a31-4017-9b03-84eb47f22e0d-istio-data\") pod \"e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x\" (UID: \"f5674e96-0a31-4017-9b03-84eb47f22e0d\") " pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:23.345442 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.345231 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5674e96-0a31-4017-9b03-84eb47f22e0d-istio-token\") pod \"e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x\" (UID: \"f5674e96-0a31-4017-9b03-84eb47f22e0d\") " pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:23.345442 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.345251 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5674e96-0a31-4017-9b03-84eb47f22e0d-credential-socket\") pod \"e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x\" (UID: \"f5674e96-0a31-4017-9b03-84eb47f22e0d\") " pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:23.345442 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.345294 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5674e96-0a31-4017-9b03-84eb47f22e0d-workload-certs\") pod \"e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x\" (UID: \"f5674e96-0a31-4017-9b03-84eb47f22e0d\") " pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:23.345442 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.345321 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv9gz\" (UniqueName: \"kubernetes.io/projected/f5674e96-0a31-4017-9b03-84eb47f22e0d-kube-api-access-cv9gz\") pod \"e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x\" (UID: \"f5674e96-0a31-4017-9b03-84eb47f22e0d\") " pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:23.345442 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.345388 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5674e96-0a31-4017-9b03-84eb47f22e0d-istiod-ca-cert\") pod \"e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x\" (UID: \"f5674e96-0a31-4017-9b03-84eb47f22e0d\") " pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:23.345714 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.345461 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5674e96-0a31-4017-9b03-84eb47f22e0d-istio-podinfo\") pod \"e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x\" (UID: \"f5674e96-0a31-4017-9b03-84eb47f22e0d\") " pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:23.345714 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.345490 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5674e96-0a31-4017-9b03-84eb47f22e0d-workload-socket\") pod \"e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x\" (UID: \"f5674e96-0a31-4017-9b03-84eb47f22e0d\") " pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:23.345714 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.345693 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5674e96-0a31-4017-9b03-84eb47f22e0d-istio-data\") pod \"e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x\" (UID: \"f5674e96-0a31-4017-9b03-84eb47f22e0d\") " pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:23.345838 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.345766 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5674e96-0a31-4017-9b03-84eb47f22e0d-workload-certs\") pod \"e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x\" (UID: \"f5674e96-0a31-4017-9b03-84eb47f22e0d\") " pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:23.345918 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.345901 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5674e96-0a31-4017-9b03-84eb47f22e0d-workload-socket\") pod \"e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x\" (UID: \"f5674e96-0a31-4017-9b03-84eb47f22e0d\") " pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:23.345966 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.345934 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5674e96-0a31-4017-9b03-84eb47f22e0d-credential-socket\") pod \"e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x\" (UID: \"f5674e96-0a31-4017-9b03-84eb47f22e0d\") " pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:23.346270 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.346247 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5674e96-0a31-4017-9b03-84eb47f22e0d-istiod-ca-cert\") pod \"e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x\" (UID: \"f5674e96-0a31-4017-9b03-84eb47f22e0d\") " pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:23.347645 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.347619 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5674e96-0a31-4017-9b03-84eb47f22e0d-istio-envoy\") pod \"e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x\" (UID: \"f5674e96-0a31-4017-9b03-84eb47f22e0d\") " pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:23.348448 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.348419 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5674e96-0a31-4017-9b03-84eb47f22e0d-istio-podinfo\") pod \"e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x\" (UID: \"f5674e96-0a31-4017-9b03-84eb47f22e0d\") " pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:23.358061 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.358036 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5674e96-0a31-4017-9b03-84eb47f22e0d-istio-token\") pod \"e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x\" (UID: \"f5674e96-0a31-4017-9b03-84eb47f22e0d\") " pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:23.358155 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.358105 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv9gz\" (UniqueName: \"kubernetes.io/projected/f5674e96-0a31-4017-9b03-84eb47f22e0d-kube-api-access-cv9gz\") pod \"e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x\" (UID: \"f5674e96-0a31-4017-9b03-84eb47f22e0d\") " pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:23.486400 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.486373 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:23.525673 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.525635 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:23.768635 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.768609 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x"] Oct 31 13:47:23.771024 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:47:23.770998 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5674e96_0a31_4017_9b03_84eb47f22e0d.slice/crio-802ab427eba16963d5640c2d9757007d9c68f35b7cae48c4912524fcdb367b75 WatchSource:0}: Error finding container 802ab427eba16963d5640c2d9757007d9c68f35b7cae48c4912524fcdb367b75: Status 404 returned error can't find the container with id 802ab427eba16963d5640c2d9757007d9c68f35b7cae48c4912524fcdb367b75 Oct 31 13:47:23.773171 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.773139 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:47:23.773274 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.773209 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:47:23.773274 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:23.773238 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:47:24.164311 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:24.164238 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" event={"ID":"f5674e96-0a31-4017-9b03-84eb47f22e0d","Type":"ContainerStarted","Data":"a4614f7923c3b37da59c81267d051af82067e8080805e2b4d53a42abdeacd0e3"} Oct 31 13:47:24.164311 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:24.164288 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" event={"ID":"f5674e96-0a31-4017-9b03-84eb47f22e0d","Type":"ContainerStarted","Data":"802ab427eba16963d5640c2d9757007d9c68f35b7cae48c4912524fcdb367b75"} Oct 31 13:47:24.487062 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:24.487024 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:24.491962 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:24.491935 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:24.517849 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:24.517804 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" podStartSLOduration=1.517789741 podStartE2EDuration="1.517789741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:47:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:47:24.204858353 +0000 UTC m=+4490.687981254" watchObservedRunningTime="2026-10-31 13:47:24.517789741 +0000 UTC m=+4491.000912642" Oct 31 13:47:25.171673 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:25.171641 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:25.172621 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:25.172601 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:25.781705 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:25.781664 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:25.994584 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:25.994548 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:26.874305 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:26.874269 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8504d853-7c5d5f9474-nj667"] Oct 31 13:47:26.882934 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:26.882906 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8504d853-7c5d5f9474-nj667" Oct 31 13:47:26.892886 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:26.892862 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8504d853-serving-cert\"" Oct 31 13:47:26.893236 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:26.893219 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:47:26.893577 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:26.893548 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:47:26.893784 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:26.893764 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 13:47:26.893904 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:26.893890 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rhftr\"" Oct 31 13:47:26.896008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:26.895965 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8504d853-7c5d5f9474-nj667"] Oct 31 13:47:26.980254 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:26.980211 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/121a41a2-f7ea-4d68-bb47-f3446fb847bd-metrics-certs\") pod \"maas-api-e2e-ait-8504d853-7c5d5f9474-nj667\" (UID: \"121a41a2-f7ea-4d68-bb47-f3446fb847bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8504d853-7c5d5f9474-nj667" Oct 31 13:47:26.980428 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:26.980259 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gthnq\" (UniqueName: \"kubernetes.io/projected/121a41a2-f7ea-4d68-bb47-f3446fb847bd-kube-api-access-gthnq\") pod \"maas-api-e2e-ait-8504d853-7c5d5f9474-nj667\" (UID: \"121a41a2-f7ea-4d68-bb47-f3446fb847bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8504d853-7c5d5f9474-nj667" Oct 31 13:47:26.980428 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:26.980336 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/121a41a2-f7ea-4d68-bb47-f3446fb847bd-maas-api-tls\") pod \"maas-api-e2e-ait-8504d853-7c5d5f9474-nj667\" (UID: \"121a41a2-f7ea-4d68-bb47-f3446fb847bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8504d853-7c5d5f9474-nj667" Oct 31 13:47:26.980428 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:26.980400 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/121a41a2-f7ea-4d68-bb47-f3446fb847bd-openshift-service-ca\") pod \"maas-api-e2e-ait-8504d853-7c5d5f9474-nj667\" (UID: \"121a41a2-f7ea-4d68-bb47-f3446fb847bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8504d853-7c5d5f9474-nj667" Oct 31 13:47:27.044841 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:27.044810 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:27.081879 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:27.081848 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/121a41a2-f7ea-4d68-bb47-f3446fb847bd-metrics-certs\") pod \"maas-api-e2e-ait-8504d853-7c5d5f9474-nj667\" (UID: \"121a41a2-f7ea-4d68-bb47-f3446fb847bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8504d853-7c5d5f9474-nj667" Oct 31 13:47:27.082077 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:27.081883 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gthnq\" (UniqueName: \"kubernetes.io/projected/121a41a2-f7ea-4d68-bb47-f3446fb847bd-kube-api-access-gthnq\") pod \"maas-api-e2e-ait-8504d853-7c5d5f9474-nj667\" (UID: \"121a41a2-f7ea-4d68-bb47-f3446fb847bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8504d853-7c5d5f9474-nj667" Oct 31 13:47:27.082077 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:27.081903 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/121a41a2-f7ea-4d68-bb47-f3446fb847bd-maas-api-tls\") pod \"maas-api-e2e-ait-8504d853-7c5d5f9474-nj667\" (UID: \"121a41a2-f7ea-4d68-bb47-f3446fb847bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8504d853-7c5d5f9474-nj667" Oct 31 13:47:27.082077 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:27.081938 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/121a41a2-f7ea-4d68-bb47-f3446fb847bd-openshift-service-ca\") pod \"maas-api-e2e-ait-8504d853-7c5d5f9474-nj667\" (UID: \"121a41a2-f7ea-4d68-bb47-f3446fb847bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8504d853-7c5d5f9474-nj667" Oct 31 13:47:27.082605 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:27.082584 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/121a41a2-f7ea-4d68-bb47-f3446fb847bd-openshift-service-ca\") pod \"maas-api-e2e-ait-8504d853-7c5d5f9474-nj667\" (UID: \"121a41a2-f7ea-4d68-bb47-f3446fb847bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8504d853-7c5d5f9474-nj667" Oct 31 13:47:27.084467 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:27.084442 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/121a41a2-f7ea-4d68-bb47-f3446fb847bd-maas-api-tls\") pod \"maas-api-e2e-ait-8504d853-7c5d5f9474-nj667\" (UID: \"121a41a2-f7ea-4d68-bb47-f3446fb847bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8504d853-7c5d5f9474-nj667" Oct 31 13:47:27.084607 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:27.084553 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/121a41a2-f7ea-4d68-bb47-f3446fb847bd-metrics-certs\") pod \"maas-api-e2e-ait-8504d853-7c5d5f9474-nj667\" (UID: \"121a41a2-f7ea-4d68-bb47-f3446fb847bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8504d853-7c5d5f9474-nj667" Oct 31 13:47:27.096184 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:27.096159 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gthnq\" (UniqueName: \"kubernetes.io/projected/121a41a2-f7ea-4d68-bb47-f3446fb847bd-kube-api-access-gthnq\") pod \"maas-api-e2e-ait-8504d853-7c5d5f9474-nj667\" (UID: \"121a41a2-f7ea-4d68-bb47-f3446fb847bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8504d853-7c5d5f9474-nj667" Oct 31 13:47:27.201155 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:27.201123 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8504d853-7c5d5f9474-nj667" Oct 31 13:47:27.283006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:27.282947 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:27.480679 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:27.480653 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8504d853-7c5d5f9474-nj667"] Oct 31 13:47:27.482808 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:47:27.482777 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod121a41a2_f7ea_4d68_bb47_f3446fb847bd.slice/crio-5d1d182ac023bdac7eafc4525842c47971f31abd9479f02745f0d051c6118f5a WatchSource:0}: Error finding container 5d1d182ac023bdac7eafc4525842c47971f31abd9479f02745f0d051c6118f5a: Status 404 returned error can't find the container with id 5d1d182ac023bdac7eafc4525842c47971f31abd9479f02745f0d051c6118f5a Oct 31 13:47:27.484240 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:27.484217 6841 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 13:47:28.067549 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:28.067523 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:28.188971 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:28.188933 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8504d853-7c5d5f9474-nj667" event={"ID":"121a41a2-f7ea-4d68-bb47-f3446fb847bd","Type":"ContainerStarted","Data":"2d56629260796aea21d5966b6351e883a35c9a6e8fc97151dec53e88923fa696"} Oct 31 13:47:28.189147 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:28.189021 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8504d853-7c5d5f9474-nj667" event={"ID":"121a41a2-f7ea-4d68-bb47-f3446fb847bd","Type":"ContainerStarted","Data":"5d1d182ac023bdac7eafc4525842c47971f31abd9479f02745f0d051c6118f5a"} Oct 31 13:47:28.189147 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:28.189124 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8504d853-7c5d5f9474-nj667" Oct 31 13:47:28.209471 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:28.209421 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8504d853-7c5d5f9474-nj667" podStartSLOduration=1.6832694030000002 podStartE2EDuration="2.209406992s" totalImagesPullingTime="526.137589ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:47:28.208451454 +0000 UTC m=+4494.691574356" watchObservedRunningTime="2026-10-31 13:47:28.209406992 +0000 UTC m=+4494.692529893" Oct 31 13:47:28.921124 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:28.921075 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:29.680511 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:29.680478 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:30.618179 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:30.618144 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:31.312290 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:31.312252 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:31.968154 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:31.965608 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:32.206726 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:32.206684 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:32.705722 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:32.705676 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:33.416014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:33.415943 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:34.104880 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:34.104845 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:34.200769 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:34.200743 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8504d853-7c5d5f9474-nj667" Oct 31 13:47:34.813280 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:34.813248 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:35.683427 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:35.683389 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:36.547586 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:36.547545 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:37.303186 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:37.303136 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:38.309974 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:38.309941 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:39.177843 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:39.177754 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:39.251397 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:39.251365 6841 generic.go:356] "Generic (PLEG): container finished" podID="f5674e96-0a31-4017-9b03-84eb47f22e0d" containerID="a4614f7923c3b37da59c81267d051af82067e8080805e2b4d53a42abdeacd0e3" exitCode=0 Oct 31 13:47:39.251583 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:39.251439 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" event={"ID":"f5674e96-0a31-4017-9b03-84eb47f22e0d","Type":"ContainerDied","Data":"a4614f7923c3b37da59c81267d051af82067e8080805e2b4d53a42abdeacd0e3"} Oct 31 13:47:39.251848 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:39.251832 6841 scope.go:122] "RemoveContainer" containerID="a4614f7923c3b37da59c81267d051af82067e8080805e2b4d53a42abdeacd0e3" Oct 31 13:47:39.253100 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:39.253069 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:47:39.253233 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:39.253153 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:47:39.253233 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:39.253220 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:47:39.926602 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:39.926567 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:40.171785 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:40.171753 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:40.272706 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:40.272672 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" event={"ID":"f5674e96-0a31-4017-9b03-84eb47f22e0d","Type":"ContainerStarted","Data":"b8146a411e40e7e84622fc9b917493de08a370f7f90d74554c5387c266f935c8"} Oct 31 13:47:40.486799 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:40.486760 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:40.488054 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:40.488022 6841 patch_prober.go:29] interesting pod/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Oct 31 13:47:40.488192 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:40.488084 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" podUID="f5674e96-0a31-4017-9b03-84eb47f22e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Oct 31 13:47:40.808616 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:40.808571 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:41.276774 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:41.276739 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:41.487448 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:41.487410 6841 patch_prober.go:29] interesting pod/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Oct 31 13:47:41.487629 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:41.487482 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" podUID="f5674e96-0a31-4017-9b03-84eb47f22e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Oct 31 13:47:41.533613 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:41.533522 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:42.487300 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:42.487266 6841 patch_prober.go:29] interesting pod/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Oct 31 13:47:42.487726 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:42.487326 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" podUID="f5674e96-0a31-4017-9b03-84eb47f22e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Oct 31 13:47:42.568439 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:42.568395 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:43.486868 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:43.486826 6841 patch_prober.go:29] interesting pod/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Oct 31 13:47:43.487052 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:43.486897 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" podUID="f5674e96-0a31-4017-9b03-84eb47f22e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Oct 31 13:47:43.529594 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:43.529556 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:44.266767 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:44.266729 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:44.487362 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:44.487323 6841 patch_prober.go:29] interesting pod/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Oct 31 13:47:44.487607 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:44.487381 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" podUID="f5674e96-0a31-4017-9b03-84eb47f22e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Oct 31 13:47:45.010528 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:45.010491 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:45.491299 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:45.491267 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:45.491647 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:45.491625 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:45.492304 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:45.492284 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:47:45.831731 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:45.831615 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:46.616413 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:46.616375 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:47.431814 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:47.431779 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:48.304301 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:48.304268 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:49.060938 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:49.060897 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:49.889900 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:49.889871 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:50.036817 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:47:50.036781 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:47:50.632736 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:50.632650 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:51.450816 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:51.450771 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:51.891037 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:51.890910 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:52.120688 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:52.120635 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:52.356777 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:52.356740 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:52.719300 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:52.719267 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8504d853-7c5d5f9474-nj667"] Oct 31 13:47:52.719809 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:52.719507 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8504d853-7c5d5f9474-nj667" podUID="121a41a2-f7ea-4d68-bb47-f3446fb847bd" containerName="maas-api" containerID="cri-o://2d56629260796aea21d5966b6351e883a35c9a6e8fc97151dec53e88923fa696" gracePeriod=30 Oct 31 13:47:52.876333 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:52.876295 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:52.985265 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:52.985204 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8504d853-7c5d5f9474-nj667" Oct 31 13:47:53.074075 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:53.074040 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/121a41a2-f7ea-4d68-bb47-f3446fb847bd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/121a41a2-f7ea-4d68-bb47-f3446fb847bd-metrics-certs\") pod \"121a41a2-f7ea-4d68-bb47-f3446fb847bd\" (UID: \"121a41a2-f7ea-4d68-bb47-f3446fb847bd\") " Oct 31 13:47:53.074267 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:53.074135 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/121a41a2-f7ea-4d68-bb47-f3446fb847bd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/121a41a2-f7ea-4d68-bb47-f3446fb847bd-maas-api-tls\") pod \"121a41a2-f7ea-4d68-bb47-f3446fb847bd\" (UID: \"121a41a2-f7ea-4d68-bb47-f3446fb847bd\") " Oct 31 13:47:53.074267 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:53.074202 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/121a41a2-f7ea-4d68-bb47-f3446fb847bd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/121a41a2-f7ea-4d68-bb47-f3446fb847bd-openshift-service-ca\") pod \"121a41a2-f7ea-4d68-bb47-f3446fb847bd\" (UID: \"121a41a2-f7ea-4d68-bb47-f3446fb847bd\") " Oct 31 13:47:53.074267 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:53.074244 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/121a41a2-f7ea-4d68-bb47-f3446fb847bd-kube-api-access-gthnq\" (UniqueName: \"kubernetes.io/projected/121a41a2-f7ea-4d68-bb47-f3446fb847bd-kube-api-access-gthnq\") pod \"121a41a2-f7ea-4d68-bb47-f3446fb847bd\" (UID: \"121a41a2-f7ea-4d68-bb47-f3446fb847bd\") " Oct 31 13:47:53.074563 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:53.074536 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/121a41a2-f7ea-4d68-bb47-f3446fb847bd-openshift-service-ca" pod "121a41a2-f7ea-4d68-bb47-f3446fb847bd" (UID: "121a41a2-f7ea-4d68-bb47-f3446fb847bd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:47:53.076340 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:53.076316 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/121a41a2-f7ea-4d68-bb47-f3446fb847bd-kube-api-access-gthnq" pod "121a41a2-f7ea-4d68-bb47-f3446fb847bd" (UID: "121a41a2-f7ea-4d68-bb47-f3446fb847bd"). InnerVolumeSpecName "kube-api-access-gthnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:47:53.076451 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:53.076368 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/121a41a2-f7ea-4d68-bb47-f3446fb847bd-maas-api-tls" pod "121a41a2-f7ea-4d68-bb47-f3446fb847bd" (UID: "121a41a2-f7ea-4d68-bb47-f3446fb847bd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:47:53.076451 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:53.076397 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/121a41a2-f7ea-4d68-bb47-f3446fb847bd-metrics-certs" pod "121a41a2-f7ea-4d68-bb47-f3446fb847bd" (UID: "121a41a2-f7ea-4d68-bb47-f3446fb847bd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:47:53.084146 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:53.084117 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:53.175240 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:53.175199 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/121a41a2-f7ea-4d68-bb47-f3446fb847bd-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:53.175240 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:53.175233 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/121a41a2-f7ea-4d68-bb47-f3446fb847bd-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:53.175240 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:53.175243 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gthnq\" (UniqueName: \"kubernetes.io/projected/121a41a2-f7ea-4d68-bb47-f3446fb847bd-kube-api-access-gthnq\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:53.175495 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:53.175252 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/121a41a2-f7ea-4d68-bb47-f3446fb847bd-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:53.291307 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:53.291217 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:47:53.350155 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:53.350122 6841 generic.go:356] "Generic (PLEG): container finished" podID="121a41a2-f7ea-4d68-bb47-f3446fb847bd" containerID="2d56629260796aea21d5966b6351e883a35c9a6e8fc97151dec53e88923fa696" exitCode=0 Oct 31 13:47:53.350325 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:53.350174 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8504d853-7c5d5f9474-nj667" Oct 31 13:47:53.350325 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:53.350173 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8504d853-7c5d5f9474-nj667" event={"ID":"121a41a2-f7ea-4d68-bb47-f3446fb847bd","Type":"ContainerDied","Data":"2d56629260796aea21d5966b6351e883a35c9a6e8fc97151dec53e88923fa696"} Oct 31 13:47:53.350325 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:53.350293 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8504d853-7c5d5f9474-nj667" event={"ID":"121a41a2-f7ea-4d68-bb47-f3446fb847bd","Type":"ContainerDied","Data":"5d1d182ac023bdac7eafc4525842c47971f31abd9479f02745f0d051c6118f5a"} Oct 31 13:47:53.350325 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:53.350315 6841 scope.go:122] "RemoveContainer" containerID="2d56629260796aea21d5966b6351e883a35c9a6e8fc97151dec53e88923fa696" Oct 31 13:47:53.360202 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:53.360184 6841 scope.go:122] "RemoveContainer" containerID="2d56629260796aea21d5966b6351e883a35c9a6e8fc97151dec53e88923fa696" Oct 31 13:47:53.360467 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:47:53.360447 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d56629260796aea21d5966b6351e883a35c9a6e8fc97151dec53e88923fa696\": container with ID starting with 2d56629260796aea21d5966b6351e883a35c9a6e8fc97151dec53e88923fa696 not found: ID does not exist" containerID="2d56629260796aea21d5966b6351e883a35c9a6e8fc97151dec53e88923fa696" Oct 31 13:47:53.360508 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:53.360479 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d56629260796aea21d5966b6351e883a35c9a6e8fc97151dec53e88923fa696"} err="failed to get container status \"2d56629260796aea21d5966b6351e883a35c9a6e8fc97151dec53e88923fa696\": rpc error: code = NotFound desc = could not find container \"2d56629260796aea21d5966b6351e883a35c9a6e8fc97151dec53e88923fa696\": container with ID starting with 2d56629260796aea21d5966b6351e883a35c9a6e8fc97151dec53e88923fa696 not found: ID does not exist" Oct 31 13:47:53.378010 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:53.377730 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8504d853-7c5d5f9474-nj667"] Oct 31 13:47:53.385189 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:53.385161 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8504d853-7c5d5f9474-nj667"] Oct 31 13:47:53.641530 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:53.641450 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="121a41a2-f7ea-4d68-bb47-f3446fb847bd" path="/var/lib/kubelet/pods/121a41a2-f7ea-4d68-bb47-f3446fb847bd/volumes" Oct 31 13:47:58.018176 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:58.018138 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x"] Oct 31 13:47:58.018617 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:58.018396 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" podUID="f5674e96-0a31-4017-9b03-84eb47f22e0d" containerName="istio-proxy" containerID="cri-o://b8146a411e40e7e84622fc9b917493de08a370f7f90d74554c5387c266f935c8" gracePeriod=30 Oct 31 13:47:58.165485 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:47:58.165448 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:48:00.494745 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:00.494706 6841 patch_prober.go:29] interesting pod/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Oct 31 13:48:00.495226 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:00.494769 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" podUID="f5674e96-0a31-4017-9b03-84eb47f22e0d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:48:01.570538 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:01.570499 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7767cbb97d-f7bp4"] Oct 31 13:48:01.570991 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:01.570939 6841 cpu_manager.go:414] "containerMap: removing container" podUID="121a41a2-f7ea-4d68-bb47-f3446fb847bd" containerName="maas-api" Oct 31 13:48:01.570991 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:01.570951 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="121a41a2-f7ea-4d68-bb47-f3446fb847bd" containerName="maas-api" Oct 31 13:48:01.571122 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:01.571108 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="121a41a2-f7ea-4d68-bb47-f3446fb847bd" containerName="maas-api" Oct 31 13:48:01.575773 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:01.575754 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7767cbb97d-f7bp4" Oct 31 13:48:01.582373 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:01.582345 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 13:48:01.584146 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:01.583931 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 13:48:01.584146 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:01.584054 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jnfxm\"" Oct 31 13:48:01.584329 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:01.584161 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 13:48:01.587008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:01.586965 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7767cbb97d-f7bp4"] Oct 31 13:48:01.668503 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:01.668449 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fb07c240-075b-4655-99eb-74d60601c845-webhook-cert\") pod \"maas-controller-7767cbb97d-f7bp4\" (UID: \"fb07c240-075b-4655-99eb-74d60601c845\") " pod="opendatahub/maas-controller-7767cbb97d-f7bp4" Oct 31 13:48:01.668697 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:01.668539 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbb74\" (UniqueName: \"kubernetes.io/projected/fb07c240-075b-4655-99eb-74d60601c845-kube-api-access-vbb74\") pod \"maas-controller-7767cbb97d-f7bp4\" (UID: \"fb07c240-075b-4655-99eb-74d60601c845\") " pod="opendatahub/maas-controller-7767cbb97d-f7bp4" Oct 31 13:48:01.668697 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:01.668621 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb07c240-075b-4655-99eb-74d60601c845-metrics-certs\") pod \"maas-controller-7767cbb97d-f7bp4\" (UID: \"fb07c240-075b-4655-99eb-74d60601c845\") " pod="opendatahub/maas-controller-7767cbb97d-f7bp4" Oct 31 13:48:01.770189 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:01.770120 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbb74\" (UniqueName: \"kubernetes.io/projected/fb07c240-075b-4655-99eb-74d60601c845-kube-api-access-vbb74\") pod \"maas-controller-7767cbb97d-f7bp4\" (UID: \"fb07c240-075b-4655-99eb-74d60601c845\") " pod="opendatahub/maas-controller-7767cbb97d-f7bp4" Oct 31 13:48:01.770427 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:01.770223 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb07c240-075b-4655-99eb-74d60601c845-metrics-certs\") pod \"maas-controller-7767cbb97d-f7bp4\" (UID: \"fb07c240-075b-4655-99eb-74d60601c845\") " pod="opendatahub/maas-controller-7767cbb97d-f7bp4" Oct 31 13:48:01.770427 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:01.770365 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fb07c240-075b-4655-99eb-74d60601c845-webhook-cert\") pod \"maas-controller-7767cbb97d-f7bp4\" (UID: \"fb07c240-075b-4655-99eb-74d60601c845\") " pod="opendatahub/maas-controller-7767cbb97d-f7bp4" Oct 31 13:48:01.798931 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:01.798869 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fb07c240-075b-4655-99eb-74d60601c845-webhook-cert\") pod \"maas-controller-7767cbb97d-f7bp4\" (UID: \"fb07c240-075b-4655-99eb-74d60601c845\") " pod="opendatahub/maas-controller-7767cbb97d-f7bp4" Oct 31 13:48:01.799584 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:01.799553 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb07c240-075b-4655-99eb-74d60601c845-metrics-certs\") pod \"maas-controller-7767cbb97d-f7bp4\" (UID: \"fb07c240-075b-4655-99eb-74d60601c845\") " pod="opendatahub/maas-controller-7767cbb97d-f7bp4" Oct 31 13:48:01.802749 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:01.802698 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbb74\" (UniqueName: \"kubernetes.io/projected/fb07c240-075b-4655-99eb-74d60601c845-kube-api-access-vbb74\") pod \"maas-controller-7767cbb97d-f7bp4\" (UID: \"fb07c240-075b-4655-99eb-74d60601c845\") " pod="opendatahub/maas-controller-7767cbb97d-f7bp4" Oct 31 13:48:01.895836 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:01.895729 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7767cbb97d-f7bp4" Oct 31 13:48:02.213591 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:02.213551 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7767cbb97d-f7bp4"] Oct 31 13:48:02.215737 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:48:02.215708 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb07c240_075b_4655_99eb_74d60601c845.slice/crio-282fea84090e340add06a0a65c49ea5719ff72530983112fc12ed25a81067814 WatchSource:0}: Error finding container 282fea84090e340add06a0a65c49ea5719ff72530983112fc12ed25a81067814: Status 404 returned error can't find the container with id 282fea84090e340add06a0a65c49ea5719ff72530983112fc12ed25a81067814 Oct 31 13:48:02.408048 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:02.408000 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7767cbb97d-f7bp4" event={"ID":"fb07c240-075b-4655-99eb-74d60601c845","Type":"ContainerStarted","Data":"282fea84090e340add06a0a65c49ea5719ff72530983112fc12ed25a81067814"} Oct 31 13:48:03.293126 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.293064 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:48:03.391513 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.391480 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5674e96-0a31-4017-9b03-84eb47f22e0d-kube-api-access-cv9gz\" (UniqueName: \"kubernetes.io/projected/f5674e96-0a31-4017-9b03-84eb47f22e0d-kube-api-access-cv9gz\") pod \"f5674e96-0a31-4017-9b03-84eb47f22e0d\" (UID: \"f5674e96-0a31-4017-9b03-84eb47f22e0d\") " Oct 31 13:48:03.391708 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.391539 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5674e96-0a31-4017-9b03-84eb47f22e0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5674e96-0a31-4017-9b03-84eb47f22e0d-istiod-ca-cert\") pod \"f5674e96-0a31-4017-9b03-84eb47f22e0d\" (UID: \"f5674e96-0a31-4017-9b03-84eb47f22e0d\") " Oct 31 13:48:03.391708 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.391560 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5674e96-0a31-4017-9b03-84eb47f22e0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5674e96-0a31-4017-9b03-84eb47f22e0d-workload-socket\") pod \"f5674e96-0a31-4017-9b03-84eb47f22e0d\" (UID: \"f5674e96-0a31-4017-9b03-84eb47f22e0d\") " Oct 31 13:48:03.391708 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.391615 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5674e96-0a31-4017-9b03-84eb47f22e0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5674e96-0a31-4017-9b03-84eb47f22e0d-istio-envoy\") pod \"f5674e96-0a31-4017-9b03-84eb47f22e0d\" (UID: \"f5674e96-0a31-4017-9b03-84eb47f22e0d\") " Oct 31 13:48:03.391708 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.391664 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5674e96-0a31-4017-9b03-84eb47f22e0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5674e96-0a31-4017-9b03-84eb47f22e0d-workload-certs\") pod \"f5674e96-0a31-4017-9b03-84eb47f22e0d\" (UID: \"f5674e96-0a31-4017-9b03-84eb47f22e0d\") " Oct 31 13:48:03.391915 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.391716 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5674e96-0a31-4017-9b03-84eb47f22e0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5674e96-0a31-4017-9b03-84eb47f22e0d-istio-podinfo\") pod \"f5674e96-0a31-4017-9b03-84eb47f22e0d\" (UID: \"f5674e96-0a31-4017-9b03-84eb47f22e0d\") " Oct 31 13:48:03.391915 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.391760 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5674e96-0a31-4017-9b03-84eb47f22e0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5674e96-0a31-4017-9b03-84eb47f22e0d-istio-data\") pod \"f5674e96-0a31-4017-9b03-84eb47f22e0d\" (UID: \"f5674e96-0a31-4017-9b03-84eb47f22e0d\") " Oct 31 13:48:03.391915 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.391804 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5674e96-0a31-4017-9b03-84eb47f22e0d-istio-token\" (UniqueName: \"kubernetes.io/projected/f5674e96-0a31-4017-9b03-84eb47f22e0d-istio-token\") pod \"f5674e96-0a31-4017-9b03-84eb47f22e0d\" (UID: \"f5674e96-0a31-4017-9b03-84eb47f22e0d\") " Oct 31 13:48:03.391915 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.391839 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5674e96-0a31-4017-9b03-84eb47f22e0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5674e96-0a31-4017-9b03-84eb47f22e0d-credential-socket\") pod \"f5674e96-0a31-4017-9b03-84eb47f22e0d\" (UID: \"f5674e96-0a31-4017-9b03-84eb47f22e0d\") " Oct 31 13:48:03.392152 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.392066 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5674e96-0a31-4017-9b03-84eb47f22e0d-workload-certs" pod "f5674e96-0a31-4017-9b03-84eb47f22e0d" (UID: "f5674e96-0a31-4017-9b03-84eb47f22e0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:48:03.392210 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.392164 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5674e96-0a31-4017-9b03-84eb47f22e0d-istio-data" pod "f5674e96-0a31-4017-9b03-84eb47f22e0d" (UID: "f5674e96-0a31-4017-9b03-84eb47f22e0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:48:03.392262 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.392203 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5674e96-0a31-4017-9b03-84eb47f22e0d-workload-socket" pod "f5674e96-0a31-4017-9b03-84eb47f22e0d" (UID: "f5674e96-0a31-4017-9b03-84eb47f22e0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:48:03.392338 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.392319 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5674e96-0a31-4017-9b03-84eb47f22e0d-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:48:03.392408 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.392344 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5674e96-0a31-4017-9b03-84eb47f22e0d-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:48:03.392408 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.392359 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5674e96-0a31-4017-9b03-84eb47f22e0d-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:48:03.392533 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.392445 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5674e96-0a31-4017-9b03-84eb47f22e0d-credential-socket" pod "f5674e96-0a31-4017-9b03-84eb47f22e0d" (UID: "f5674e96-0a31-4017-9b03-84eb47f22e0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:48:03.392533 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.392526 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5674e96-0a31-4017-9b03-84eb47f22e0d-istiod-ca-cert" pod "f5674e96-0a31-4017-9b03-84eb47f22e0d" (UID: "f5674e96-0a31-4017-9b03-84eb47f22e0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:48:03.394475 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.394442 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5674e96-0a31-4017-9b03-84eb47f22e0d-kube-api-access-cv9gz" pod "f5674e96-0a31-4017-9b03-84eb47f22e0d" (UID: "f5674e96-0a31-4017-9b03-84eb47f22e0d"). InnerVolumeSpecName "kube-api-access-cv9gz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:48:03.394645 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.394607 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5674e96-0a31-4017-9b03-84eb47f22e0d-istio-podinfo" pod "f5674e96-0a31-4017-9b03-84eb47f22e0d" (UID: "f5674e96-0a31-4017-9b03-84eb47f22e0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:48:03.394793 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.394762 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5674e96-0a31-4017-9b03-84eb47f22e0d-istio-token" pod "f5674e96-0a31-4017-9b03-84eb47f22e0d" (UID: "f5674e96-0a31-4017-9b03-84eb47f22e0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:48:03.395085 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.395065 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5674e96-0a31-4017-9b03-84eb47f22e0d-istio-envoy" pod "f5674e96-0a31-4017-9b03-84eb47f22e0d" (UID: "f5674e96-0a31-4017-9b03-84eb47f22e0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:48:03.416795 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.416765 6841 generic.go:356] "Generic (PLEG): container finished" podID="f5674e96-0a31-4017-9b03-84eb47f22e0d" containerID="b8146a411e40e7e84622fc9b917493de08a370f7f90d74554c5387c266f935c8" exitCode=0 Oct 31 13:48:03.416918 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.416836 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" Oct 31 13:48:03.416918 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.416846 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" event={"ID":"f5674e96-0a31-4017-9b03-84eb47f22e0d","Type":"ContainerDied","Data":"b8146a411e40e7e84622fc9b917493de08a370f7f90d74554c5387c266f935c8"} Oct 31 13:48:03.416918 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.416914 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x" event={"ID":"f5674e96-0a31-4017-9b03-84eb47f22e0d","Type":"ContainerDied","Data":"802ab427eba16963d5640c2d9757007d9c68f35b7cae48c4912524fcdb367b75"} Oct 31 13:48:03.417131 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.416945 6841 scope.go:122] "RemoveContainer" containerID="b8146a411e40e7e84622fc9b917493de08a370f7f90d74554c5387c266f935c8" Oct 31 13:48:03.428135 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.428118 6841 scope.go:122] "RemoveContainer" containerID="a4614f7923c3b37da59c81267d051af82067e8080805e2b4d53a42abdeacd0e3" Oct 31 13:48:03.436947 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.436927 6841 scope.go:122] "RemoveContainer" containerID="b8146a411e40e7e84622fc9b917493de08a370f7f90d74554c5387c266f935c8" Oct 31 13:48:03.437288 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:48:03.437247 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8146a411e40e7e84622fc9b917493de08a370f7f90d74554c5387c266f935c8\": container with ID starting with b8146a411e40e7e84622fc9b917493de08a370f7f90d74554c5387c266f935c8 not found: ID does not exist" containerID="b8146a411e40e7e84622fc9b917493de08a370f7f90d74554c5387c266f935c8" Oct 31 13:48:03.437381 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.437300 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8146a411e40e7e84622fc9b917493de08a370f7f90d74554c5387c266f935c8"} err="failed to get container status \"b8146a411e40e7e84622fc9b917493de08a370f7f90d74554c5387c266f935c8\": rpc error: code = NotFound desc = could not find container \"b8146a411e40e7e84622fc9b917493de08a370f7f90d74554c5387c266f935c8\": container with ID starting with b8146a411e40e7e84622fc9b917493de08a370f7f90d74554c5387c266f935c8 not found: ID does not exist" Oct 31 13:48:03.437381 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.437323 6841 scope.go:122] "RemoveContainer" containerID="a4614f7923c3b37da59c81267d051af82067e8080805e2b4d53a42abdeacd0e3" Oct 31 13:48:03.437597 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:48:03.437578 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4614f7923c3b37da59c81267d051af82067e8080805e2b4d53a42abdeacd0e3\": container with ID starting with a4614f7923c3b37da59c81267d051af82067e8080805e2b4d53a42abdeacd0e3 not found: ID does not exist" containerID="a4614f7923c3b37da59c81267d051af82067e8080805e2b4d53a42abdeacd0e3" Oct 31 13:48:03.437643 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.437602 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4614f7923c3b37da59c81267d051af82067e8080805e2b4d53a42abdeacd0e3"} err="failed to get container status \"a4614f7923c3b37da59c81267d051af82067e8080805e2b4d53a42abdeacd0e3\": rpc error: code = NotFound desc = could not find container \"a4614f7923c3b37da59c81267d051af82067e8080805e2b4d53a42abdeacd0e3\": container with ID starting with a4614f7923c3b37da59c81267d051af82067e8080805e2b4d53a42abdeacd0e3 not found: ID does not exist" Oct 31 13:48:03.442682 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.442657 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x"] Oct 31 13:48:03.446679 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.446658 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8504d853-openshift-default-75ccd7d89f-jk69x"] Oct 31 13:48:03.493749 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.493723 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5674e96-0a31-4017-9b03-84eb47f22e0d-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:48:03.493848 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.493752 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv9gz\" (UniqueName: \"kubernetes.io/projected/f5674e96-0a31-4017-9b03-84eb47f22e0d-kube-api-access-cv9gz\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:48:03.493848 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.493773 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5674e96-0a31-4017-9b03-84eb47f22e0d-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:48:03.493848 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.493789 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5674e96-0a31-4017-9b03-84eb47f22e0d-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:48:03.493848 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.493801 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5674e96-0a31-4017-9b03-84eb47f22e0d-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:48:03.493848 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.493814 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5674e96-0a31-4017-9b03-84eb47f22e0d-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:48:03.643486 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:03.643410 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5674e96-0a31-4017-9b03-84eb47f22e0d" path="/var/lib/kubelet/pods/f5674e96-0a31-4017-9b03-84eb47f22e0d/volumes" Oct 31 13:48:04.423474 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:04.423435 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7767cbb97d-f7bp4" event={"ID":"fb07c240-075b-4655-99eb-74d60601c845","Type":"ContainerStarted","Data":"02430da7a826f598ee4aff75d32ea626660fd5c35a26d0d802a2cd86b44aa49a"} Oct 31 13:48:04.423901 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:04.423550 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7767cbb97d-f7bp4" Oct 31 13:48:04.445606 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:04.445559 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7767cbb97d-f7bp4" podStartSLOduration=1.487251984 podStartE2EDuration="3.445546497s" totalImagesPullingTime="1.958294513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:48:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:48:04.444223379 +0000 UTC m=+4530.927346282" watchObservedRunningTime="2026-10-31 13:48:04.445546497 +0000 UTC m=+4530.928669392" Oct 31 13:48:08.709749 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:08.709710 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx"] Oct 31 13:48:08.710336 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:08.710314 6841 cpu_manager.go:414] "containerMap: removing container" podUID="f5674e96-0a31-4017-9b03-84eb47f22e0d" containerName="istio-proxy" Oct 31 13:48:08.710446 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:08.710338 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5674e96-0a31-4017-9b03-84eb47f22e0d" containerName="istio-proxy" Oct 31 13:48:08.710555 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:08.710539 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5674e96-0a31-4017-9b03-84eb47f22e0d" containerName="istio-proxy" Oct 31 13:48:08.710620 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:08.710564 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5674e96-0a31-4017-9b03-84eb47f22e0d" containerName="istio-proxy" Oct 31 13:48:08.714044 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:08.714023 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" Oct 31 13:48:08.732507 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:08.732478 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx"] Oct 31 13:48:08.846034 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:08.845977 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-xn5wx\" (UID: \"2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" Oct 31 13:48:08.846221 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:08.846040 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-xn5wx\" (UID: \"2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" Oct 31 13:48:08.846221 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:08.846064 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-xn5wx\" (UID: \"2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" Oct 31 13:48:08.846221 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:08.846201 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-xn5wx\" (UID: \"2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" Oct 31 13:48:08.846402 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:08.846257 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-xn5wx\" (UID: \"2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" Oct 31 13:48:08.846402 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:08.846282 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbwqc\" (UniqueName: \"kubernetes.io/projected/2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11-kube-api-access-vbwqc\") pod \"maas-default-gateway-openshift-default-fc8c56547-xn5wx\" (UID: \"2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" Oct 31 13:48:08.846402 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:08.846302 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-xn5wx\" (UID: \"2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" Oct 31 13:48:08.846402 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:08.846362 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-xn5wx\" (UID: \"2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" Oct 31 13:48:08.846563 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:08.846414 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-xn5wx\" (UID: \"2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" Oct 31 13:48:08.947342 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:08.947313 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-xn5wx\" (UID: \"2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" Oct 31 13:48:08.947523 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:08.947359 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-xn5wx\" (UID: \"2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" Oct 31 13:48:08.947523 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:08.947392 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-xn5wx\" (UID: \"2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" Oct 31 13:48:08.947523 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:08.947411 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-xn5wx\" (UID: \"2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" Oct 31 13:48:08.947523 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:08.947430 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-xn5wx\" (UID: \"2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" Oct 31 13:48:08.947523 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:08.947494 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-xn5wx\" (UID: \"2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" Oct 31 13:48:08.947523 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:08.947516 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-xn5wx\" (UID: \"2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" Oct 31 13:48:08.947904 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:08.947550 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbwqc\" (UniqueName: \"kubernetes.io/projected/2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11-kube-api-access-vbwqc\") pod \"maas-default-gateway-openshift-default-fc8c56547-xn5wx\" (UID: \"2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" Oct 31 13:48:08.947904 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:08.947586 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-xn5wx\" (UID: \"2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" Oct 31 13:48:08.948064 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:08.947967 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-xn5wx\" (UID: \"2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" Oct 31 13:48:08.948347 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:08.948310 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-xn5wx\" (UID: \"2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" Oct 31 13:48:08.948347 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:08.948315 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-xn5wx\" (UID: \"2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" Oct 31 13:48:08.948509 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:08.948350 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-xn5wx\" (UID: \"2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" Oct 31 13:48:08.948509 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:08.948317 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-xn5wx\" (UID: \"2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" Oct 31 13:48:08.949737 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:08.949712 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-xn5wx\" (UID: \"2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" Oct 31 13:48:08.949994 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:08.949969 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-xn5wx\" (UID: \"2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" Oct 31 13:48:08.961200 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:08.961145 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbwqc\" (UniqueName: \"kubernetes.io/projected/2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11-kube-api-access-vbwqc\") pod \"maas-default-gateway-openshift-default-fc8c56547-xn5wx\" (UID: \"2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" Oct 31 13:48:08.961823 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:08.961806 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-xn5wx\" (UID: \"2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" Oct 31 13:48:09.027226 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:09.027186 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" Oct 31 13:48:09.309608 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:09.309578 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx"] Oct 31 13:48:09.311716 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:48:09.311688 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2977a1b7_e6f5_4ba2_9d69_b0f8a9247d11.slice/crio-80fc52d70456be7367f9ac994947ac8318d9067e79ae1ad675008fcb04b2d80a WatchSource:0}: Error finding container 80fc52d70456be7367f9ac994947ac8318d9067e79ae1ad675008fcb04b2d80a: Status 404 returned error can't find the container with id 80fc52d70456be7367f9ac994947ac8318d9067e79ae1ad675008fcb04b2d80a Oct 31 13:48:09.313871 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:09.313840 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:48:09.313950 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:09.313904 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:48:09.313950 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:09.313932 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:48:09.453821 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:09.453782 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" event={"ID":"2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11","Type":"ContainerStarted","Data":"bb3fb2c4f49b3aef553423064cae2dbd701bed5efc55598a51d2eff332c5043a"} Oct 31 13:48:09.454041 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:09.453848 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" event={"ID":"2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11","Type":"ContainerStarted","Data":"80fc52d70456be7367f9ac994947ac8318d9067e79ae1ad675008fcb04b2d80a"} Oct 31 13:48:09.479994 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:09.479927 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" podStartSLOduration=1.47990743 podStartE2EDuration="1.47990743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:48:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:48:09.477191391 +0000 UTC m=+4535.960314293" watchObservedRunningTime="2026-10-31 13:48:09.47990743 +0000 UTC m=+4535.963030332" Oct 31 13:48:10.027924 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:10.027885 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" Oct 31 13:48:10.029542 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:10.029514 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-xn5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Oct 31 13:48:10.029675 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:10.029562 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" podUID="2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Oct 31 13:48:11.028327 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:11.028291 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-xn5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Oct 31 13:48:11.028710 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:11.028357 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" podUID="2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Oct 31 13:48:12.027650 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:12.027614 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-xn5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Oct 31 13:48:12.027846 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:12.027670 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" podUID="2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Oct 31 13:48:13.027801 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:13.027763 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-xn5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Oct 31 13:48:13.028212 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:13.027816 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" podUID="2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Oct 31 13:48:14.028202 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:14.028162 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-xn5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Oct 31 13:48:14.028618 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:14.028223 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" podUID="2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Oct 31 13:48:15.028162 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:15.028126 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-xn5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Oct 31 13:48:15.028361 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:15.028178 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" podUID="2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Oct 31 13:48:15.432783 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:15.432699 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7767cbb97d-f7bp4" Oct 31 13:48:16.027505 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:16.027469 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-xn5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Oct 31 13:48:16.027722 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:16.027529 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" podUID="2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Oct 31 13:48:17.569364 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:17.569331 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" Oct 31 13:48:17.570051 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:17.570025 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" Oct 31 13:48:17.572136 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:17.572114 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" Oct 31 13:48:20.033857 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:48:20.033821 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:48:22.927541 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:22.927506 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wj64z"] Oct 31 13:48:22.927949 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:22.927936 6841 cpu_manager.go:414] "containerMap: removing container" podUID="f5674e96-0a31-4017-9b03-84eb47f22e0d" containerName="istio-proxy" Oct 31 13:48:22.928024 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:22.927950 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5674e96-0a31-4017-9b03-84eb47f22e0d" containerName="istio-proxy" Oct 31 13:48:22.931296 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:22.931279 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wj64z" Oct 31 13:48:22.943170 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:22.943143 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:48:22.948012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:22.944291 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 13:48:22.948012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:22.945230 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:48:22.948012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:22.945817 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:48:22.948012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:22.947011 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wj64z"] Oct 31 13:48:22.990427 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:22.990394 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47n6s\" (UniqueName: \"kubernetes.io/projected/31c6b47f-a737-4c7e-a4c0-81746942fb89-kube-api-access-47n6s\") pod \"maas-api-revoke-sub-85237e45-wj64z\" (UID: \"31c6b47f-a737-4c7e-a4c0-81746942fb89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wj64z" Oct 31 13:48:22.990602 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:22.990447 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31c6b47f-a737-4c7e-a4c0-81746942fb89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-wj64z\" (UID: \"31c6b47f-a737-4c7e-a4c0-81746942fb89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wj64z" Oct 31 13:48:22.990602 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:22.990526 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31c6b47f-a737-4c7e-a4c0-81746942fb89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-wj64z\" (UID: \"31c6b47f-a737-4c7e-a4c0-81746942fb89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wj64z" Oct 31 13:48:23.091951 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:23.091915 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31c6b47f-a737-4c7e-a4c0-81746942fb89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-wj64z\" (UID: \"31c6b47f-a737-4c7e-a4c0-81746942fb89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wj64z" Oct 31 13:48:23.091951 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:23.091955 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31c6b47f-a737-4c7e-a4c0-81746942fb89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-wj64z\" (UID: \"31c6b47f-a737-4c7e-a4c0-81746942fb89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wj64z" Oct 31 13:48:23.092214 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:23.092081 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47n6s\" (UniqueName: \"kubernetes.io/projected/31c6b47f-a737-4c7e-a4c0-81746942fb89-kube-api-access-47n6s\") pod \"maas-api-revoke-sub-85237e45-wj64z\" (UID: \"31c6b47f-a737-4c7e-a4c0-81746942fb89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wj64z" Oct 31 13:48:23.092615 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:23.092593 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31c6b47f-a737-4c7e-a4c0-81746942fb89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-wj64z\" (UID: \"31c6b47f-a737-4c7e-a4c0-81746942fb89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wj64z" Oct 31 13:48:23.094324 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:23.094299 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31c6b47f-a737-4c7e-a4c0-81746942fb89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-wj64z\" (UID: \"31c6b47f-a737-4c7e-a4c0-81746942fb89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wj64z" Oct 31 13:48:23.124056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:23.124031 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47n6s\" (UniqueName: \"kubernetes.io/projected/31c6b47f-a737-4c7e-a4c0-81746942fb89-kube-api-access-47n6s\") pod \"maas-api-revoke-sub-85237e45-wj64z\" (UID: \"31c6b47f-a737-4c7e-a4c0-81746942fb89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wj64z" Oct 31 13:48:23.248630 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:23.248598 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wj64z" Oct 31 13:48:23.539811 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:23.539786 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wj64z"] Oct 31 13:48:23.542277 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:48:23.542234 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31c6b47f_a737_4c7e_a4c0_81746942fb89.slice/crio-b72ff0d3d5e61f600faea9698efed4b13494a47c70c6a5caeb944e7f7fe37d20 WatchSource:0}: Error finding container b72ff0d3d5e61f600faea9698efed4b13494a47c70c6a5caeb944e7f7fe37d20: Status 404 returned error can't find the container with id b72ff0d3d5e61f600faea9698efed4b13494a47c70c6a5caeb944e7f7fe37d20 Oct 31 13:48:23.792785 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:23.792714 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4"] Oct 31 13:48:23.797873 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:23.797852 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" Oct 31 13:48:23.822944 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:23.822916 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4"] Oct 31 13:48:23.899599 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:23.899565 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-bp2d4\" (UID: \"51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" Oct 31 13:48:23.899728 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:23.899629 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-bp2d4\" (UID: \"51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" Oct 31 13:48:23.899728 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:23.899702 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-bp2d4\" (UID: \"51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" Oct 31 13:48:23.899802 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:23.899734 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-bp2d4\" (UID: \"51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" Oct 31 13:48:23.899842 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:23.899814 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-bp2d4\" (UID: \"51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" Oct 31 13:48:23.899881 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:23.899850 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-bp2d4\" (UID: \"51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" Oct 31 13:48:23.899881 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:23.899875 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-bp2d4\" (UID: \"51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" Oct 31 13:48:23.899948 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:23.899897 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhxhz\" (UniqueName: \"kubernetes.io/projected/51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25-kube-api-access-rhxhz\") pod \"maas-default-gateway-openshift-default-fc8c56547-bp2d4\" (UID: \"51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" Oct 31 13:48:23.900014 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:23.899948 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-bp2d4\" (UID: \"51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" Oct 31 13:48:24.001192 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:24.001160 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-bp2d4\" (UID: \"51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" Oct 31 13:48:24.001192 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:24.001195 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-bp2d4\" (UID: \"51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" Oct 31 13:48:24.001760 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:24.001238 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-bp2d4\" (UID: \"51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" Oct 31 13:48:24.001760 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:24.001274 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-bp2d4\" (UID: \"51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" Oct 31 13:48:24.001760 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:24.001303 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-bp2d4\" (UID: \"51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" Oct 31 13:48:24.001760 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:24.001333 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhxhz\" (UniqueName: \"kubernetes.io/projected/51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25-kube-api-access-rhxhz\") pod \"maas-default-gateway-openshift-default-fc8c56547-bp2d4\" (UID: \"51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" Oct 31 13:48:24.001760 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:24.001362 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-bp2d4\" (UID: \"51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" Oct 31 13:48:24.001760 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:24.001416 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-bp2d4\" (UID: \"51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" Oct 31 13:48:24.001760 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:24.001489 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-bp2d4\" (UID: \"51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" Oct 31 13:48:24.001760 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:24.001757 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-bp2d4\" (UID: \"51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" Oct 31 13:48:24.002308 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:24.001807 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-bp2d4\" (UID: \"51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" Oct 31 13:48:24.002308 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:24.001915 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-bp2d4\" (UID: \"51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" Oct 31 13:48:24.002308 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:24.002008 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-bp2d4\" (UID: \"51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" Oct 31 13:48:24.002308 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:24.002067 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-bp2d4\" (UID: \"51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" Oct 31 13:48:24.003681 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:24.003656 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-bp2d4\" (UID: \"51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" Oct 31 13:48:24.003850 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:24.003824 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-bp2d4\" (UID: \"51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" Oct 31 13:48:24.015550 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:24.015526 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhxhz\" (UniqueName: \"kubernetes.io/projected/51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25-kube-api-access-rhxhz\") pod \"maas-default-gateway-openshift-default-fc8c56547-bp2d4\" (UID: \"51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" Oct 31 13:48:24.015932 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:24.015910 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-bp2d4\" (UID: \"51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" Oct 31 13:48:24.111565 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:24.111495 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" Oct 31 13:48:24.393495 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:24.393459 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4"] Oct 31 13:48:24.395344 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:48:24.395314 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a3b5f8_b1f5_46a4_8784_d1e00e9e3e25.slice/crio-670a1b2577a5363cbbcca3191f9bafef8557d9a87160ae6f94af8dc655867b7b WatchSource:0}: Error finding container 670a1b2577a5363cbbcca3191f9bafef8557d9a87160ae6f94af8dc655867b7b: Status 404 returned error can't find the container with id 670a1b2577a5363cbbcca3191f9bafef8557d9a87160ae6f94af8dc655867b7b Oct 31 13:48:24.397681 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:24.397647 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:48:24.397785 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:24.397711 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:48:24.397785 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:24.397741 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:48:24.555968 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:24.555930 6841 generic.go:356] "Generic (PLEG): container finished" podID="31c6b47f-a737-4c7e-a4c0-81746942fb89" containerID="a7abd4228da7fc8f8882c3fe8ef9d1b78a096dc62b002a0e996630d647045480" exitCode=0 Oct 31 13:48:24.556176 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:24.556037 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wj64z" event={"ID":"31c6b47f-a737-4c7e-a4c0-81746942fb89","Type":"ContainerDied","Data":"a7abd4228da7fc8f8882c3fe8ef9d1b78a096dc62b002a0e996630d647045480"} Oct 31 13:48:24.556176 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:24.556101 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wj64z" event={"ID":"31c6b47f-a737-4c7e-a4c0-81746942fb89","Type":"ContainerStarted","Data":"b72ff0d3d5e61f600faea9698efed4b13494a47c70c6a5caeb944e7f7fe37d20"} Oct 31 13:48:24.558026 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:24.557999 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" event={"ID":"51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25","Type":"ContainerStarted","Data":"45d065e440111e6b740e5c117a6387529d3a6730835961657ea508a329f17eb5"} Oct 31 13:48:24.558156 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:24.558054 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" event={"ID":"51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25","Type":"ContainerStarted","Data":"670a1b2577a5363cbbcca3191f9bafef8557d9a87160ae6f94af8dc655867b7b"} Oct 31 13:48:25.112047 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:25.112006 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" Oct 31 13:48:25.113446 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:25.113422 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-bp2d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Oct 31 13:48:25.113526 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:25.113472 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" podUID="51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Oct 31 13:48:25.729618 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:25.729593 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wj64z" Oct 31 13:48:25.765006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:25.759353 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" podStartSLOduration=2.75933367 podStartE2EDuration="2.75933367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:48:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:48:24.601829511 +0000 UTC m=+4551.084952453" watchObservedRunningTime="2026-10-31 13:48:25.75933367 +0000 UTC m=+4552.242456571" Oct 31 13:48:25.827101 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:25.827015 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31c6b47f-a737-4c7e-a4c0-81746942fb89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31c6b47f-a737-4c7e-a4c0-81746942fb89-maas-api-xks-ca\") pod \"31c6b47f-a737-4c7e-a4c0-81746942fb89\" (UID: \"31c6b47f-a737-4c7e-a4c0-81746942fb89\") " Oct 31 13:48:25.827288 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:25.827112 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31c6b47f-a737-4c7e-a4c0-81746942fb89-kube-api-access-47n6s\" (UniqueName: \"kubernetes.io/projected/31c6b47f-a737-4c7e-a4c0-81746942fb89-kube-api-access-47n6s\") pod \"31c6b47f-a737-4c7e-a4c0-81746942fb89\" (UID: \"31c6b47f-a737-4c7e-a4c0-81746942fb89\") " Oct 31 13:48:25.827288 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:25.827136 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31c6b47f-a737-4c7e-a4c0-81746942fb89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31c6b47f-a737-4c7e-a4c0-81746942fb89-maas-api-openshift-ca\") pod \"31c6b47f-a737-4c7e-a4c0-81746942fb89\" (UID: \"31c6b47f-a737-4c7e-a4c0-81746942fb89\") " Oct 31 13:48:25.827553 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:25.827525 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31c6b47f-a737-4c7e-a4c0-81746942fb89-maas-api-openshift-ca" pod "31c6b47f-a737-4c7e-a4c0-81746942fb89" (UID: "31c6b47f-a737-4c7e-a4c0-81746942fb89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:48:25.829195 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:25.829164 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31c6b47f-a737-4c7e-a4c0-81746942fb89-maas-api-xks-ca" pod "31c6b47f-a737-4c7e-a4c0-81746942fb89" (UID: "31c6b47f-a737-4c7e-a4c0-81746942fb89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:48:25.829280 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:25.829238 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31c6b47f-a737-4c7e-a4c0-81746942fb89-kube-api-access-47n6s" pod "31c6b47f-a737-4c7e-a4c0-81746942fb89" (UID: "31c6b47f-a737-4c7e-a4c0-81746942fb89"). InnerVolumeSpecName "kube-api-access-47n6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:48:25.929027 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:25.928976 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31c6b47f-a737-4c7e-a4c0-81746942fb89-maas-api-xks-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:48:25.929027 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:25.929021 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47n6s\" (UniqueName: \"kubernetes.io/projected/31c6b47f-a737-4c7e-a4c0-81746942fb89-kube-api-access-47n6s\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:48:25.929027 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:25.929034 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31c6b47f-a737-4c7e-a4c0-81746942fb89-maas-api-openshift-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:48:26.112899 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:26.112794 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-bp2d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Oct 31 13:48:26.112899 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:26.112863 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" podUID="51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Oct 31 13:48:26.582680 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:26.582643 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wj64z" Oct 31 13:48:26.582680 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:26.582664 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wj64z" event={"ID":"31c6b47f-a737-4c7e-a4c0-81746942fb89","Type":"ContainerDied","Data":"b72ff0d3d5e61f600faea9698efed4b13494a47c70c6a5caeb944e7f7fe37d20"} Oct 31 13:48:26.582922 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:26.582728 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b72ff0d3d5e61f600faea9698efed4b13494a47c70c6a5caeb944e7f7fe37d20" Oct 31 13:48:27.112762 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:27.112723 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-bp2d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Oct 31 13:48:27.113025 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:27.112784 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" podUID="51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Oct 31 13:48:28.112296 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:28.112260 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-bp2d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Oct 31 13:48:28.112490 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:28.112313 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" podUID="51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Oct 31 13:48:29.112876 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:29.112838 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-bp2d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Oct 31 13:48:29.113305 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:29.112902 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" podUID="51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Oct 31 13:48:29.237287 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:29.237249 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:48:29.498367 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:29.498325 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:48:30.112921 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:30.112872 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-bp2d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Oct 31 13:48:30.113372 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:30.112941 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" podUID="51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Oct 31 13:48:31.112545 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:31.112501 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-bp2d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Oct 31 13:48:31.112785 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:31.112557 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" podUID="51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Oct 31 13:48:32.814917 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:32.814882 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" Oct 31 13:48:32.815448 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:32.815277 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" Oct 31 13:48:32.816139 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:32.816115 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" Oct 31 13:48:34.620950 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:34.620908 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:48:34.637994 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:34.637948 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7767cbb97d-f7bp4"] Oct 31 13:48:34.638175 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:34.638135 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7767cbb97d-f7bp4" podUID="fb07c240-075b-4655-99eb-74d60601c845" containerName="manager" containerID="cri-o://02430da7a826f598ee4aff75d32ea626660fd5c35a26d0d802a2cd86b44aa49a" gracePeriod=30 Oct 31 13:48:34.892469 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:34.892446 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7767cbb97d-f7bp4" Oct 31 13:48:35.041748 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:35.041717 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb07c240-075b-4655-99eb-74d60601c845-webhook-cert\" (UniqueName: \"kubernetes.io/secret/fb07c240-075b-4655-99eb-74d60601c845-webhook-cert\") pod \"fb07c240-075b-4655-99eb-74d60601c845\" (UID: \"fb07c240-075b-4655-99eb-74d60601c845\") " Oct 31 13:48:35.041748 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:35.041754 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb07c240-075b-4655-99eb-74d60601c845-kube-api-access-vbb74\" (UniqueName: \"kubernetes.io/projected/fb07c240-075b-4655-99eb-74d60601c845-kube-api-access-vbb74\") pod \"fb07c240-075b-4655-99eb-74d60601c845\" (UID: \"fb07c240-075b-4655-99eb-74d60601c845\") " Oct 31 13:48:35.042128 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:35.041778 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb07c240-075b-4655-99eb-74d60601c845-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb07c240-075b-4655-99eb-74d60601c845-metrics-certs\") pod \"fb07c240-075b-4655-99eb-74d60601c845\" (UID: \"fb07c240-075b-4655-99eb-74d60601c845\") " Oct 31 13:48:35.043974 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:35.043939 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb07c240-075b-4655-99eb-74d60601c845-webhook-cert" pod "fb07c240-075b-4655-99eb-74d60601c845" (UID: "fb07c240-075b-4655-99eb-74d60601c845"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:48:35.043974 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:35.043974 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb07c240-075b-4655-99eb-74d60601c845-metrics-certs" pod "fb07c240-075b-4655-99eb-74d60601c845" (UID: "fb07c240-075b-4655-99eb-74d60601c845"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:48:35.044159 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:35.044027 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb07c240-075b-4655-99eb-74d60601c845-kube-api-access-vbb74" pod "fb07c240-075b-4655-99eb-74d60601c845" (UID: "fb07c240-075b-4655-99eb-74d60601c845"). InnerVolumeSpecName "kube-api-access-vbb74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:48:35.142893 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:35.142816 6841 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fb07c240-075b-4655-99eb-74d60601c845-webhook-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:48:35.142893 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:35.142844 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbb74\" (UniqueName: \"kubernetes.io/projected/fb07c240-075b-4655-99eb-74d60601c845-kube-api-access-vbb74\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:48:35.142893 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:35.142855 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb07c240-075b-4655-99eb-74d60601c845-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:48:35.632761 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:35.632725 6841 generic.go:356] "Generic (PLEG): container finished" podID="fb07c240-075b-4655-99eb-74d60601c845" containerID="02430da7a826f598ee4aff75d32ea626660fd5c35a26d0d802a2cd86b44aa49a" exitCode=0 Oct 31 13:48:35.633220 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:35.632770 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7767cbb97d-f7bp4" Oct 31 13:48:35.633220 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:35.632807 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7767cbb97d-f7bp4" event={"ID":"fb07c240-075b-4655-99eb-74d60601c845","Type":"ContainerDied","Data":"02430da7a826f598ee4aff75d32ea626660fd5c35a26d0d802a2cd86b44aa49a"} Oct 31 13:48:35.633220 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:35.632864 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7767cbb97d-f7bp4" event={"ID":"fb07c240-075b-4655-99eb-74d60601c845","Type":"ContainerDied","Data":"282fea84090e340add06a0a65c49ea5719ff72530983112fc12ed25a81067814"} Oct 31 13:48:35.633220 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:35.632889 6841 scope.go:122] "RemoveContainer" containerID="02430da7a826f598ee4aff75d32ea626660fd5c35a26d0d802a2cd86b44aa49a" Oct 31 13:48:35.644605 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:35.644411 6841 scope.go:122] "RemoveContainer" containerID="02430da7a826f598ee4aff75d32ea626660fd5c35a26d0d802a2cd86b44aa49a" Oct 31 13:48:35.644694 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:48:35.644649 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02430da7a826f598ee4aff75d32ea626660fd5c35a26d0d802a2cd86b44aa49a\": container with ID starting with 02430da7a826f598ee4aff75d32ea626660fd5c35a26d0d802a2cd86b44aa49a not found: ID does not exist" containerID="02430da7a826f598ee4aff75d32ea626660fd5c35a26d0d802a2cd86b44aa49a" Oct 31 13:48:35.644741 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:35.644682 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02430da7a826f598ee4aff75d32ea626660fd5c35a26d0d802a2cd86b44aa49a"} err="failed to get container status \"02430da7a826f598ee4aff75d32ea626660fd5c35a26d0d802a2cd86b44aa49a\": rpc error: code = NotFound desc = could not find container \"02430da7a826f598ee4aff75d32ea626660fd5c35a26d0d802a2cd86b44aa49a\": container with ID starting with 02430da7a826f598ee4aff75d32ea626660fd5c35a26d0d802a2cd86b44aa49a not found: ID does not exist" Oct 31 13:48:35.659311 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:35.659275 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7767cbb97d-f7bp4"] Oct 31 13:48:35.663547 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:35.663528 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7767cbb97d-f7bp4"] Oct 31 13:48:36.834163 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:36.834121 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:48:36.883190 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:36.883155 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7767cbb97d-nhclq"] Oct 31 13:48:36.883698 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:36.883678 6841 cpu_manager.go:414] "containerMap: removing container" podUID="fb07c240-075b-4655-99eb-74d60601c845" containerName="manager" Oct 31 13:48:36.883807 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:36.883698 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb07c240-075b-4655-99eb-74d60601c845" containerName="manager" Oct 31 13:48:36.883807 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:36.883726 6841 cpu_manager.go:414] "containerMap: removing container" podUID="31c6b47f-a737-4c7e-a4c0-81746942fb89" containerName="revoke-keys" Oct 31 13:48:36.883807 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:36.883734 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31c6b47f-a737-4c7e-a4c0-81746942fb89" containerName="revoke-keys" Oct 31 13:48:36.884078 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:36.883848 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb07c240-075b-4655-99eb-74d60601c845" containerName="manager" Oct 31 13:48:36.884078 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:36.883865 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="31c6b47f-a737-4c7e-a4c0-81746942fb89" containerName="revoke-keys" Oct 31 13:48:36.889634 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:36.889613 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7767cbb97d-nhclq" Oct 31 13:48:36.896364 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:36.896337 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 13:48:36.897377 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:36.897357 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 13:48:36.897504 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:36.897458 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 13:48:36.897582 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:36.897506 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jnfxm\"" Oct 31 13:48:36.900186 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:36.900165 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7767cbb97d-nhclq"] Oct 31 13:48:37.061147 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:37.061106 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a155191a-e6bc-4cb4-b471-a68a27b1eb5f-metrics-certs\") pod \"maas-controller-7767cbb97d-nhclq\" (UID: \"a155191a-e6bc-4cb4-b471-a68a27b1eb5f\") " pod="opendatahub/maas-controller-7767cbb97d-nhclq" Oct 31 13:48:37.061352 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:37.061180 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww4d9\" (UniqueName: \"kubernetes.io/projected/a155191a-e6bc-4cb4-b471-a68a27b1eb5f-kube-api-access-ww4d9\") pod \"maas-controller-7767cbb97d-nhclq\" (UID: \"a155191a-e6bc-4cb4-b471-a68a27b1eb5f\") " pod="opendatahub/maas-controller-7767cbb97d-nhclq" Oct 31 13:48:37.061352 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:37.061227 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a155191a-e6bc-4cb4-b471-a68a27b1eb5f-webhook-cert\") pod \"maas-controller-7767cbb97d-nhclq\" (UID: \"a155191a-e6bc-4cb4-b471-a68a27b1eb5f\") " pod="opendatahub/maas-controller-7767cbb97d-nhclq" Oct 31 13:48:37.162099 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:37.161975 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a155191a-e6bc-4cb4-b471-a68a27b1eb5f-metrics-certs\") pod \"maas-controller-7767cbb97d-nhclq\" (UID: \"a155191a-e6bc-4cb4-b471-a68a27b1eb5f\") " pod="opendatahub/maas-controller-7767cbb97d-nhclq" Oct 31 13:48:37.162099 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:37.162073 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww4d9\" (UniqueName: \"kubernetes.io/projected/a155191a-e6bc-4cb4-b471-a68a27b1eb5f-kube-api-access-ww4d9\") pod \"maas-controller-7767cbb97d-nhclq\" (UID: \"a155191a-e6bc-4cb4-b471-a68a27b1eb5f\") " pod="opendatahub/maas-controller-7767cbb97d-nhclq" Oct 31 13:48:37.162358 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:37.162108 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a155191a-e6bc-4cb4-b471-a68a27b1eb5f-webhook-cert\") pod \"maas-controller-7767cbb97d-nhclq\" (UID: \"a155191a-e6bc-4cb4-b471-a68a27b1eb5f\") " pod="opendatahub/maas-controller-7767cbb97d-nhclq" Oct 31 13:48:37.164634 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:37.164606 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a155191a-e6bc-4cb4-b471-a68a27b1eb5f-webhook-cert\") pod \"maas-controller-7767cbb97d-nhclq\" (UID: \"a155191a-e6bc-4cb4-b471-a68a27b1eb5f\") " pod="opendatahub/maas-controller-7767cbb97d-nhclq" Oct 31 13:48:37.164770 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:37.164745 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a155191a-e6bc-4cb4-b471-a68a27b1eb5f-metrics-certs\") pod \"maas-controller-7767cbb97d-nhclq\" (UID: \"a155191a-e6bc-4cb4-b471-a68a27b1eb5f\") " pod="opendatahub/maas-controller-7767cbb97d-nhclq" Oct 31 13:48:37.175366 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:37.175341 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww4d9\" (UniqueName: \"kubernetes.io/projected/a155191a-e6bc-4cb4-b471-a68a27b1eb5f-kube-api-access-ww4d9\") pod \"maas-controller-7767cbb97d-nhclq\" (UID: \"a155191a-e6bc-4cb4-b471-a68a27b1eb5f\") " pod="opendatahub/maas-controller-7767cbb97d-nhclq" Oct 31 13:48:37.206366 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:37.206342 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7767cbb97d-nhclq" Oct 31 13:48:37.486343 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:37.486316 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7767cbb97d-nhclq"] Oct 31 13:48:37.488399 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:48:37.488369 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda155191a_e6bc_4cb4_b471_a68a27b1eb5f.slice/crio-452529324fb2b5d0e6c5fa029a843eb6b55b1acaf57715aa849b34056b1d697e WatchSource:0}: Error finding container 452529324fb2b5d0e6c5fa029a843eb6b55b1acaf57715aa849b34056b1d697e: Status 404 returned error can't find the container with id 452529324fb2b5d0e6c5fa029a843eb6b55b1acaf57715aa849b34056b1d697e Oct 31 13:48:37.641278 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:37.641243 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb07c240-075b-4655-99eb-74d60601c845" path="/var/lib/kubelet/pods/fb07c240-075b-4655-99eb-74d60601c845/volumes" Oct 31 13:48:37.644449 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:37.644419 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7767cbb97d-nhclq" event={"ID":"a155191a-e6bc-4cb4-b471-a68a27b1eb5f","Type":"ContainerStarted","Data":"452529324fb2b5d0e6c5fa029a843eb6b55b1acaf57715aa849b34056b1d697e"} Oct 31 13:48:38.650929 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:38.650890 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7767cbb97d-nhclq" event={"ID":"a155191a-e6bc-4cb4-b471-a68a27b1eb5f","Type":"ContainerStarted","Data":"269495f97c1a7579c33c16760ac577e5348f834b602c7874074b4ac303c62b40"} Oct 31 13:48:38.651398 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:38.651116 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7767cbb97d-nhclq" Oct 31 13:48:38.674932 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:38.674871 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7767cbb97d-nhclq" podStartSLOduration=2.283210559 podStartE2EDuration="2.674858806s" totalImagesPullingTime="391.648247ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:48:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:48:38.671963227 +0000 UTC m=+4565.155086129" watchObservedRunningTime="2026-10-31 13:48:38.674858806 +0000 UTC m=+4565.157981698" Oct 31 13:48:41.318008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:41.317962 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:48:43.028188 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:43.028150 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:48:49.658369 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:48:49.658337 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7767cbb97d-nhclq" Oct 31 13:48:50.034244 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:48:50.034213 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:49:00.608908 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:00.608857 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:49:00.813343 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:00.813308 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:49:01.519068 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.519030 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-tk8j6"] Oct 31 13:49:01.522864 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.522842 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-tk8j6" Oct 31 13:49:01.532213 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.532183 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:49:01.532393 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.532371 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 13:49:01.532496 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.532227 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:49:01.532496 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.532219 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:49:01.534363 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.534339 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-tk8j6"] Oct 31 13:49:01.547600 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.547570 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxcw2\" (UniqueName: \"kubernetes.io/projected/54802583-fbdc-48a9-90de-262d5aecb544-kube-api-access-dxcw2\") pod \"maas-api-revoke-sub-f904bff9-tk8j6\" (UID: \"54802583-fbdc-48a9-90de-262d5aecb544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-tk8j6" Oct 31 13:49:01.547748 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.547637 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54802583-fbdc-48a9-90de-262d5aecb544-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-tk8j6\" (UID: \"54802583-fbdc-48a9-90de-262d5aecb544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-tk8j6" Oct 31 13:49:01.547748 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.547712 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54802583-fbdc-48a9-90de-262d5aecb544-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-tk8j6\" (UID: \"54802583-fbdc-48a9-90de-262d5aecb544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-tk8j6" Oct 31 13:49:01.560598 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.560564 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl"] Oct 31 13:49:01.560828 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.560778 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl" podUID="c408c29a-b6d8-4b6d-96a3-a6c7c65c729a" containerName="discovery" containerID="cri-o://ec47d83b2a8d004a41e2bcac35dd09bd59247e6910a753825e12c6ae207f5632" gracePeriod=30 Oct 31 13:49:01.649180 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.649147 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxcw2\" (UniqueName: \"kubernetes.io/projected/54802583-fbdc-48a9-90de-262d5aecb544-kube-api-access-dxcw2\") pod \"maas-api-revoke-sub-f904bff9-tk8j6\" (UID: \"54802583-fbdc-48a9-90de-262d5aecb544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-tk8j6" Oct 31 13:49:01.649561 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.649196 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54802583-fbdc-48a9-90de-262d5aecb544-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-tk8j6\" (UID: \"54802583-fbdc-48a9-90de-262d5aecb544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-tk8j6" Oct 31 13:49:01.649561 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.649247 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54802583-fbdc-48a9-90de-262d5aecb544-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-tk8j6\" (UID: \"54802583-fbdc-48a9-90de-262d5aecb544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-tk8j6" Oct 31 13:49:01.649836 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.649807 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54802583-fbdc-48a9-90de-262d5aecb544-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-tk8j6\" (UID: \"54802583-fbdc-48a9-90de-262d5aecb544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-tk8j6" Oct 31 13:49:01.651590 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.651570 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54802583-fbdc-48a9-90de-262d5aecb544-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-tk8j6\" (UID: \"54802583-fbdc-48a9-90de-262d5aecb544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-tk8j6" Oct 31 13:49:01.662138 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.662116 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxcw2\" (UniqueName: \"kubernetes.io/projected/54802583-fbdc-48a9-90de-262d5aecb544-kube-api-access-dxcw2\") pod \"maas-api-revoke-sub-f904bff9-tk8j6\" (UID: \"54802583-fbdc-48a9-90de-262d5aecb544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-tk8j6" Oct 31 13:49:01.800299 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.800265 6841 generic.go:356] "Generic (PLEG): container finished" podID="c408c29a-b6d8-4b6d-96a3-a6c7c65c729a" containerID="ec47d83b2a8d004a41e2bcac35dd09bd59247e6910a753825e12c6ae207f5632" exitCode=0 Oct 31 13:49:01.800525 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.800350 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl" event={"ID":"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a","Type":"ContainerDied","Data":"ec47d83b2a8d004a41e2bcac35dd09bd59247e6910a753825e12c6ae207f5632"} Oct 31 13:49:01.817246 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.817223 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl" Oct 31 13:49:01.837922 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.837898 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-tk8j6" Oct 31 13:49:01.851806 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.851784 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-local-certs\") pod \"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a\" (UID: \"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a\") " Oct 31 13:49:01.851962 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.851835 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-istio-token\" (UniqueName: \"kubernetes.io/projected/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-istio-token\") pod \"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a\" (UID: \"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a\") " Oct 31 13:49:01.851962 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.851871 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-kube-api-access-xv7p8\" (UniqueName: \"kubernetes.io/projected/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-kube-api-access-xv7p8\") pod \"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a\" (UID: \"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a\") " Oct 31 13:49:01.852116 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.851971 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-istio-csr-dns-cert\") pod \"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a\" (UID: \"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a\") " Oct 31 13:49:01.852116 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.852096 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-istio-kubeconfig\") pod \"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a\" (UID: \"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a\") " Oct 31 13:49:01.852225 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.852133 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-cacerts\" (UniqueName: \"kubernetes.io/secret/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-cacerts\") pod \"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a\" (UID: \"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a\") " Oct 31 13:49:01.852225 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.852199 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-istio-csr-ca-configmap\") pod \"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a\" (UID: \"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a\") " Oct 31 13:49:01.852805 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.852773 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-istio-csr-ca-configmap" pod "c408c29a-b6d8-4b6d-96a3-a6c7c65c729a" (UID: "c408c29a-b6d8-4b6d-96a3-a6c7c65c729a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:49:01.854805 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.854776 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-local-certs" pod "c408c29a-b6d8-4b6d-96a3-a6c7c65c729a" (UID: "c408c29a-b6d8-4b6d-96a3-a6c7c65c729a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:49:01.854805 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.854784 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-istio-csr-dns-cert" pod "c408c29a-b6d8-4b6d-96a3-a6c7c65c729a" (UID: "c408c29a-b6d8-4b6d-96a3-a6c7c65c729a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:49:01.855267 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.855239 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-istio-kubeconfig" pod "c408c29a-b6d8-4b6d-96a3-a6c7c65c729a" (UID: "c408c29a-b6d8-4b6d-96a3-a6c7c65c729a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:49:01.857125 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.857106 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-istio-token" pod "c408c29a-b6d8-4b6d-96a3-a6c7c65c729a" (UID: "c408c29a-b6d8-4b6d-96a3-a6c7c65c729a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:49:01.857285 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.857265 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-kube-api-access-xv7p8" pod "c408c29a-b6d8-4b6d-96a3-a6c7c65c729a" (UID: "c408c29a-b6d8-4b6d-96a3-a6c7c65c729a"). InnerVolumeSpecName "kube-api-access-xv7p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:49:01.857376 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.857332 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-cacerts" pod "c408c29a-b6d8-4b6d-96a3-a6c7c65c729a" (UID: "c408c29a-b6d8-4b6d-96a3-a6c7c65c729a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:49:01.953924 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.953892 6841 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-istio-csr-dns-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:49:01.953924 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.953922 6841 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-istio-kubeconfig\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:49:01.954140 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.953933 6841 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-cacerts\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:49:01.954140 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.953944 6841 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-istio-csr-ca-configmap\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:49:01.954140 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.953953 6841 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-local-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:49:01.954140 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.953961 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:49:01.954140 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:01.953970 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv7p8\" (UniqueName: \"kubernetes.io/projected/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a-kube-api-access-xv7p8\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:49:02.116883 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:02.116850 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-tk8j6"] Oct 31 13:49:02.118430 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:49:02.118406 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54802583_fbdc_48a9_90de_262d5aecb544.slice/crio-443d1d14f7c70d4a62f70649144b69758eecdf2e63e37b260c426936f41ffd61 WatchSource:0}: Error finding container 443d1d14f7c70d4a62f70649144b69758eecdf2e63e37b260c426936f41ffd61: Status 404 returned error can't find the container with id 443d1d14f7c70d4a62f70649144b69758eecdf2e63e37b260c426936f41ffd61 Oct 31 13:49:02.807633 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:02.807551 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl" Oct 31 13:49:02.807633 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:02.807581 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl" event={"ID":"c408c29a-b6d8-4b6d-96a3-a6c7c65c729a","Type":"ContainerDied","Data":"ac5658f098330d046a6097237d6cad8d4c9fc69c18760e5a20f79899a8bf7bbf"} Oct 31 13:49:02.808168 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:02.807645 6841 scope.go:122] "RemoveContainer" containerID="ec47d83b2a8d004a41e2bcac35dd09bd59247e6910a753825e12c6ae207f5632" Oct 31 13:49:02.809293 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:02.809271 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-tk8j6" event={"ID":"54802583-fbdc-48a9-90de-262d5aecb544","Type":"ContainerStarted","Data":"bed11a3c1258f98e70f2a968df7febe322fc01e66213922bb07f2cdcf80d2d5d"} Oct 31 13:49:02.809434 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:02.809308 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-tk8j6" event={"ID":"54802583-fbdc-48a9-90de-262d5aecb544","Type":"ContainerStarted","Data":"443d1d14f7c70d4a62f70649144b69758eecdf2e63e37b260c426936f41ffd61"} Oct 31 13:49:02.831567 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:02.831525 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-tk8j6" podStartSLOduration=1.831513602 podStartE2EDuration="1.831513602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:49:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:49:02.829175133 +0000 UTC m=+4589.312298046" watchObservedRunningTime="2026-10-31 13:49:02.831513602 +0000 UTC m=+4589.314636502" Oct 31 13:49:02.843868 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:02.843845 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl"] Oct 31 13:49:02.847997 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:02.847960 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r5tbl"] Oct 31 13:49:03.641589 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:03.641555 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c408c29a-b6d8-4b6d-96a3-a6c7c65c729a" path="/var/lib/kubelet/pods/c408c29a-b6d8-4b6d-96a3-a6c7c65c729a/volumes" Oct 31 13:49:03.817831 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:03.817783 6841 generic.go:356] "Generic (PLEG): container finished" podID="54802583-fbdc-48a9-90de-262d5aecb544" containerID="bed11a3c1258f98e70f2a968df7febe322fc01e66213922bb07f2cdcf80d2d5d" exitCode=0 Oct 31 13:49:03.818264 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:03.817826 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-tk8j6" event={"ID":"54802583-fbdc-48a9-90de-262d5aecb544","Type":"ContainerDied","Data":"bed11a3c1258f98e70f2a968df7febe322fc01e66213922bb07f2cdcf80d2d5d"} Oct 31 13:49:04.970135 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:04.970109 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-tk8j6" Oct 31 13:49:05.084930 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:05.084844 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54802583-fbdc-48a9-90de-262d5aecb544-kube-api-access-dxcw2\" (UniqueName: \"kubernetes.io/projected/54802583-fbdc-48a9-90de-262d5aecb544-kube-api-access-dxcw2\") pod \"54802583-fbdc-48a9-90de-262d5aecb544\" (UID: \"54802583-fbdc-48a9-90de-262d5aecb544\") " Oct 31 13:49:05.085122 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:05.084975 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54802583-fbdc-48a9-90de-262d5aecb544-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54802583-fbdc-48a9-90de-262d5aecb544-maas-api-xks-ca\") pod \"54802583-fbdc-48a9-90de-262d5aecb544\" (UID: \"54802583-fbdc-48a9-90de-262d5aecb544\") " Oct 31 13:49:05.085122 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:05.085026 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54802583-fbdc-48a9-90de-262d5aecb544-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54802583-fbdc-48a9-90de-262d5aecb544-maas-api-openshift-ca\") pod \"54802583-fbdc-48a9-90de-262d5aecb544\" (UID: \"54802583-fbdc-48a9-90de-262d5aecb544\") " Oct 31 13:49:05.085433 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:05.085410 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54802583-fbdc-48a9-90de-262d5aecb544-maas-api-openshift-ca" pod "54802583-fbdc-48a9-90de-262d5aecb544" (UID: "54802583-fbdc-48a9-90de-262d5aecb544"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:49:05.086977 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:05.086946 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54802583-fbdc-48a9-90de-262d5aecb544-kube-api-access-dxcw2" pod "54802583-fbdc-48a9-90de-262d5aecb544" (UID: "54802583-fbdc-48a9-90de-262d5aecb544"). InnerVolumeSpecName "kube-api-access-dxcw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:49:05.087110 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:05.087010 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54802583-fbdc-48a9-90de-262d5aecb544-maas-api-xks-ca" pod "54802583-fbdc-48a9-90de-262d5aecb544" (UID: "54802583-fbdc-48a9-90de-262d5aecb544"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:49:05.186134 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:05.186096 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54802583-fbdc-48a9-90de-262d5aecb544-maas-api-xks-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:49:05.186300 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:05.186138 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54802583-fbdc-48a9-90de-262d5aecb544-maas-api-openshift-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:49:05.186300 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:05.186155 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxcw2\" (UniqueName: \"kubernetes.io/projected/54802583-fbdc-48a9-90de-262d5aecb544-kube-api-access-dxcw2\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:49:05.829906 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:05.829877 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-tk8j6" Oct 31 13:49:05.830098 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:05.829914 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-tk8j6" event={"ID":"54802583-fbdc-48a9-90de-262d5aecb544","Type":"ContainerDied","Data":"443d1d14f7c70d4a62f70649144b69758eecdf2e63e37b260c426936f41ffd61"} Oct 31 13:49:05.830098 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:05.829966 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="443d1d14f7c70d4a62f70649144b69758eecdf2e63e37b260c426936f41ffd61" Oct 31 13:49:20.034540 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:49:20.034505 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:49:24.673432 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:24.673391 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5kjcn"] Oct 31 13:49:24.674055 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:24.674028 6841 cpu_manager.go:414] "containerMap: removing container" podUID="c408c29a-b6d8-4b6d-96a3-a6c7c65c729a" containerName="discovery" Oct 31 13:49:24.674055 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:24.674052 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c408c29a-b6d8-4b6d-96a3-a6c7c65c729a" containerName="discovery" Oct 31 13:49:24.674250 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:24.674098 6841 cpu_manager.go:414] "containerMap: removing container" podUID="54802583-fbdc-48a9-90de-262d5aecb544" containerName="revoke-keys" Oct 31 13:49:24.674250 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:24.674107 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54802583-fbdc-48a9-90de-262d5aecb544" containerName="revoke-keys" Oct 31 13:49:24.674250 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:24.674184 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="c408c29a-b6d8-4b6d-96a3-a6c7c65c729a" containerName="discovery" Oct 31 13:49:24.674250 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:24.674204 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="54802583-fbdc-48a9-90de-262d5aecb544" containerName="revoke-keys" Oct 31 13:49:24.677281 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:24.677264 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5kjcn" Oct 31 13:49:24.681643 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:24.681623 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 13:49:24.684330 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:24.684239 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:49:24.684733 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:24.684699 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:49:24.687429 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:24.684806 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:49:24.688903 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:24.688875 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5kjcn"] Oct 31 13:49:24.740468 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:24.740441 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6604b972-8235-4362-8ac7-5698f25099a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-5kjcn\" (UID: \"6604b972-8235-4362-8ac7-5698f25099a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5kjcn" Oct 31 13:49:24.740585 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:24.740505 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6604b972-8235-4362-8ac7-5698f25099a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-5kjcn\" (UID: \"6604b972-8235-4362-8ac7-5698f25099a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5kjcn" Oct 31 13:49:24.740585 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:24.740526 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6xns\" (UniqueName: \"kubernetes.io/projected/6604b972-8235-4362-8ac7-5698f25099a1-kube-api-access-z6xns\") pod \"maas-api-revoke-sub-f03629db-5kjcn\" (UID: \"6604b972-8235-4362-8ac7-5698f25099a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5kjcn" Oct 31 13:49:24.841753 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:24.841717 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6604b972-8235-4362-8ac7-5698f25099a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-5kjcn\" (UID: \"6604b972-8235-4362-8ac7-5698f25099a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5kjcn" Oct 31 13:49:24.841909 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:24.841775 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6604b972-8235-4362-8ac7-5698f25099a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-5kjcn\" (UID: \"6604b972-8235-4362-8ac7-5698f25099a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5kjcn" Oct 31 13:49:24.841909 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:24.841795 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6xns\" (UniqueName: \"kubernetes.io/projected/6604b972-8235-4362-8ac7-5698f25099a1-kube-api-access-z6xns\") pod \"maas-api-revoke-sub-f03629db-5kjcn\" (UID: \"6604b972-8235-4362-8ac7-5698f25099a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5kjcn" Oct 31 13:49:24.842384 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:24.842358 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6604b972-8235-4362-8ac7-5698f25099a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-5kjcn\" (UID: \"6604b972-8235-4362-8ac7-5698f25099a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5kjcn" Oct 31 13:49:24.844233 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:24.844211 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6604b972-8235-4362-8ac7-5698f25099a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-5kjcn\" (UID: \"6604b972-8235-4362-8ac7-5698f25099a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5kjcn" Oct 31 13:49:24.854577 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:24.854555 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6xns\" (UniqueName: \"kubernetes.io/projected/6604b972-8235-4362-8ac7-5698f25099a1-kube-api-access-z6xns\") pod \"maas-api-revoke-sub-f03629db-5kjcn\" (UID: \"6604b972-8235-4362-8ac7-5698f25099a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5kjcn" Oct 31 13:49:24.991034 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:24.990999 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5kjcn" Oct 31 13:49:25.269413 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:25.269386 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5kjcn"] Oct 31 13:49:25.271218 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:49:25.271188 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6604b972_8235_4362_8ac7_5698f25099a1.slice/crio-a138cee53338dff8a0eb203e75420e0c0370e633da94d3579d5e82565ffee976 WatchSource:0}: Error finding container a138cee53338dff8a0eb203e75420e0c0370e633da94d3579d5e82565ffee976: Status 404 returned error can't find the container with id a138cee53338dff8a0eb203e75420e0c0370e633da94d3579d5e82565ffee976 Oct 31 13:49:25.981817 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:25.981777 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5kjcn" event={"ID":"6604b972-8235-4362-8ac7-5698f25099a1","Type":"ContainerStarted","Data":"1158f2a756832676f4b89c597c48b565de6a5035be08653799d7d788c61b963a"} Oct 31 13:49:25.982298 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:25.981842 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5kjcn" event={"ID":"6604b972-8235-4362-8ac7-5698f25099a1","Type":"ContainerStarted","Data":"a138cee53338dff8a0eb203e75420e0c0370e633da94d3579d5e82565ffee976"} Oct 31 13:49:26.002419 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:26.002367 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5kjcn" podStartSLOduration=2.002353981 podStartE2EDuration="2.002353981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:49:26.000319323 +0000 UTC m=+4612.483442244" watchObservedRunningTime="2026-10-31 13:49:26.002353981 +0000 UTC m=+4612.485476884" Oct 31 13:49:26.987913 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:26.987876 6841 generic.go:356] "Generic (PLEG): container finished" podID="6604b972-8235-4362-8ac7-5698f25099a1" containerID="1158f2a756832676f4b89c597c48b565de6a5035be08653799d7d788c61b963a" exitCode=0 Oct 31 13:49:26.988319 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:26.987951 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5kjcn" event={"ID":"6604b972-8235-4362-8ac7-5698f25099a1","Type":"ContainerDied","Data":"1158f2a756832676f4b89c597c48b565de6a5035be08653799d7d788c61b963a"} Oct 31 13:49:28.129654 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:28.129624 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5kjcn" Oct 31 13:49:28.278161 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:28.278083 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6604b972-8235-4362-8ac7-5698f25099a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6604b972-8235-4362-8ac7-5698f25099a1-maas-api-xks-ca\") pod \"6604b972-8235-4362-8ac7-5698f25099a1\" (UID: \"6604b972-8235-4362-8ac7-5698f25099a1\") " Oct 31 13:49:28.278161 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:28.278146 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6604b972-8235-4362-8ac7-5698f25099a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6604b972-8235-4362-8ac7-5698f25099a1-maas-api-openshift-ca\") pod \"6604b972-8235-4362-8ac7-5698f25099a1\" (UID: \"6604b972-8235-4362-8ac7-5698f25099a1\") " Oct 31 13:49:28.278349 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:28.278186 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6604b972-8235-4362-8ac7-5698f25099a1-kube-api-access-z6xns\" (UniqueName: \"kubernetes.io/projected/6604b972-8235-4362-8ac7-5698f25099a1-kube-api-access-z6xns\") pod \"6604b972-8235-4362-8ac7-5698f25099a1\" (UID: \"6604b972-8235-4362-8ac7-5698f25099a1\") " Oct 31 13:49:28.278507 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:28.278483 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6604b972-8235-4362-8ac7-5698f25099a1-maas-api-openshift-ca" pod "6604b972-8235-4362-8ac7-5698f25099a1" (UID: "6604b972-8235-4362-8ac7-5698f25099a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:49:28.280276 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:28.280252 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6604b972-8235-4362-8ac7-5698f25099a1-kube-api-access-z6xns" pod "6604b972-8235-4362-8ac7-5698f25099a1" (UID: "6604b972-8235-4362-8ac7-5698f25099a1"). InnerVolumeSpecName "kube-api-access-z6xns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:49:28.280276 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:28.280267 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6604b972-8235-4362-8ac7-5698f25099a1-maas-api-xks-ca" pod "6604b972-8235-4362-8ac7-5698f25099a1" (UID: "6604b972-8235-4362-8ac7-5698f25099a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:49:28.379234 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:28.379205 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6xns\" (UniqueName: \"kubernetes.io/projected/6604b972-8235-4362-8ac7-5698f25099a1-kube-api-access-z6xns\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:49:28.379234 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:28.379232 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6604b972-8235-4362-8ac7-5698f25099a1-maas-api-xks-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:49:28.379441 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:28.379242 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6604b972-8235-4362-8ac7-5698f25099a1-maas-api-openshift-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:49:29.000069 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:29.000041 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5kjcn" Oct 31 13:49:29.000254 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:29.000038 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5kjcn" event={"ID":"6604b972-8235-4362-8ac7-5698f25099a1","Type":"ContainerDied","Data":"a138cee53338dff8a0eb203e75420e0c0370e633da94d3579d5e82565ffee976"} Oct 31 13:49:29.000254 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:29.000171 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a138cee53338dff8a0eb203e75420e0c0370e633da94d3579d5e82565ffee976" Oct 31 13:49:40.747679 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:40.747637 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w9dcl"] Oct 31 13:49:40.748312 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:40.748283 6841 cpu_manager.go:414] "containerMap: removing container" podUID="6604b972-8235-4362-8ac7-5698f25099a1" containerName="revoke-keys" Oct 31 13:49:40.748312 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:40.748299 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6604b972-8235-4362-8ac7-5698f25099a1" containerName="revoke-keys" Oct 31 13:49:40.748452 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:40.748439 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="6604b972-8235-4362-8ac7-5698f25099a1" containerName="revoke-keys" Oct 31 13:49:40.752454 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:40.752434 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w9dcl" Oct 31 13:49:40.760109 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:40.759905 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:49:40.761473 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:40.761447 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:49:40.761596 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:40.761537 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 13:49:40.761663 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:40.761600 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:49:40.762898 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:40.762874 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w9dcl"] Oct 31 13:49:40.844926 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:40.844890 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-972cd\" (UniqueName: \"kubernetes.io/projected/c9a4c1cd-e46c-41b3-b201-a62f51880a3f-kube-api-access-972cd\") pod \"maas-api-revoke-sub-0165d7d2-w9dcl\" (UID: \"c9a4c1cd-e46c-41b3-b201-a62f51880a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w9dcl" Oct 31 13:49:40.845134 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:40.844949 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9a4c1cd-e46c-41b3-b201-a62f51880a3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-w9dcl\" (UID: \"c9a4c1cd-e46c-41b3-b201-a62f51880a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w9dcl" Oct 31 13:49:40.845134 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:40.845118 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9a4c1cd-e46c-41b3-b201-a62f51880a3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-w9dcl\" (UID: \"c9a4c1cd-e46c-41b3-b201-a62f51880a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w9dcl" Oct 31 13:49:40.945562 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:40.945530 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9a4c1cd-e46c-41b3-b201-a62f51880a3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-w9dcl\" (UID: \"c9a4c1cd-e46c-41b3-b201-a62f51880a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w9dcl" Oct 31 13:49:40.945728 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:40.945586 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-972cd\" (UniqueName: \"kubernetes.io/projected/c9a4c1cd-e46c-41b3-b201-a62f51880a3f-kube-api-access-972cd\") pod \"maas-api-revoke-sub-0165d7d2-w9dcl\" (UID: \"c9a4c1cd-e46c-41b3-b201-a62f51880a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w9dcl" Oct 31 13:49:40.945728 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:40.945636 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9a4c1cd-e46c-41b3-b201-a62f51880a3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-w9dcl\" (UID: \"c9a4c1cd-e46c-41b3-b201-a62f51880a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w9dcl" Oct 31 13:49:40.946324 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:40.946302 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9a4c1cd-e46c-41b3-b201-a62f51880a3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-w9dcl\" (UID: \"c9a4c1cd-e46c-41b3-b201-a62f51880a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w9dcl" Oct 31 13:49:40.948043 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:40.948022 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9a4c1cd-e46c-41b3-b201-a62f51880a3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-w9dcl\" (UID: \"c9a4c1cd-e46c-41b3-b201-a62f51880a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w9dcl" Oct 31 13:49:40.958209 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:40.958181 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-972cd\" (UniqueName: \"kubernetes.io/projected/c9a4c1cd-e46c-41b3-b201-a62f51880a3f-kube-api-access-972cd\") pod \"maas-api-revoke-sub-0165d7d2-w9dcl\" (UID: \"c9a4c1cd-e46c-41b3-b201-a62f51880a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w9dcl" Oct 31 13:49:41.067372 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:41.067303 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w9dcl" Oct 31 13:49:41.343360 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:41.343330 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w9dcl"] Oct 31 13:49:41.345163 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:49:41.345130 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9a4c1cd_e46c_41b3_b201_a62f51880a3f.slice/crio-38e4ff1ec0272e77c2fe36cf7ee4617f35302eaa44ac2bf2fdb854cb1186b7ed WatchSource:0}: Error finding container 38e4ff1ec0272e77c2fe36cf7ee4617f35302eaa44ac2bf2fdb854cb1186b7ed: Status 404 returned error can't find the container with id 38e4ff1ec0272e77c2fe36cf7ee4617f35302eaa44ac2bf2fdb854cb1186b7ed Oct 31 13:49:41.376772 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:41.376744 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:49:42.102374 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:42.102335 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w9dcl" event={"ID":"c9a4c1cd-e46c-41b3-b201-a62f51880a3f","Type":"ContainerStarted","Data":"cec2bd58920f98715aebd07dc7385f4451c5798911c59f95600b1665e75309d9"} Oct 31 13:49:42.102784 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:42.102392 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w9dcl" event={"ID":"c9a4c1cd-e46c-41b3-b201-a62f51880a3f","Type":"ContainerStarted","Data":"38e4ff1ec0272e77c2fe36cf7ee4617f35302eaa44ac2bf2fdb854cb1186b7ed"} Oct 31 13:49:42.121226 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:42.121174 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w9dcl" podStartSLOduration=2.12115738 podStartE2EDuration="2.12115738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:49:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:49:42.119902017 +0000 UTC m=+4628.603024923" watchObservedRunningTime="2026-10-31 13:49:42.12115738 +0000 UTC m=+4628.604280275" Oct 31 13:49:43.109182 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:43.109149 6841 generic.go:356] "Generic (PLEG): container finished" podID="c9a4c1cd-e46c-41b3-b201-a62f51880a3f" containerID="cec2bd58920f98715aebd07dc7385f4451c5798911c59f95600b1665e75309d9" exitCode=0 Oct 31 13:49:43.109641 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:43.109194 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w9dcl" event={"ID":"c9a4c1cd-e46c-41b3-b201-a62f51880a3f","Type":"ContainerDied","Data":"cec2bd58920f98715aebd07dc7385f4451c5798911c59f95600b1665e75309d9"} Oct 31 13:49:44.256905 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:44.256871 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w9dcl" Oct 31 13:49:44.379554 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:44.379469 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9a4c1cd-e46c-41b3-b201-a62f51880a3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9a4c1cd-e46c-41b3-b201-a62f51880a3f-maas-api-openshift-ca\") pod \"c9a4c1cd-e46c-41b3-b201-a62f51880a3f\" (UID: \"c9a4c1cd-e46c-41b3-b201-a62f51880a3f\") " Oct 31 13:49:44.379554 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:44.379554 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9a4c1cd-e46c-41b3-b201-a62f51880a3f-kube-api-access-972cd\" (UniqueName: \"kubernetes.io/projected/c9a4c1cd-e46c-41b3-b201-a62f51880a3f-kube-api-access-972cd\") pod \"c9a4c1cd-e46c-41b3-b201-a62f51880a3f\" (UID: \"c9a4c1cd-e46c-41b3-b201-a62f51880a3f\") " Oct 31 13:49:44.379838 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:44.379612 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9a4c1cd-e46c-41b3-b201-a62f51880a3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9a4c1cd-e46c-41b3-b201-a62f51880a3f-maas-api-xks-ca\") pod \"c9a4c1cd-e46c-41b3-b201-a62f51880a3f\" (UID: \"c9a4c1cd-e46c-41b3-b201-a62f51880a3f\") " Oct 31 13:49:44.379946 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:44.379919 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9a4c1cd-e46c-41b3-b201-a62f51880a3f-maas-api-openshift-ca" pod "c9a4c1cd-e46c-41b3-b201-a62f51880a3f" (UID: "c9a4c1cd-e46c-41b3-b201-a62f51880a3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:49:44.381679 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:44.381654 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9a4c1cd-e46c-41b3-b201-a62f51880a3f-maas-api-xks-ca" pod "c9a4c1cd-e46c-41b3-b201-a62f51880a3f" (UID: "c9a4c1cd-e46c-41b3-b201-a62f51880a3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:49:44.381811 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:44.381781 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9a4c1cd-e46c-41b3-b201-a62f51880a3f-kube-api-access-972cd" pod "c9a4c1cd-e46c-41b3-b201-a62f51880a3f" (UID: "c9a4c1cd-e46c-41b3-b201-a62f51880a3f"). InnerVolumeSpecName "kube-api-access-972cd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:49:44.480529 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:44.480496 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9a4c1cd-e46c-41b3-b201-a62f51880a3f-maas-api-xks-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:49:44.480529 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:44.480527 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9a4c1cd-e46c-41b3-b201-a62f51880a3f-maas-api-openshift-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:49:44.480754 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:44.480540 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-972cd\" (UniqueName: \"kubernetes.io/projected/c9a4c1cd-e46c-41b3-b201-a62f51880a3f-kube-api-access-972cd\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:49:45.122447 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:45.122417 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w9dcl" Oct 31 13:49:45.122621 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:45.122464 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w9dcl" event={"ID":"c9a4c1cd-e46c-41b3-b201-a62f51880a3f","Type":"ContainerDied","Data":"38e4ff1ec0272e77c2fe36cf7ee4617f35302eaa44ac2bf2fdb854cb1186b7ed"} Oct 31 13:49:45.122621 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:45.122517 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38e4ff1ec0272e77c2fe36cf7ee4617f35302eaa44ac2bf2fdb854cb1186b7ed" Oct 31 13:49:50.036606 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:49:50.036570 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:49:51.524789 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:51.524748 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8nm9f"] Oct 31 13:49:51.525258 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:51.525224 6841 cpu_manager.go:414] "containerMap: removing container" podUID="c9a4c1cd-e46c-41b3-b201-a62f51880a3f" containerName="revoke-keys" Oct 31 13:49:51.525258 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:51.525238 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9a4c1cd-e46c-41b3-b201-a62f51880a3f" containerName="revoke-keys" Oct 31 13:49:51.525380 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:51.525349 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9a4c1cd-e46c-41b3-b201-a62f51880a3f" containerName="revoke-keys" Oct 31 13:49:51.528690 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:51.528673 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8nm9f" Oct 31 13:49:51.533398 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:51.533377 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:49:51.533519 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:51.533436 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 13:49:51.534530 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:51.534514 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:49:51.539181 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:51.539160 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8nm9f"] Oct 31 13:49:51.545743 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:51.545711 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fc9c711-807a-4378-98d8-8e39feda77d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-8nm9f\" (UID: \"6fc9c711-807a-4378-98d8-8e39feda77d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8nm9f" Oct 31 13:49:51.545892 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:51.545874 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvbqw\" (UniqueName: \"kubernetes.io/projected/6fc9c711-807a-4378-98d8-8e39feda77d5-kube-api-access-rvbqw\") pod \"maas-api-revoke-sub-6b19338f-8nm9f\" (UID: \"6fc9c711-807a-4378-98d8-8e39feda77d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8nm9f" Oct 31 13:49:51.545952 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:51.545913 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fc9c711-807a-4378-98d8-8e39feda77d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-8nm9f\" (UID: \"6fc9c711-807a-4378-98d8-8e39feda77d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8nm9f" Oct 31 13:49:51.634327 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:51.634304 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:49:51.646746 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:51.646723 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvbqw\" (UniqueName: \"kubernetes.io/projected/6fc9c711-807a-4378-98d8-8e39feda77d5-kube-api-access-rvbqw\") pod \"maas-api-revoke-sub-6b19338f-8nm9f\" (UID: \"6fc9c711-807a-4378-98d8-8e39feda77d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8nm9f" Oct 31 13:49:51.646890 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:51.646761 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fc9c711-807a-4378-98d8-8e39feda77d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-8nm9f\" (UID: \"6fc9c711-807a-4378-98d8-8e39feda77d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8nm9f" Oct 31 13:49:51.646890 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:51.646843 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fc9c711-807a-4378-98d8-8e39feda77d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-8nm9f\" (UID: \"6fc9c711-807a-4378-98d8-8e39feda77d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8nm9f" Oct 31 13:49:51.647685 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:51.647661 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fc9c711-807a-4378-98d8-8e39feda77d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-8nm9f\" (UID: \"6fc9c711-807a-4378-98d8-8e39feda77d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8nm9f" Oct 31 13:49:51.649343 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:51.649317 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fc9c711-807a-4378-98d8-8e39feda77d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-8nm9f\" (UID: \"6fc9c711-807a-4378-98d8-8e39feda77d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8nm9f" Oct 31 13:49:51.659675 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:51.659652 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvbqw\" (UniqueName: \"kubernetes.io/projected/6fc9c711-807a-4378-98d8-8e39feda77d5-kube-api-access-rvbqw\") pod \"maas-api-revoke-sub-6b19338f-8nm9f\" (UID: \"6fc9c711-807a-4378-98d8-8e39feda77d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8nm9f" Oct 31 13:49:51.842023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:51.841918 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8nm9f" Oct 31 13:49:52.118800 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:52.118770 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8nm9f"] Oct 31 13:49:52.120669 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:49:52.120642 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc9c711_807a_4378_98d8_8e39feda77d5.slice/crio-a86b033f9698e142baae08aa53108d9c6ad9d166e0972779778082e8831ec7fe WatchSource:0}: Error finding container a86b033f9698e142baae08aa53108d9c6ad9d166e0972779778082e8831ec7fe: Status 404 returned error can't find the container with id a86b033f9698e142baae08aa53108d9c6ad9d166e0972779778082e8831ec7fe Oct 31 13:49:52.160891 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:52.160864 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8nm9f" event={"ID":"6fc9c711-807a-4378-98d8-8e39feda77d5","Type":"ContainerStarted","Data":"a86b033f9698e142baae08aa53108d9c6ad9d166e0972779778082e8831ec7fe"} Oct 31 13:49:53.167554 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:53.167519 6841 generic.go:356] "Generic (PLEG): container finished" podID="6fc9c711-807a-4378-98d8-8e39feda77d5" containerID="90c52c8f4490aca0374e7a6231aebf69b42b4035e0bd2a083c315b358d7da8fd" exitCode=0 Oct 31 13:49:53.167918 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:53.167573 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8nm9f" event={"ID":"6fc9c711-807a-4378-98d8-8e39feda77d5","Type":"ContainerDied","Data":"90c52c8f4490aca0374e7a6231aebf69b42b4035e0bd2a083c315b358d7da8fd"} Oct 31 13:49:54.326708 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:54.326680 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8nm9f" Oct 31 13:49:54.371726 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:54.371684 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fc9c711-807a-4378-98d8-8e39feda77d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fc9c711-807a-4378-98d8-8e39feda77d5-maas-api-xks-ca\") pod \"6fc9c711-807a-4378-98d8-8e39feda77d5\" (UID: \"6fc9c711-807a-4378-98d8-8e39feda77d5\") " Oct 31 13:49:54.371726 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:54.371728 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fc9c711-807a-4378-98d8-8e39feda77d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fc9c711-807a-4378-98d8-8e39feda77d5-maas-api-openshift-ca\") pod \"6fc9c711-807a-4378-98d8-8e39feda77d5\" (UID: \"6fc9c711-807a-4378-98d8-8e39feda77d5\") " Oct 31 13:49:54.372012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:54.371814 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fc9c711-807a-4378-98d8-8e39feda77d5-kube-api-access-rvbqw\" (UniqueName: \"kubernetes.io/projected/6fc9c711-807a-4378-98d8-8e39feda77d5-kube-api-access-rvbqw\") pod \"6fc9c711-807a-4378-98d8-8e39feda77d5\" (UID: \"6fc9c711-807a-4378-98d8-8e39feda77d5\") " Oct 31 13:49:54.372220 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:54.372196 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fc9c711-807a-4378-98d8-8e39feda77d5-maas-api-openshift-ca" pod "6fc9c711-807a-4378-98d8-8e39feda77d5" (UID: "6fc9c711-807a-4378-98d8-8e39feda77d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:49:54.373909 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:54.373883 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fc9c711-807a-4378-98d8-8e39feda77d5-maas-api-xks-ca" pod "6fc9c711-807a-4378-98d8-8e39feda77d5" (UID: "6fc9c711-807a-4378-98d8-8e39feda77d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:49:54.374410 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:54.374392 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fc9c711-807a-4378-98d8-8e39feda77d5-kube-api-access-rvbqw" pod "6fc9c711-807a-4378-98d8-8e39feda77d5" (UID: "6fc9c711-807a-4378-98d8-8e39feda77d5"). InnerVolumeSpecName "kube-api-access-rvbqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:49:54.473445 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:54.473409 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvbqw\" (UniqueName: \"kubernetes.io/projected/6fc9c711-807a-4378-98d8-8e39feda77d5-kube-api-access-rvbqw\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:49:54.473445 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:54.473443 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fc9c711-807a-4378-98d8-8e39feda77d5-maas-api-xks-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:49:54.473445 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:54.473453 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fc9c711-807a-4378-98d8-8e39feda77d5-maas-api-openshift-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:49:55.180355 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:55.180325 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8nm9f" Oct 31 13:49:55.180355 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:55.180343 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8nm9f" event={"ID":"6fc9c711-807a-4378-98d8-8e39feda77d5","Type":"ContainerDied","Data":"a86b033f9698e142baae08aa53108d9c6ad9d166e0972779778082e8831ec7fe"} Oct 31 13:49:55.180569 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:49:55.180393 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a86b033f9698e142baae08aa53108d9c6ad9d166e0972779778082e8831ec7fe" Oct 31 13:50:03.692321 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:03.691712 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk"] Oct 31 13:50:03.692861 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:03.692465 6841 cpu_manager.go:414] "containerMap: removing container" podUID="6fc9c711-807a-4378-98d8-8e39feda77d5" containerName="revoke-keys" Oct 31 13:50:03.692861 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:03.692486 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fc9c711-807a-4378-98d8-8e39feda77d5" containerName="revoke-keys" Oct 31 13:50:03.692861 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:03.692637 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fc9c711-807a-4378-98d8-8e39feda77d5" containerName="revoke-keys" Oct 31 13:50:03.793429 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:03.779286 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk"] Oct 31 13:50:03.793429 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:03.779435 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk" Oct 31 13:50:03.793429 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:03.786077 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-7217b54d-kserve-self-signed-certs\"" Oct 31 13:50:03.793429 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:03.792549 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8820104e-614b-472c-80cc-f36198a33a8c-home\") pod \"e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk\" (UID: \"8820104e-614b-472c-80cc-f36198a33a8c\") " pod="llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk" Oct 31 13:50:03.793429 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:03.792599 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8820104e-614b-472c-80cc-f36198a33a8c-tmp-dir\") pod \"e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk\" (UID: \"8820104e-614b-472c-80cc-f36198a33a8c\") " pod="llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk" Oct 31 13:50:03.793429 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:03.792652 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8820104e-614b-472c-80cc-f36198a33a8c-dshm\") pod \"e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk\" (UID: \"8820104e-614b-472c-80cc-f36198a33a8c\") " pod="llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk" Oct 31 13:50:03.793429 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:03.792747 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wpqf\" (UniqueName: \"kubernetes.io/projected/8820104e-614b-472c-80cc-f36198a33a8c-kube-api-access-7wpqf\") pod \"e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk\" (UID: \"8820104e-614b-472c-80cc-f36198a33a8c\") " pod="llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk" Oct 31 13:50:03.793429 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:03.792777 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8820104e-614b-472c-80cc-f36198a33a8c-tls-certs\") pod \"e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk\" (UID: \"8820104e-614b-472c-80cc-f36198a33a8c\") " pod="llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk" Oct 31 13:50:03.793429 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:03.792855 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8820104e-614b-472c-80cc-f36198a33a8c-model-cache\") pod \"e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk\" (UID: \"8820104e-614b-472c-80cc-f36198a33a8c\") " pod="llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk" Oct 31 13:50:03.926006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:03.901581 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wpqf\" (UniqueName: \"kubernetes.io/projected/8820104e-614b-472c-80cc-f36198a33a8c-kube-api-access-7wpqf\") pod \"e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk\" (UID: \"8820104e-614b-472c-80cc-f36198a33a8c\") " pod="llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk" Oct 31 13:50:03.926006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:03.901641 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8820104e-614b-472c-80cc-f36198a33a8c-tls-certs\") pod \"e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk\" (UID: \"8820104e-614b-472c-80cc-f36198a33a8c\") " pod="llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk" Oct 31 13:50:03.926006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:03.901723 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8820104e-614b-472c-80cc-f36198a33a8c-model-cache\") pod \"e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk\" (UID: \"8820104e-614b-472c-80cc-f36198a33a8c\") " pod="llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk" Oct 31 13:50:03.926006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:03.901781 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8820104e-614b-472c-80cc-f36198a33a8c-home\") pod \"e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk\" (UID: \"8820104e-614b-472c-80cc-f36198a33a8c\") " pod="llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk" Oct 31 13:50:03.926006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:03.901811 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8820104e-614b-472c-80cc-f36198a33a8c-tmp-dir\") pod \"e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk\" (UID: \"8820104e-614b-472c-80cc-f36198a33a8c\") " pod="llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk" Oct 31 13:50:03.926006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:03.901864 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8820104e-614b-472c-80cc-f36198a33a8c-dshm\") pod \"e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk\" (UID: \"8820104e-614b-472c-80cc-f36198a33a8c\") " pod="llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk" Oct 31 13:50:03.926006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:03.906852 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8820104e-614b-472c-80cc-f36198a33a8c-home\") pod \"e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk\" (UID: \"8820104e-614b-472c-80cc-f36198a33a8c\") " pod="llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk" Oct 31 13:50:03.926006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:03.907146 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8820104e-614b-472c-80cc-f36198a33a8c-model-cache\") pod \"e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk\" (UID: \"8820104e-614b-472c-80cc-f36198a33a8c\") " pod="llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk" Oct 31 13:50:03.926006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:03.907390 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8820104e-614b-472c-80cc-f36198a33a8c-tmp-dir\") pod \"e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk\" (UID: \"8820104e-614b-472c-80cc-f36198a33a8c\") " pod="llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk" Oct 31 13:50:03.937908 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:03.937876 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:50:03.953235 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:03.945237 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wpqf\" (UniqueName: \"kubernetes.io/projected/8820104e-614b-472c-80cc-f36198a33a8c-kube-api-access-7wpqf\") pod \"e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk\" (UID: \"8820104e-614b-472c-80cc-f36198a33a8c\") " pod="llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk" Oct 31 13:50:03.976027 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:03.966698 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8820104e-614b-472c-80cc-f36198a33a8c-dshm\") pod \"e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk\" (UID: \"8820104e-614b-472c-80cc-f36198a33a8c\") " pod="llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk" Oct 31 13:50:03.976027 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:03.974329 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8820104e-614b-472c-80cc-f36198a33a8c-tls-certs\") pod \"e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk\" (UID: \"8820104e-614b-472c-80cc-f36198a33a8c\") " pod="llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk" Oct 31 13:50:04.171387 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:04.147040 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-7217b54d-kserve-router-scheduler-677c77874-dtdpn"] Oct 31 13:50:04.379057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:04.375204 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:50:04.405130 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:04.398826 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk" Oct 31 13:50:04.420212 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:04.420184 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-7217b54d-kserve-router-scheduler-677c77874-dtdpn" Oct 31 13:50:04.422647 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:04.422614 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-7217b54d-kserve-router-scheduler-677c77874-dtdpn"] Oct 31 13:50:04.430974 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:04.430950 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-7217b54d-epp-sa-dockercfg-w76ql\"" Oct 31 13:50:04.508440 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:04.508406 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8f9725f-6f10-4670-af92-1df7fc4b1e6b-tls-certs\") pod \"e2e-prio-pool-7217b54d-kserve-router-scheduler-677c77874-dtdpn\" (UID: \"d8f9725f-6f10-4670-af92-1df7fc4b1e6b\") " pod="llm/e2e-prio-pool-7217b54d-kserve-router-scheduler-677c77874-dtdpn" Oct 31 13:50:04.508702 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:04.508678 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7x98\" (UniqueName: \"kubernetes.io/projected/d8f9725f-6f10-4670-af92-1df7fc4b1e6b-kube-api-access-c7x98\") pod \"e2e-prio-pool-7217b54d-kserve-router-scheduler-677c77874-dtdpn\" (UID: \"d8f9725f-6f10-4670-af92-1df7fc4b1e6b\") " pod="llm/e2e-prio-pool-7217b54d-kserve-router-scheduler-677c77874-dtdpn" Oct 31 13:50:04.616004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:04.615688 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8f9725f-6f10-4670-af92-1df7fc4b1e6b-tls-certs\") pod \"e2e-prio-pool-7217b54d-kserve-router-scheduler-677c77874-dtdpn\" (UID: \"d8f9725f-6f10-4670-af92-1df7fc4b1e6b\") " pod="llm/e2e-prio-pool-7217b54d-kserve-router-scheduler-677c77874-dtdpn" Oct 31 13:50:04.616004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:04.615792 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7x98\" (UniqueName: \"kubernetes.io/projected/d8f9725f-6f10-4670-af92-1df7fc4b1e6b-kube-api-access-c7x98\") pod \"e2e-prio-pool-7217b54d-kserve-router-scheduler-677c77874-dtdpn\" (UID: \"d8f9725f-6f10-4670-af92-1df7fc4b1e6b\") " pod="llm/e2e-prio-pool-7217b54d-kserve-router-scheduler-677c77874-dtdpn" Oct 31 13:50:04.630822 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:04.630737 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8f9725f-6f10-4670-af92-1df7fc4b1e6b-tls-certs\") pod \"e2e-prio-pool-7217b54d-kserve-router-scheduler-677c77874-dtdpn\" (UID: \"d8f9725f-6f10-4670-af92-1df7fc4b1e6b\") " pod="llm/e2e-prio-pool-7217b54d-kserve-router-scheduler-677c77874-dtdpn" Oct 31 13:50:04.633025 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:04.632976 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7x98\" (UniqueName: \"kubernetes.io/projected/d8f9725f-6f10-4670-af92-1df7fc4b1e6b-kube-api-access-c7x98\") pod \"e2e-prio-pool-7217b54d-kserve-router-scheduler-677c77874-dtdpn\" (UID: \"d8f9725f-6f10-4670-af92-1df7fc4b1e6b\") " pod="llm/e2e-prio-pool-7217b54d-kserve-router-scheduler-677c77874-dtdpn" Oct 31 13:50:04.852996 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:04.852951 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-7217b54d-kserve-router-scheduler-677c77874-dtdpn" Oct 31 13:50:04.942447 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:04.942411 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk"] Oct 31 13:50:05.134917 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:05.134885 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-7217b54d-kserve-router-scheduler-677c77874-dtdpn"] Oct 31 13:50:05.136233 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:50:05.136196 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8f9725f_6f10_4670_af92_1df7fc4b1e6b.slice/crio-416ae4d7f4420fdb389186b85bc7617d421e7612d0c0feac276b59b15b4db9c6 WatchSource:0}: Error finding container 416ae4d7f4420fdb389186b85bc7617d421e7612d0c0feac276b59b15b4db9c6: Status 404 returned error can't find the container with id 416ae4d7f4420fdb389186b85bc7617d421e7612d0c0feac276b59b15b4db9c6 Oct 31 13:50:05.471070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:05.471029 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk" event={"ID":"8820104e-614b-472c-80cc-f36198a33a8c","Type":"ContainerStarted","Data":"077bc5f7eb7ab35ec9ffe5a46f1fd1e27d69aecec08b2f9bfb019c4e3bea50ae"} Oct 31 13:50:05.471284 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:05.471086 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk" event={"ID":"8820104e-614b-472c-80cc-f36198a33a8c","Type":"ContainerStarted","Data":"22c42040c8f688d56a2105ed28f1b166a7d504ec108831f610c0f47eb385f717"} Oct 31 13:50:05.472728 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:05.472694 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-7217b54d-kserve-router-scheduler-677c77874-dtdpn" event={"ID":"d8f9725f-6f10-4670-af92-1df7fc4b1e6b","Type":"ContainerStarted","Data":"416ae4d7f4420fdb389186b85bc7617d421e7612d0c0feac276b59b15b4db9c6"} Oct 31 13:50:05.492639 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:05.492575 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk" podStartSLOduration=2.492558226 podStartE2EDuration="2.492558226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:50:05.492234596 +0000 UTC m=+4651.975357498" watchObservedRunningTime="2026-10-31 13:50:05.492558226 +0000 UTC m=+4651.975681121" Oct 31 13:50:05.996408 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:05.996371 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:50:06.402667 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:06.402573 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk" Oct 31 13:50:09.535235 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:09.535198 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-7217b54d-kserve-router-scheduler-677c77874-dtdpn" event={"ID":"d8f9725f-6f10-4670-af92-1df7fc4b1e6b","Type":"ContainerStarted","Data":"6bc0a8d8e7f25bdc46a45bf0491d3c9f79eb26a8d42266eb378cafc3661103d7"} Oct 31 13:50:09.535647 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:09.535289 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-7217b54d-kserve-router-scheduler-677c77874-dtdpn" Oct 31 13:50:09.556543 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:09.556498 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-7217b54d-kserve-router-scheduler-677c77874-dtdpn" podStartSLOduration=2.27222649 podStartE2EDuration="6.556484096s" totalImagesPullingTime="4.284257606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:50:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:50:09.554468259 +0000 UTC m=+4656.037591161" watchObservedRunningTime="2026-10-31 13:50:09.556484096 +0000 UTC m=+4656.039607019" Oct 31 13:50:14.402907 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:14.402861 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk" Oct 31 13:50:14.415475 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:14.415448 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk" Oct 31 13:50:14.575280 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:14.575250 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk" Oct 31 13:50:20.034426 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:50:20.034390 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:50:22.024010 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:22.023964 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-214e2225-fl86v"] Oct 31 13:50:22.028584 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:22.028559 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-214e2225-fl86v"] Oct 31 13:50:23.441161 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:23.441107 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:50:23.641518 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:23.641481 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51feb884-049e-45fb-a054-b2f8022622a9" path="/var/lib/kubelet/pods/51feb884-049e-45fb-a054-b2f8022622a9/volumes" Oct 31 13:50:40.543576 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:40.543543 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-7217b54d-kserve-router-scheduler-677c77874-dtdpn" Oct 31 13:50:41.425830 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:41.425797 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:50:42.737050 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:42.737017 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:50:42.936729 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:42.936696 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:50:48.867065 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:48.866609 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-bp2d4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:50:48.867065 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:50:48.866718 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" podUID="51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:50:50.034351 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:50:50.034315 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:51:05.728332 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:05.727712 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk"] Oct 31 13:51:05.733886 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:05.733849 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-7217b54d-kserve-router-scheduler-677c77874-dtdpn"] Oct 31 13:51:05.854654 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:05.854617 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:06.070449 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:06.070367 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:08.181750 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:08.181705 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:10.543449 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:10.543404 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-7217b54d-kserve-router-scheduler-677c77874-dtdpn" podUID="d8f9725f-6f10-4670-af92-1df7fc4b1e6b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:51:12.068128 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:12.068082 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fk2tx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:51:12.068763 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:12.068727 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" podUID="aa7e0a48-9dc5-4735-b1ae-cafd28ff982f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:51:20.034515 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:51:20.034477 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:51:20.543123 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:20.543081 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-7217b54d-kserve-router-scheduler-677c77874-dtdpn" podUID="d8f9725f-6f10-4670-af92-1df7fc4b1e6b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:51:20.773847 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:20.773786 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk" podUID="8820104e-614b-472c-80cc-f36198a33a8c" containerName="main" containerID="cri-o://077bc5f7eb7ab35ec9ffe5a46f1fd1e27d69aecec08b2f9bfb019c4e3bea50ae" gracePeriod=45 Oct 31 13:51:20.799453 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:20.799387 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-7217b54d-kserve-router-scheduler-677c77874-dtdpn" podUID="d8f9725f-6f10-4670-af92-1df7fc4b1e6b" containerName="main" containerID="cri-o://6bc0a8d8e7f25bdc46a45bf0491d3c9f79eb26a8d42266eb378cafc3661103d7" gracePeriod=45 Oct 31 13:51:21.023821 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:21.023797 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk" Oct 31 13:51:21.051737 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:21.051661 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8820104e-614b-472c-80cc-f36198a33a8c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8820104e-614b-472c-80cc-f36198a33a8c-dshm\") pod \"8820104e-614b-472c-80cc-f36198a33a8c\" (UID: \"8820104e-614b-472c-80cc-f36198a33a8c\") " Oct 31 13:51:21.051737 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:21.051723 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8820104e-614b-472c-80cc-f36198a33a8c-home\" (UniqueName: \"kubernetes.io/empty-dir/8820104e-614b-472c-80cc-f36198a33a8c-home\") pod \"8820104e-614b-472c-80cc-f36198a33a8c\" (UID: \"8820104e-614b-472c-80cc-f36198a33a8c\") " Oct 31 13:51:21.052192 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:21.051778 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8820104e-614b-472c-80cc-f36198a33a8c-kube-api-access-7wpqf\" (UniqueName: \"kubernetes.io/projected/8820104e-614b-472c-80cc-f36198a33a8c-kube-api-access-7wpqf\") pod \"8820104e-614b-472c-80cc-f36198a33a8c\" (UID: \"8820104e-614b-472c-80cc-f36198a33a8c\") " Oct 31 13:51:21.052192 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:21.051859 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8820104e-614b-472c-80cc-f36198a33a8c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8820104e-614b-472c-80cc-f36198a33a8c-model-cache\") pod \"8820104e-614b-472c-80cc-f36198a33a8c\" (UID: \"8820104e-614b-472c-80cc-f36198a33a8c\") " Oct 31 13:51:21.052192 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:21.051946 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8820104e-614b-472c-80cc-f36198a33a8c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8820104e-614b-472c-80cc-f36198a33a8c-tmp-dir\") pod \"8820104e-614b-472c-80cc-f36198a33a8c\" (UID: \"8820104e-614b-472c-80cc-f36198a33a8c\") " Oct 31 13:51:21.052192 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:21.052049 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8820104e-614b-472c-80cc-f36198a33a8c-tls-certs\" (UniqueName: \"kubernetes.io/secret/8820104e-614b-472c-80cc-f36198a33a8c-tls-certs\") pod \"8820104e-614b-472c-80cc-f36198a33a8c\" (UID: \"8820104e-614b-472c-80cc-f36198a33a8c\") " Oct 31 13:51:21.052440 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:21.052327 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8820104e-614b-472c-80cc-f36198a33a8c-tmp-dir" pod "8820104e-614b-472c-80cc-f36198a33a8c" (UID: "8820104e-614b-472c-80cc-f36198a33a8c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:51:21.052491 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:21.052450 6841 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8820104e-614b-472c-80cc-f36198a33a8c-tmp-dir\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:51:21.052590 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:21.052248 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8820104e-614b-472c-80cc-f36198a33a8c-model-cache" pod "8820104e-614b-472c-80cc-f36198a33a8c" (UID: "8820104e-614b-472c-80cc-f36198a33a8c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:51:21.052729 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:21.052640 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8820104e-614b-472c-80cc-f36198a33a8c-home" pod "8820104e-614b-472c-80cc-f36198a33a8c" (UID: "8820104e-614b-472c-80cc-f36198a33a8c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:51:21.054334 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:21.054308 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8820104e-614b-472c-80cc-f36198a33a8c-tls-certs" pod "8820104e-614b-472c-80cc-f36198a33a8c" (UID: "8820104e-614b-472c-80cc-f36198a33a8c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:51:21.054481 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:21.054462 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8820104e-614b-472c-80cc-f36198a33a8c-dshm" pod "8820104e-614b-472c-80cc-f36198a33a8c" (UID: "8820104e-614b-472c-80cc-f36198a33a8c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:51:21.054552 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:21.054539 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8820104e-614b-472c-80cc-f36198a33a8c-kube-api-access-7wpqf" pod "8820104e-614b-472c-80cc-f36198a33a8c" (UID: "8820104e-614b-472c-80cc-f36198a33a8c"). InnerVolumeSpecName "kube-api-access-7wpqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:51:21.074069 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:21.074040 6841 generic.go:356] "Generic (PLEG): container finished" podID="8820104e-614b-472c-80cc-f36198a33a8c" containerID="077bc5f7eb7ab35ec9ffe5a46f1fd1e27d69aecec08b2f9bfb019c4e3bea50ae" exitCode=0 Oct 31 13:51:21.074173 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:21.074150 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk" event={"ID":"8820104e-614b-472c-80cc-f36198a33a8c","Type":"ContainerDied","Data":"077bc5f7eb7ab35ec9ffe5a46f1fd1e27d69aecec08b2f9bfb019c4e3bea50ae"} Oct 31 13:51:21.074219 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:21.074184 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk" event={"ID":"8820104e-614b-472c-80cc-f36198a33a8c","Type":"ContainerDied","Data":"22c42040c8f688d56a2105ed28f1b166a7d504ec108831f610c0f47eb385f717"} Oct 31 13:51:21.074219 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:21.074204 6841 scope.go:122] "RemoveContainer" containerID="077bc5f7eb7ab35ec9ffe5a46f1fd1e27d69aecec08b2f9bfb019c4e3bea50ae" Oct 31 13:51:21.074371 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:21.074358 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk" Oct 31 13:51:21.084053 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:21.084034 6841 scope.go:122] "RemoveContainer" containerID="077bc5f7eb7ab35ec9ffe5a46f1fd1e27d69aecec08b2f9bfb019c4e3bea50ae" Oct 31 13:51:21.084344 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:51:21.084319 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"077bc5f7eb7ab35ec9ffe5a46f1fd1e27d69aecec08b2f9bfb019c4e3bea50ae\": container with ID starting with 077bc5f7eb7ab35ec9ffe5a46f1fd1e27d69aecec08b2f9bfb019c4e3bea50ae not found: ID does not exist" containerID="077bc5f7eb7ab35ec9ffe5a46f1fd1e27d69aecec08b2f9bfb019c4e3bea50ae" Oct 31 13:51:21.084434 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:21.084351 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"077bc5f7eb7ab35ec9ffe5a46f1fd1e27d69aecec08b2f9bfb019c4e3bea50ae"} err="failed to get container status \"077bc5f7eb7ab35ec9ffe5a46f1fd1e27d69aecec08b2f9bfb019c4e3bea50ae\": rpc error: code = NotFound desc = could not find container \"077bc5f7eb7ab35ec9ffe5a46f1fd1e27d69aecec08b2f9bfb019c4e3bea50ae\": container with ID starting with 077bc5f7eb7ab35ec9ffe5a46f1fd1e27d69aecec08b2f9bfb019c4e3bea50ae not found: ID does not exist" Oct 31 13:51:21.107451 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:21.107426 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk"] Oct 31 13:51:21.110656 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:21.110633 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-7217b54d-kserve-569f46764c-gtxnk"] Oct 31 13:51:21.153989 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:21.153947 6841 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8820104e-614b-472c-80cc-f36198a33a8c-dshm\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:51:21.153989 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:21.153974 6841 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8820104e-614b-472c-80cc-f36198a33a8c-home\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:51:21.154167 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:21.154022 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wpqf\" (UniqueName: \"kubernetes.io/projected/8820104e-614b-472c-80cc-f36198a33a8c-kube-api-access-7wpqf\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:51:21.154167 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:21.154035 6841 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8820104e-614b-472c-80cc-f36198a33a8c-model-cache\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:51:21.154167 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:21.154047 6841 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8820104e-614b-472c-80cc-f36198a33a8c-tls-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:51:21.609837 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:21.609792 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:21.641358 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:21.641325 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8820104e-614b-472c-80cc-f36198a33a8c" path="/var/lib/kubelet/pods/8820104e-614b-472c-80cc-f36198a33a8c/volumes" Oct 31 13:51:22.075856 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:22.075802 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:22.393129 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:22.393033 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:25.652147 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:25.652098 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:25.872017 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:25.871944 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:26.879033 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:26.878997 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:27.081892 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:27.081855 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:27.965567 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:27.965526 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:28.775369 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:28.775336 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:29.593339 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:29.593282 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:30.394812 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:30.394776 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:30.543212 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:30.543167 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-7217b54d-kserve-router-scheduler-677c77874-dtdpn" podUID="d8f9725f-6f10-4670-af92-1df7fc4b1e6b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:51:30.543403 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:30.543319 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-7217b54d-kserve-router-scheduler-677c77874-dtdpn" Oct 31 13:51:31.190213 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:31.190179 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:31.887061 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:31.887017 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:32.567783 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:32.567748 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:33.280111 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:33.280072 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:33.983108 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:33.983079 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:34.668922 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:34.668880 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:35.498468 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:35.498421 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:36.380762 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:36.380722 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:37.178671 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:37.178629 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:37.739670 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:37.739630 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:37.946791 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:37.946757 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:38.579054 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:38.579008 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:39.409113 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:39.409078 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:40.184680 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:40.184650 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:40.542800 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:40.542758 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-7217b54d-kserve-router-scheduler-677c77874-dtdpn" podUID="d8f9725f-6f10-4670-af92-1df7fc4b1e6b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:51:40.703322 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:40.703287 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:40.900459 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:40.900367 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:41.337280 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:41.337232 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:41.466041 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:41.465861 6841 scope.go:122] "RemoveContainer" containerID="9e04b0f2e2ec19f00cba7c60836957ecef0bbc92a5639cc2099d2091ba4d16b7" Oct 31 13:51:41.535247 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:41.535215 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:41.763487 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:41.763452 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:42.527207 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:42.527170 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:43.280858 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:43.280813 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:43.943779 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:43.943740 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:44.155828 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:44.155788 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:44.947953 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:44.947922 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:45.022892 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:45.022722 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79aa7668-bl9km"] Oct 31 13:51:45.025155 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:45.025129 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79aa7668-bl9km"] Oct 31 13:51:45.640733 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:45.640701 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e701b9b8-f494-4a8f-ae55-87dda0640089" path="/var/lib/kubelet/pods/e701b9b8-f494-4a8f-ae55-87dda0640089/volumes" Oct 31 13:51:45.670322 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:45.670285 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:46.418994 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:46.418928 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:47.284310 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:47.284268 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:48.168911 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:48.168861 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:48.990047 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:48.990010 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:49.681902 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:49.681863 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:50.033787 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:51:50.033747 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:51:50.378703 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:50.378666 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:50.542789 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:50.542750 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-7217b54d-kserve-router-scheduler-677c77874-dtdpn" podUID="d8f9725f-6f10-4670-af92-1df7fc4b1e6b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:51:51.063961 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:51.063895 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-7217b54d-kserve-router-scheduler-677c77874-dtdpn" Oct 31 13:51:51.074529 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:51.074501 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8f9725f-6f10-4670-af92-1df7fc4b1e6b-kube-api-access-c7x98\" (UniqueName: \"kubernetes.io/projected/d8f9725f-6f10-4670-af92-1df7fc4b1e6b-kube-api-access-c7x98\") pod \"d8f9725f-6f10-4670-af92-1df7fc4b1e6b\" (UID: \"d8f9725f-6f10-4670-af92-1df7fc4b1e6b\") " Oct 31 13:51:51.074716 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:51.074600 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8f9725f-6f10-4670-af92-1df7fc4b1e6b-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8f9725f-6f10-4670-af92-1df7fc4b1e6b-tls-certs\") pod \"d8f9725f-6f10-4670-af92-1df7fc4b1e6b\" (UID: \"d8f9725f-6f10-4670-af92-1df7fc4b1e6b\") " Oct 31 13:51:51.076722 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:51.076693 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8f9725f-6f10-4670-af92-1df7fc4b1e6b-tls-certs" pod "d8f9725f-6f10-4670-af92-1df7fc4b1e6b" (UID: "d8f9725f-6f10-4670-af92-1df7fc4b1e6b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:51:51.076856 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:51.076832 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8f9725f-6f10-4670-af92-1df7fc4b1e6b-kube-api-access-c7x98" pod "d8f9725f-6f10-4670-af92-1df7fc4b1e6b" (UID: "d8f9725f-6f10-4670-af92-1df7fc4b1e6b"). InnerVolumeSpecName "kube-api-access-c7x98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:51:51.078743 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:51.078711 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:51.176284 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:51.176247 6841 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8f9725f-6f10-4670-af92-1df7fc4b1e6b-tls-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:51:51.176284 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:51.176279 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7x98\" (UniqueName: \"kubernetes.io/projected/d8f9725f-6f10-4670-af92-1df7fc4b1e6b-kube-api-access-c7x98\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:51:51.249111 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:51.249075 6841 generic.go:356] "Generic (PLEG): container finished" podID="d8f9725f-6f10-4670-af92-1df7fc4b1e6b" containerID="6bc0a8d8e7f25bdc46a45bf0491d3c9f79eb26a8d42266eb378cafc3661103d7" exitCode=0 Oct 31 13:51:51.249304 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:51.249125 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-7217b54d-kserve-router-scheduler-677c77874-dtdpn" Oct 31 13:51:51.249304 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:51.249165 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-7217b54d-kserve-router-scheduler-677c77874-dtdpn" event={"ID":"d8f9725f-6f10-4670-af92-1df7fc4b1e6b","Type":"ContainerDied","Data":"6bc0a8d8e7f25bdc46a45bf0491d3c9f79eb26a8d42266eb378cafc3661103d7"} Oct 31 13:51:51.249304 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:51.249233 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-7217b54d-kserve-router-scheduler-677c77874-dtdpn" event={"ID":"d8f9725f-6f10-4670-af92-1df7fc4b1e6b","Type":"ContainerDied","Data":"416ae4d7f4420fdb389186b85bc7617d421e7612d0c0feac276b59b15b4db9c6"} Oct 31 13:51:51.249304 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:51.249257 6841 scope.go:122] "RemoveContainer" containerID="6bc0a8d8e7f25bdc46a45bf0491d3c9f79eb26a8d42266eb378cafc3661103d7" Oct 31 13:51:51.259391 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:51.259369 6841 scope.go:122] "RemoveContainer" containerID="6bc0a8d8e7f25bdc46a45bf0491d3c9f79eb26a8d42266eb378cafc3661103d7" Oct 31 13:51:51.259658 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:51:51.259638 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bc0a8d8e7f25bdc46a45bf0491d3c9f79eb26a8d42266eb378cafc3661103d7\": container with ID starting with 6bc0a8d8e7f25bdc46a45bf0491d3c9f79eb26a8d42266eb378cafc3661103d7 not found: ID does not exist" containerID="6bc0a8d8e7f25bdc46a45bf0491d3c9f79eb26a8d42266eb378cafc3661103d7" Oct 31 13:51:51.259714 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:51.259668 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bc0a8d8e7f25bdc46a45bf0491d3c9f79eb26a8d42266eb378cafc3661103d7"} err="failed to get container status \"6bc0a8d8e7f25bdc46a45bf0491d3c9f79eb26a8d42266eb378cafc3661103d7\": rpc error: code = NotFound desc = could not find container \"6bc0a8d8e7f25bdc46a45bf0491d3c9f79eb26a8d42266eb378cafc3661103d7\": container with ID starting with 6bc0a8d8e7f25bdc46a45bf0491d3c9f79eb26a8d42266eb378cafc3661103d7 not found: ID does not exist" Oct 31 13:51:51.272852 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:51.272826 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-7217b54d-kserve-router-scheduler-677c77874-dtdpn"] Oct 31 13:51:51.280505 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:51.280480 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-7217b54d-kserve-router-scheduler-677c77874-dtdpn"] Oct 31 13:51:51.641483 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:51.641447 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8f9725f-6f10-4670-af92-1df7fc4b1e6b" path="/var/lib/kubelet/pods/d8f9725f-6f10-4670-af92-1df7fc4b1e6b/volumes" Oct 31 13:51:51.782592 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:51.782563 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:52.475445 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:52.475404 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:53.175717 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:53.175681 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:53.977009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:53.976945 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:54.682829 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:54.682788 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:55.488455 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:55.488421 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:56.179797 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:56.179758 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:56.894539 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:56.894501 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:57.022721 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:57.022676 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3efb793-45fcq"] Oct 31 13:51:57.024752 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:57.024730 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3efb793-45fcq"] Oct 31 13:51:57.584811 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:57.584769 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:57.640990 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:57.640949 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9" path="/var/lib/kubelet/pods/d5bce8ca-9936-4e67-a8cd-98c9e91b9bd9/volumes" Oct 31 13:51:58.284235 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:58.284198 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:59.089304 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:59.089269 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:51:59.985689 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:51:59.985655 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:00.886201 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:00.886164 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:01.802527 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:01.802493 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:02.576728 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:02.576690 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:03.370242 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:03.370203 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:04.084566 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:04.084521 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:04.777779 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:04.777749 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:05.476713 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:05.476667 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:06.252682 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:06.252643 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:07.664533 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:07.664500 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:08.397704 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:08.397662 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:09.178390 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:09.178351 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:09.975447 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:09.975408 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:10.678173 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:10.678133 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:11.370953 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:11.370918 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:12.075344 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:12.075300 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:12.772597 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:12.772554 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:12.857545 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:12.857512 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cc05b22-hp424"] Oct 31 13:52:12.857959 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:12.857945 6841 cpu_manager.go:414] "containerMap: removing container" podUID="d8f9725f-6f10-4670-af92-1df7fc4b1e6b" containerName="main" Oct 31 13:52:12.858033 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:12.857960 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8f9725f-6f10-4670-af92-1df7fc4b1e6b" containerName="main" Oct 31 13:52:12.858033 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:12.857970 6841 cpu_manager.go:414] "containerMap: removing container" podUID="8820104e-614b-472c-80cc-f36198a33a8c" containerName="main" Oct 31 13:52:12.858033 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:12.857975 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8820104e-614b-472c-80cc-f36198a33a8c" containerName="main" Oct 31 13:52:12.858151 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:12.858062 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8f9725f-6f10-4670-af92-1df7fc4b1e6b" containerName="main" Oct 31 13:52:12.858151 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:12.858072 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="8820104e-614b-472c-80cc-f36198a33a8c" containerName="main" Oct 31 13:52:12.862931 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:12.862914 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc05b22-hp424" Oct 31 13:52:12.868525 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:12.868485 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:52:12.868669 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:12.868576 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:52:12.868752 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:12.868657 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 13:52:12.871543 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:12.871520 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cc05b22-hp424"] Oct 31 13:52:12.871722 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:12.871704 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:52:12.999991 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:12.999948 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a162f846-3c4e-4905-9172-fab1fcbac33f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cc05b22-hp424\" (UID: \"a162f846-3c4e-4905-9172-fab1fcbac33f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc05b22-hp424" Oct 31 13:52:13.000164 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:13.000004 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a162f846-3c4e-4905-9172-fab1fcbac33f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cc05b22-hp424\" (UID: \"a162f846-3c4e-4905-9172-fab1fcbac33f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc05b22-hp424" Oct 31 13:52:13.000164 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:13.000101 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvl8k\" (UniqueName: \"kubernetes.io/projected/a162f846-3c4e-4905-9172-fab1fcbac33f-kube-api-access-bvl8k\") pod \"maas-api-revoke-sub-9cc05b22-hp424\" (UID: \"a162f846-3c4e-4905-9172-fab1fcbac33f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc05b22-hp424" Oct 31 13:52:13.100972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:13.100868 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a162f846-3c4e-4905-9172-fab1fcbac33f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cc05b22-hp424\" (UID: \"a162f846-3c4e-4905-9172-fab1fcbac33f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc05b22-hp424" Oct 31 13:52:13.100972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:13.100917 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a162f846-3c4e-4905-9172-fab1fcbac33f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cc05b22-hp424\" (UID: \"a162f846-3c4e-4905-9172-fab1fcbac33f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc05b22-hp424" Oct 31 13:52:13.100972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:13.100951 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvl8k\" (UniqueName: \"kubernetes.io/projected/a162f846-3c4e-4905-9172-fab1fcbac33f-kube-api-access-bvl8k\") pod \"maas-api-revoke-sub-9cc05b22-hp424\" (UID: \"a162f846-3c4e-4905-9172-fab1fcbac33f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc05b22-hp424" Oct 31 13:52:13.101688 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:13.101666 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a162f846-3c4e-4905-9172-fab1fcbac33f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cc05b22-hp424\" (UID: \"a162f846-3c4e-4905-9172-fab1fcbac33f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc05b22-hp424" Oct 31 13:52:13.103419 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:13.103392 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a162f846-3c4e-4905-9172-fab1fcbac33f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cc05b22-hp424\" (UID: \"a162f846-3c4e-4905-9172-fab1fcbac33f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc05b22-hp424" Oct 31 13:52:13.113154 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:13.113126 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvl8k\" (UniqueName: \"kubernetes.io/projected/a162f846-3c4e-4905-9172-fab1fcbac33f-kube-api-access-bvl8k\") pod \"maas-api-revoke-sub-9cc05b22-hp424\" (UID: \"a162f846-3c4e-4905-9172-fab1fcbac33f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc05b22-hp424" Oct 31 13:52:13.178324 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:13.178290 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc05b22-hp424" Oct 31 13:52:13.455669 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:13.455603 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cc05b22-hp424"] Oct 31 13:52:13.459154 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:52:13.459118 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda162f846_3c4e_4905_9172_fab1fcbac33f.slice/crio-aadb37f67c429a1bcdba6b9de2fdf206452ef03c5295c15bed8d3ae27f41e7ce WatchSource:0}: Error finding container aadb37f67c429a1bcdba6b9de2fdf206452ef03c5295c15bed8d3ae27f41e7ce: Status 404 returned error can't find the container with id aadb37f67c429a1bcdba6b9de2fdf206452ef03c5295c15bed8d3ae27f41e7ce Oct 31 13:52:13.504137 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:13.504105 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:14.223685 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:14.223649 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:14.381097 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:14.381060 6841 generic.go:356] "Generic (PLEG): container finished" podID="a162f846-3c4e-4905-9172-fab1fcbac33f" containerID="9609bf5546de3650a6c60b61bc6fef6735d4e10ae278359bbad7c82a58326da7" exitCode=0 Oct 31 13:52:14.381279 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:14.381103 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc05b22-hp424" event={"ID":"a162f846-3c4e-4905-9172-fab1fcbac33f","Type":"ContainerDied","Data":"9609bf5546de3650a6c60b61bc6fef6735d4e10ae278359bbad7c82a58326da7"} Oct 31 13:52:14.381279 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:14.381152 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc05b22-hp424" event={"ID":"a162f846-3c4e-4905-9172-fab1fcbac33f","Type":"ContainerStarted","Data":"aadb37f67c429a1bcdba6b9de2fdf206452ef03c5295c15bed8d3ae27f41e7ce"} Oct 31 13:52:14.958325 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:14.958284 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:15.518495 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:15.518470 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc05b22-hp424" Oct 31 13:52:15.623255 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:15.623219 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a162f846-3c4e-4905-9172-fab1fcbac33f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a162f846-3c4e-4905-9172-fab1fcbac33f-maas-api-xks-ca\") pod \"a162f846-3c4e-4905-9172-fab1fcbac33f\" (UID: \"a162f846-3c4e-4905-9172-fab1fcbac33f\") " Oct 31 13:52:15.623441 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:15.623296 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a162f846-3c4e-4905-9172-fab1fcbac33f-kube-api-access-bvl8k\" (UniqueName: \"kubernetes.io/projected/a162f846-3c4e-4905-9172-fab1fcbac33f-kube-api-access-bvl8k\") pod \"a162f846-3c4e-4905-9172-fab1fcbac33f\" (UID: \"a162f846-3c4e-4905-9172-fab1fcbac33f\") " Oct 31 13:52:15.623441 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:15.623327 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a162f846-3c4e-4905-9172-fab1fcbac33f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a162f846-3c4e-4905-9172-fab1fcbac33f-maas-api-openshift-ca\") pod \"a162f846-3c4e-4905-9172-fab1fcbac33f\" (UID: \"a162f846-3c4e-4905-9172-fab1fcbac33f\") " Oct 31 13:52:15.623827 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:15.623793 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a162f846-3c4e-4905-9172-fab1fcbac33f-maas-api-openshift-ca" pod "a162f846-3c4e-4905-9172-fab1fcbac33f" (UID: "a162f846-3c4e-4905-9172-fab1fcbac33f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:52:15.625750 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:15.625717 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a162f846-3c4e-4905-9172-fab1fcbac33f-maas-api-xks-ca" pod "a162f846-3c4e-4905-9172-fab1fcbac33f" (UID: "a162f846-3c4e-4905-9172-fab1fcbac33f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:52:15.626179 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:15.626152 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a162f846-3c4e-4905-9172-fab1fcbac33f-kube-api-access-bvl8k" pod "a162f846-3c4e-4905-9172-fab1fcbac33f" (UID: "a162f846-3c4e-4905-9172-fab1fcbac33f"). InnerVolumeSpecName "kube-api-access-bvl8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:52:15.677523 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:15.677493 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:15.724712 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:15.724686 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a162f846-3c4e-4905-9172-fab1fcbac33f-maas-api-xks-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:15.724712 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:15.724712 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvl8k\" (UniqueName: \"kubernetes.io/projected/a162f846-3c4e-4905-9172-fab1fcbac33f-kube-api-access-bvl8k\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:15.724871 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:15.724722 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a162f846-3c4e-4905-9172-fab1fcbac33f-maas-api-openshift-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:16.393660 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:16.393625 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc05b22-hp424" Oct 31 13:52:16.393865 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:16.393623 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc05b22-hp424" event={"ID":"a162f846-3c4e-4905-9172-fab1fcbac33f","Type":"ContainerDied","Data":"aadb37f67c429a1bcdba6b9de2fdf206452ef03c5295c15bed8d3ae27f41e7ce"} Oct 31 13:52:16.393865 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:16.393752 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aadb37f67c429a1bcdba6b9de2fdf206452ef03c5295c15bed8d3ae27f41e7ce" Oct 31 13:52:16.464328 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:16.464288 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:17.169351 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:17.169314 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:17.980056 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:17.980023 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:18.783097 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:18.783056 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:19.476316 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:19.476279 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:20.034876 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:52:20.034841 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:52:20.192350 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:20.192317 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:20.893018 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:20.892969 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:21.579888 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:21.579852 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:22.282338 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:22.282301 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:22.987798 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:22.987764 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:23.685846 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:23.685809 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:23.885971 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:23.885933 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:24.188281 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:24.188241 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:24.460717 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:24.460683 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:25.270343 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:25.270307 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:25.968804 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:25.968756 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:26.787620 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:26.787584 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:27.571325 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:27.571291 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:28.350775 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:28.350739 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:28.850064 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:28.850026 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:29.129789 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:29.129703 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:29.330160 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:29.330129 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:29.891561 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:29.891526 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:30.461399 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:30.461362 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:31.064148 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:31.064115 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:36.798922 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:36.798878 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:36.996630 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:36.996600 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:41.559358 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:52:41.559324 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:52:50.034232 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:52:50.034185 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:53:00.652894 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:00.652849 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:00.857377 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:00.857339 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:01.789299 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:01.789262 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj"] Oct 31 13:53:01.789930 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:01.789907 6841 cpu_manager.go:414] "containerMap: removing container" podUID="a162f846-3c4e-4905-9172-fab1fcbac33f" containerName="revoke-keys" Oct 31 13:53:01.790045 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:01.789932 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a162f846-3c4e-4905-9172-fab1fcbac33f" containerName="revoke-keys" Oct 31 13:53:01.790115 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:01.790079 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="a162f846-3c4e-4905-9172-fab1fcbac33f" containerName="revoke-keys" Oct 31 13:53:01.793316 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:01.793293 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj" Oct 31 13:53:01.803912 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:01.803844 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 13:53:01.804107 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:01.803919 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:53:01.804107 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:01.804035 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:53:01.804262 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:01.804119 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-3a5889-serving-cert\"" Oct 31 13:53:01.804310 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:01.804254 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rhftr\"" Oct 31 13:53:01.807238 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:01.807216 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj"] Oct 31 13:53:01.834466 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:01.834436 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4dmj44f"] Oct 31 13:53:01.838377 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:01.838358 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4dmj44f" Oct 31 13:53:01.849639 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:01.849612 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4dmj44f"] Oct 31 13:53:01.867165 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:01.867133 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:01.877972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:01.877941 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495qd7jn"] Oct 31 13:53:01.881716 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:01.881698 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495qd7jn" Oct 31 13:53:01.894602 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:01.894563 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495qd7jn"] Oct 31 13:53:01.904123 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:01.904095 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-3a5889\"" Oct 31 13:53:01.904273 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:01.904130 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-5be48be7-dockercfg-x99n2\"" Oct 31 13:53:01.977461 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:01.977421 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b263bb54-a3b4-4cbc-a1cb-67612f01314a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj\" (UID: \"b263bb54-a3b4-4cbc-a1cb-67612f01314a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj" Oct 31 13:53:01.977766 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:01.977737 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0082bcdc-04ae-43b5-82ad-9a5052f535f2-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-3a5889-cc4947495qd7jn\" (UID: \"0082bcdc-04ae-43b5-82ad-9a5052f535f2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495qd7jn" Oct 31 13:53:01.977925 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:01.977791 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed78ec8c-6582-4454-b810-9d37124a82a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3a5889-5bd4dmj44f\" (UID: \"ed78ec8c-6582-4454-b810-9d37124a82a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4dmj44f" Oct 31 13:53:01.977925 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:01.977885 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b263bb54-a3b4-4cbc-a1cb-67612f01314a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj\" (UID: \"b263bb54-a3b4-4cbc-a1cb-67612f01314a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj" Oct 31 13:53:01.978134 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:01.977926 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b5jp\" (UniqueName: \"kubernetes.io/projected/b263bb54-a3b4-4cbc-a1cb-67612f01314a-kube-api-access-6b5jp\") pod \"maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj\" (UID: \"b263bb54-a3b4-4cbc-a1cb-67612f01314a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj" Oct 31 13:53:01.978134 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:01.977973 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slprv\" (UniqueName: \"kubernetes.io/projected/0082bcdc-04ae-43b5-82ad-9a5052f535f2-kube-api-access-slprv\") pod \"payload-processing-e2e-worker-single-main-3a5889-cc4947495qd7jn\" (UID: \"0082bcdc-04ae-43b5-82ad-9a5052f535f2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495qd7jn" Oct 31 13:53:01.978134 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:01.978052 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b263bb54-a3b4-4cbc-a1cb-67612f01314a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj\" (UID: \"b263bb54-a3b4-4cbc-a1cb-67612f01314a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj" Oct 31 13:53:01.978322 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:01.978301 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4t6h\" (UniqueName: \"kubernetes.io/projected/ed78ec8c-6582-4454-b810-9d37124a82a5-kube-api-access-t4t6h\") pod \"payload-pre-processing-e2e-worker-single-main-3a5889-5bd4dmj44f\" (UID: \"ed78ec8c-6582-4454-b810-9d37124a82a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4dmj44f" Oct 31 13:53:02.069007 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:02.068901 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:02.079323 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:02.079273 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b263bb54-a3b4-4cbc-a1cb-67612f01314a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj\" (UID: \"b263bb54-a3b4-4cbc-a1cb-67612f01314a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj" Oct 31 13:53:02.079483 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:02.079397 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4t6h\" (UniqueName: \"kubernetes.io/projected/ed78ec8c-6582-4454-b810-9d37124a82a5-kube-api-access-t4t6h\") pod \"payload-pre-processing-e2e-worker-single-main-3a5889-5bd4dmj44f\" (UID: \"ed78ec8c-6582-4454-b810-9d37124a82a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4dmj44f" Oct 31 13:53:02.079483 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:02.079428 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b263bb54-a3b4-4cbc-a1cb-67612f01314a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj\" (UID: \"b263bb54-a3b4-4cbc-a1cb-67612f01314a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj" Oct 31 13:53:02.079483 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:02.079469 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0082bcdc-04ae-43b5-82ad-9a5052f535f2-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-3a5889-cc4947495qd7jn\" (UID: \"0082bcdc-04ae-43b5-82ad-9a5052f535f2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495qd7jn" Oct 31 13:53:02.079734 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:02.079646 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed78ec8c-6582-4454-b810-9d37124a82a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3a5889-5bd4dmj44f\" (UID: \"ed78ec8c-6582-4454-b810-9d37124a82a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4dmj44f" Oct 31 13:53:02.079734 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:02.079712 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b263bb54-a3b4-4cbc-a1cb-67612f01314a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj\" (UID: \"b263bb54-a3b4-4cbc-a1cb-67612f01314a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj" Oct 31 13:53:02.080052 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:02.080024 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b5jp\" (UniqueName: \"kubernetes.io/projected/b263bb54-a3b4-4cbc-a1cb-67612f01314a-kube-api-access-6b5jp\") pod \"maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj\" (UID: \"b263bb54-a3b4-4cbc-a1cb-67612f01314a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj" Oct 31 13:53:02.080177 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:02.080094 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slprv\" (UniqueName: \"kubernetes.io/projected/0082bcdc-04ae-43b5-82ad-9a5052f535f2-kube-api-access-slprv\") pod \"payload-processing-e2e-worker-single-main-3a5889-cc4947495qd7jn\" (UID: \"0082bcdc-04ae-43b5-82ad-9a5052f535f2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495qd7jn" Oct 31 13:53:02.080598 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:02.080574 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b263bb54-a3b4-4cbc-a1cb-67612f01314a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj\" (UID: \"b263bb54-a3b4-4cbc-a1cb-67612f01314a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj" Oct 31 13:53:02.080658 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:02.080633 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0082bcdc-04ae-43b5-82ad-9a5052f535f2-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-3a5889-cc4947495qd7jn\" (UID: \"0082bcdc-04ae-43b5-82ad-9a5052f535f2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495qd7jn" Oct 31 13:53:02.080697 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:02.080663 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed78ec8c-6582-4454-b810-9d37124a82a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3a5889-5bd4dmj44f\" (UID: \"ed78ec8c-6582-4454-b810-9d37124a82a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4dmj44f" Oct 31 13:53:02.082136 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:02.082115 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b263bb54-a3b4-4cbc-a1cb-67612f01314a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj\" (UID: \"b263bb54-a3b4-4cbc-a1cb-67612f01314a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj" Oct 31 13:53:02.082279 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:02.082263 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b263bb54-a3b4-4cbc-a1cb-67612f01314a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj\" (UID: \"b263bb54-a3b4-4cbc-a1cb-67612f01314a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj" Oct 31 13:53:02.091676 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:02.091649 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4t6h\" (UniqueName: \"kubernetes.io/projected/ed78ec8c-6582-4454-b810-9d37124a82a5-kube-api-access-t4t6h\") pod \"payload-pre-processing-e2e-worker-single-main-3a5889-5bd4dmj44f\" (UID: \"ed78ec8c-6582-4454-b810-9d37124a82a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4dmj44f" Oct 31 13:53:02.091783 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:02.091766 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slprv\" (UniqueName: \"kubernetes.io/projected/0082bcdc-04ae-43b5-82ad-9a5052f535f2-kube-api-access-slprv\") pod \"payload-processing-e2e-worker-single-main-3a5889-cc4947495qd7jn\" (UID: \"0082bcdc-04ae-43b5-82ad-9a5052f535f2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495qd7jn" Oct 31 13:53:02.092336 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:02.092318 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b5jp\" (UniqueName: \"kubernetes.io/projected/b263bb54-a3b4-4cbc-a1cb-67612f01314a-kube-api-access-6b5jp\") pod \"maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj\" (UID: \"b263bb54-a3b4-4cbc-a1cb-67612f01314a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj" Oct 31 13:53:02.108122 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:02.108103 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj" Oct 31 13:53:02.152545 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:02.152513 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4dmj44f" Oct 31 13:53:02.211460 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:02.210594 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495qd7jn" Oct 31 13:53:02.387094 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:02.387064 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj"] Oct 31 13:53:02.389136 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:53:02.389110 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb263bb54_a3b4_4cbc_a1cb_67612f01314a.slice/crio-0ae5daade21206fe655b7a64add8cc998e3e99110315fca29db3ec935496a770 WatchSource:0}: Error finding container 0ae5daade21206fe655b7a64add8cc998e3e99110315fca29db3ec935496a770: Status 404 returned error can't find the container with id 0ae5daade21206fe655b7a64add8cc998e3e99110315fca29db3ec935496a770 Oct 31 13:53:02.390464 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:02.390436 6841 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 13:53:02.435840 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:02.435815 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4dmj44f"] Oct 31 13:53:02.437073 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:53:02.437035 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded78ec8c_6582_4454_b810_9d37124a82a5.slice/crio-a4657af5277b9bc6991aaa43d0c0156b1fb683dbd3376d0999e14173627ba5d1 WatchSource:0}: Error finding container a4657af5277b9bc6991aaa43d0c0156b1fb683dbd3376d0999e14173627ba5d1: Status 404 returned error can't find the container with id a4657af5277b9bc6991aaa43d0c0156b1fb683dbd3376d0999e14173627ba5d1 Oct 31 13:53:02.499437 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:02.499412 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495qd7jn"] Oct 31 13:53:02.500599 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:53:02.500574 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0082bcdc_04ae_43b5_82ad_9a5052f535f2.slice/crio-27fe5738cc4fc3db8508ef18dd46641785979a5e3efda1558b8ace0f022b92dd WatchSource:0}: Error finding container 27fe5738cc4fc3db8508ef18dd46641785979a5e3efda1558b8ace0f022b92dd: Status 404 returned error can't find the container with id 27fe5738cc4fc3db8508ef18dd46641785979a5e3efda1558b8ace0f022b92dd Oct 31 13:53:02.649232 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:02.649140 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4dmj44f" event={"ID":"ed78ec8c-6582-4454-b810-9d37124a82a5","Type":"ContainerStarted","Data":"93999161036a460fb0e315d4b9bc9c3b139ff45c337cb8573210ff6a951df1f4"} Oct 31 13:53:02.649232 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:02.649202 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4dmj44f" event={"ID":"ed78ec8c-6582-4454-b810-9d37124a82a5","Type":"ContainerStarted","Data":"a4657af5277b9bc6991aaa43d0c0156b1fb683dbd3376d0999e14173627ba5d1"} Oct 31 13:53:02.649487 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:02.649314 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4dmj44f" Oct 31 13:53:02.650491 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:02.650461 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj" event={"ID":"b263bb54-a3b4-4cbc-a1cb-67612f01314a","Type":"ContainerStarted","Data":"0ae5daade21206fe655b7a64add8cc998e3e99110315fca29db3ec935496a770"} Oct 31 13:53:02.652034 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:02.652011 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495qd7jn" event={"ID":"0082bcdc-04ae-43b5-82ad-9a5052f535f2","Type":"ContainerStarted","Data":"5125a50bf41b442abbebdbb0c6f244871c48d5bdde99dcd2b915d48b498c80df"} Oct 31 13:53:02.652142 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:02.652059 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495qd7jn" event={"ID":"0082bcdc-04ae-43b5-82ad-9a5052f535f2","Type":"ContainerStarted","Data":"27fe5738cc4fc3db8508ef18dd46641785979a5e3efda1558b8ace0f022b92dd"} Oct 31 13:53:02.652142 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:02.652090 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495qd7jn" Oct 31 13:53:02.669313 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:02.669267 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4dmj44f" podStartSLOduration=1.669254248 podStartE2EDuration="1.669254248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:53:02.666875747 +0000 UTC m=+4829.149998647" watchObservedRunningTime="2026-10-31 13:53:02.669254248 +0000 UTC m=+4829.152377149" Oct 31 13:53:02.675236 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:02.675215 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:02.686970 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:02.686929 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495qd7jn" podStartSLOduration=1.686916059 podStartE2EDuration="1.686916059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:53:02.685066273 +0000 UTC m=+4829.168189177" watchObservedRunningTime="2026-10-31 13:53:02.686916059 +0000 UTC m=+4829.170038960" Oct 31 13:53:03.382578 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:03.382529 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:03.659232 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:03.659136 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj" event={"ID":"b263bb54-a3b4-4cbc-a1cb-67612f01314a","Type":"ContainerStarted","Data":"e0c89a2d47acc714439e1e9d92c7e0bc65fc5ea07263fb0eb9bba05514c3e671"} Oct 31 13:53:03.659442 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:03.659422 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj" Oct 31 13:53:03.679437 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:03.679387 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj" podStartSLOduration=2.155976909 podStartE2EDuration="2.679371162s" totalImagesPullingTime="523.394253ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:53:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:53:03.678948219 +0000 UTC m=+4830.162071120" watchObservedRunningTime="2026-10-31 13:53:03.679371162 +0000 UTC m=+4830.162494063" Oct 31 13:53:04.080263 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:04.080230 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:04.818808 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:04.818772 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:05.588605 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:05.588574 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:06.117573 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:06.117535 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:06.315585 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:06.315541 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:06.610527 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:06.610491 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:07.186190 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:07.186151 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:07.976976 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:07.976934 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:08.680188 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:08.680147 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:09.370922 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:09.370887 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:09.672216 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:09.672043 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj" Oct 31 13:53:10.186371 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:10.186342 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:10.894432 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:10.894388 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:11.769958 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:11.769914 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:12.466491 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:12.466453 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:12.716733 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:12.716643 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:12.917775 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:12.917735 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:13.173087 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:13.172999 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:13.662424 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:13.662396 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495qd7jn" Oct 31 13:53:13.662621 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:13.662456 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4dmj44f" Oct 31 13:53:13.881388 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:13.881354 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:14.580725 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:14.580681 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:15.272713 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:15.272678 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:15.970670 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:15.970637 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:16.742330 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:16.742277 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:17.375911 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:17.375877 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:18.070573 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:18.070539 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:18.780731 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:18.780688 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:19.482891 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:19.482853 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:20.034459 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:53:20.034424 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:53:20.039097 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:20.039069 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495qd7jn"] Oct 31 13:53:20.054954 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:20.054924 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4dmj44f"] Oct 31 13:53:20.178564 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:20.178533 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:20.383260 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:20.383168 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:20.551269 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:20.551233 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj"] Oct 31 13:53:20.551506 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:20.551475 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj" podUID="b263bb54-a3b4-4cbc-a1cb-67612f01314a" containerName="maas-api" containerID="cri-o://e0c89a2d47acc714439e1e9d92c7e0bc65fc5ea07263fb0eb9bba05514c3e671" gracePeriod=30 Oct 31 13:53:20.703585 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:20.703552 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:20.793269 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:20.793239 6841 generic.go:356] "Generic (PLEG): container finished" podID="b263bb54-a3b4-4cbc-a1cb-67612f01314a" containerID="e0c89a2d47acc714439e1e9d92c7e0bc65fc5ea07263fb0eb9bba05514c3e671" exitCode=0 Oct 31 13:53:20.793472 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:20.793287 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj" event={"ID":"b263bb54-a3b4-4cbc-a1cb-67612f01314a","Type":"ContainerDied","Data":"e0c89a2d47acc714439e1e9d92c7e0bc65fc5ea07263fb0eb9bba05514c3e671"} Oct 31 13:53:20.830393 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:20.830370 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj" Oct 31 13:53:20.905373 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:20.905323 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:20.982420 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:20.982375 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b263bb54-a3b4-4cbc-a1cb-67612f01314a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b263bb54-a3b4-4cbc-a1cb-67612f01314a-metrics-certs\") pod \"b263bb54-a3b4-4cbc-a1cb-67612f01314a\" (UID: \"b263bb54-a3b4-4cbc-a1cb-67612f01314a\") " Oct 31 13:53:20.982611 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:20.982567 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b263bb54-a3b4-4cbc-a1cb-67612f01314a-kube-api-access-6b5jp\" (UniqueName: \"kubernetes.io/projected/b263bb54-a3b4-4cbc-a1cb-67612f01314a-kube-api-access-6b5jp\") pod \"b263bb54-a3b4-4cbc-a1cb-67612f01314a\" (UID: \"b263bb54-a3b4-4cbc-a1cb-67612f01314a\") " Oct 31 13:53:20.982687 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:20.982639 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b263bb54-a3b4-4cbc-a1cb-67612f01314a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b263bb54-a3b4-4cbc-a1cb-67612f01314a-openshift-service-ca\") pod \"b263bb54-a3b4-4cbc-a1cb-67612f01314a\" (UID: \"b263bb54-a3b4-4cbc-a1cb-67612f01314a\") " Oct 31 13:53:20.982786 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:20.982684 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b263bb54-a3b4-4cbc-a1cb-67612f01314a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b263bb54-a3b4-4cbc-a1cb-67612f01314a-maas-api-tls\") pod \"b263bb54-a3b4-4cbc-a1cb-67612f01314a\" (UID: \"b263bb54-a3b4-4cbc-a1cb-67612f01314a\") " Oct 31 13:53:20.983094 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:20.983061 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b263bb54-a3b4-4cbc-a1cb-67612f01314a-openshift-service-ca" pod "b263bb54-a3b4-4cbc-a1cb-67612f01314a" (UID: "b263bb54-a3b4-4cbc-a1cb-67612f01314a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:53:20.985048 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:20.984977 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b263bb54-a3b4-4cbc-a1cb-67612f01314a-maas-api-tls" pod "b263bb54-a3b4-4cbc-a1cb-67612f01314a" (UID: "b263bb54-a3b4-4cbc-a1cb-67612f01314a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:53:20.985160 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:20.985043 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b263bb54-a3b4-4cbc-a1cb-67612f01314a-metrics-certs" pod "b263bb54-a3b4-4cbc-a1cb-67612f01314a" (UID: "b263bb54-a3b4-4cbc-a1cb-67612f01314a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:53:20.985274 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:20.985256 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b263bb54-a3b4-4cbc-a1cb-67612f01314a-kube-api-access-6b5jp" pod "b263bb54-a3b4-4cbc-a1cb-67612f01314a" (UID: "b263bb54-a3b4-4cbc-a1cb-67612f01314a"). InnerVolumeSpecName "kube-api-access-6b5jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:53:21.083618 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:21.083584 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b5jp\" (UniqueName: \"kubernetes.io/projected/b263bb54-a3b4-4cbc-a1cb-67612f01314a-kube-api-access-6b5jp\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:21.083618 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:21.083615 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b263bb54-a3b4-4cbc-a1cb-67612f01314a-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:21.084068 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:21.083627 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b263bb54-a3b4-4cbc-a1cb-67612f01314a-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:21.084068 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:21.083636 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b263bb54-a3b4-4cbc-a1cb-67612f01314a-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:21.101257 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:21.101190 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:21.397133 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:21.397029 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:21.800861 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:21.800829 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj" event={"ID":"b263bb54-a3b4-4cbc-a1cb-67612f01314a","Type":"ContainerDied","Data":"0ae5daade21206fe655b7a64add8cc998e3e99110315fca29db3ec935496a770"} Oct 31 13:53:21.800861 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:21.800833 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj" Oct 31 13:53:21.801131 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:21.800903 6841 scope.go:122] "RemoveContainer" containerID="e0c89a2d47acc714439e1e9d92c7e0bc65fc5ea07263fb0eb9bba05514c3e671" Oct 31 13:53:21.818463 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:21.818438 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj"] Oct 31 13:53:21.822799 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:21.822771 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-2h5bj"] Oct 31 13:53:23.648136 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:23.648101 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b263bb54-a3b4-4cbc-a1cb-67612f01314a" path="/var/lib/kubelet/pods/b263bb54-a3b4-4cbc-a1cb-67612f01314a/volumes" Oct 31 13:53:25.040238 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.040194 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495qd7jn" podUID="0082bcdc-04ae-43b5-82ad-9a5052f535f2" containerName="payload-processing" containerID="cri-o://5125a50bf41b442abbebdbb0c6f244871c48d5bdde99dcd2b915d48b498c80df" gracePeriod=25 Oct 31 13:53:25.056048 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.055965 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4dmj44f" podUID="ed78ec8c-6582-4454-b810-9d37124a82a5" containerName="payload-pre-processing" containerID="cri-o://93999161036a460fb0e315d4b9bc9c3b139ff45c337cb8573210ff6a951df1f4" gracePeriod=25 Oct 31 13:53:25.071500 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.071467 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw"] Oct 31 13:53:25.071906 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.071893 6841 cpu_manager.go:414] "containerMap: removing container" podUID="b263bb54-a3b4-4cbc-a1cb-67612f01314a" containerName="maas-api" Oct 31 13:53:25.071977 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.071907 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b263bb54-a3b4-4cbc-a1cb-67612f01314a" containerName="maas-api" Oct 31 13:53:25.072062 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.072019 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="b263bb54-a3b4-4cbc-a1cb-67612f01314a" containerName="maas-api" Oct 31 13:53:25.075435 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.075413 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw" Oct 31 13:53:25.084810 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.084760 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw"] Oct 31 13:53:25.086509 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.086472 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-9ebe6601-kserve-self-signed-certs\"" Oct 31 13:53:25.225607 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.225570 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bca485f-16b7-43e2-b4b3-f81997be011b-tls-certs\") pod \"e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw\" (UID: \"5bca485f-16b7-43e2-b4b3-f81997be011b\") " pod="llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw" Oct 31 13:53:25.225790 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.225636 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bca485f-16b7-43e2-b4b3-f81997be011b-home\") pod \"e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw\" (UID: \"5bca485f-16b7-43e2-b4b3-f81997be011b\") " pod="llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw" Oct 31 13:53:25.225790 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.225676 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bca485f-16b7-43e2-b4b3-f81997be011b-dshm\") pod \"e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw\" (UID: \"5bca485f-16b7-43e2-b4b3-f81997be011b\") " pod="llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw" Oct 31 13:53:25.225790 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.225695 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bca485f-16b7-43e2-b4b3-f81997be011b-model-cache\") pod \"e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw\" (UID: \"5bca485f-16b7-43e2-b4b3-f81997be011b\") " pod="llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw" Oct 31 13:53:25.225790 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.225755 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chjvt\" (UniqueName: \"kubernetes.io/projected/5bca485f-16b7-43e2-b4b3-f81997be011b-kube-api-access-chjvt\") pod \"e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw\" (UID: \"5bca485f-16b7-43e2-b4b3-f81997be011b\") " pod="llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw" Oct 31 13:53:25.226006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.225826 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bca485f-16b7-43e2-b4b3-f81997be011b-tmp-dir\") pod \"e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw\" (UID: \"5bca485f-16b7-43e2-b4b3-f81997be011b\") " pod="llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw" Oct 31 13:53:25.227667 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.227637 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:25.324118 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.324092 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495qd7jn" Oct 31 13:53:25.327238 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.327211 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chjvt\" (UniqueName: \"kubernetes.io/projected/5bca485f-16b7-43e2-b4b3-f81997be011b-kube-api-access-chjvt\") pod \"e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw\" (UID: \"5bca485f-16b7-43e2-b4b3-f81997be011b\") " pod="llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw" Oct 31 13:53:25.327340 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.327298 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bca485f-16b7-43e2-b4b3-f81997be011b-tmp-dir\") pod \"e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw\" (UID: \"5bca485f-16b7-43e2-b4b3-f81997be011b\") " pod="llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw" Oct 31 13:53:25.327404 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.327362 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bca485f-16b7-43e2-b4b3-f81997be011b-tls-certs\") pod \"e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw\" (UID: \"5bca485f-16b7-43e2-b4b3-f81997be011b\") " pod="llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw" Oct 31 13:53:25.327464 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.327411 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bca485f-16b7-43e2-b4b3-f81997be011b-home\") pod \"e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw\" (UID: \"5bca485f-16b7-43e2-b4b3-f81997be011b\") " pod="llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw" Oct 31 13:53:25.327464 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.327450 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bca485f-16b7-43e2-b4b3-f81997be011b-dshm\") pod \"e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw\" (UID: \"5bca485f-16b7-43e2-b4b3-f81997be011b\") " pod="llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw" Oct 31 13:53:25.327581 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.327473 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bca485f-16b7-43e2-b4b3-f81997be011b-model-cache\") pod \"e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw\" (UID: \"5bca485f-16b7-43e2-b4b3-f81997be011b\") " pod="llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw" Oct 31 13:53:25.327884 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.327838 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bca485f-16b7-43e2-b4b3-f81997be011b-home\") pod \"e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw\" (UID: \"5bca485f-16b7-43e2-b4b3-f81997be011b\") " pod="llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw" Oct 31 13:53:25.327884 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.327846 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bca485f-16b7-43e2-b4b3-f81997be011b-tmp-dir\") pod \"e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw\" (UID: \"5bca485f-16b7-43e2-b4b3-f81997be011b\") " pod="llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw" Oct 31 13:53:25.328086 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.327904 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bca485f-16b7-43e2-b4b3-f81997be011b-model-cache\") pod \"e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw\" (UID: \"5bca485f-16b7-43e2-b4b3-f81997be011b\") " pod="llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw" Oct 31 13:53:25.329738 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.329719 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bca485f-16b7-43e2-b4b3-f81997be011b-dshm\") pod \"e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw\" (UID: \"5bca485f-16b7-43e2-b4b3-f81997be011b\") " pod="llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw" Oct 31 13:53:25.330208 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.330185 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bca485f-16b7-43e2-b4b3-f81997be011b-tls-certs\") pod \"e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw\" (UID: \"5bca485f-16b7-43e2-b4b3-f81997be011b\") " pod="llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw" Oct 31 13:53:25.339674 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.339651 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chjvt\" (UniqueName: \"kubernetes.io/projected/5bca485f-16b7-43e2-b4b3-f81997be011b-kube-api-access-chjvt\") pod \"e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw\" (UID: \"5bca485f-16b7-43e2-b4b3-f81997be011b\") " pod="llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw" Oct 31 13:53:25.356798 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.356774 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4dmj44f" Oct 31 13:53:25.428009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.427961 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed78ec8c-6582-4454-b810-9d37124a82a5-kube-api-access-t4t6h\" (UniqueName: \"kubernetes.io/projected/ed78ec8c-6582-4454-b810-9d37124a82a5-kube-api-access-t4t6h\") pod \"ed78ec8c-6582-4454-b810-9d37124a82a5\" (UID: \"ed78ec8c-6582-4454-b810-9d37124a82a5\") " Oct 31 13:53:25.428187 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.428064 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0082bcdc-04ae-43b5-82ad-9a5052f535f2-kube-api-access-slprv\" (UniqueName: \"kubernetes.io/projected/0082bcdc-04ae-43b5-82ad-9a5052f535f2-kube-api-access-slprv\") pod \"0082bcdc-04ae-43b5-82ad-9a5052f535f2\" (UID: \"0082bcdc-04ae-43b5-82ad-9a5052f535f2\") " Oct 31 13:53:25.428187 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.428102 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0082bcdc-04ae-43b5-82ad-9a5052f535f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0082bcdc-04ae-43b5-82ad-9a5052f535f2-plugins-config-volume\") pod \"0082bcdc-04ae-43b5-82ad-9a5052f535f2\" (UID: \"0082bcdc-04ae-43b5-82ad-9a5052f535f2\") " Oct 31 13:53:25.428187 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.428124 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed78ec8c-6582-4454-b810-9d37124a82a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed78ec8c-6582-4454-b810-9d37124a82a5-plugins-config-volume\") pod \"ed78ec8c-6582-4454-b810-9d37124a82a5\" (UID: \"ed78ec8c-6582-4454-b810-9d37124a82a5\") " Oct 31 13:53:25.428534 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.428497 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0082bcdc-04ae-43b5-82ad-9a5052f535f2-plugins-config-volume" pod "0082bcdc-04ae-43b5-82ad-9a5052f535f2" (UID: "0082bcdc-04ae-43b5-82ad-9a5052f535f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:53:25.428639 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.428529 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed78ec8c-6582-4454-b810-9d37124a82a5-plugins-config-volume" pod "ed78ec8c-6582-4454-b810-9d37124a82a5" (UID: "ed78ec8c-6582-4454-b810-9d37124a82a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:53:25.430279 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.430247 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed78ec8c-6582-4454-b810-9d37124a82a5-kube-api-access-t4t6h" pod "ed78ec8c-6582-4454-b810-9d37124a82a5" (UID: "ed78ec8c-6582-4454-b810-9d37124a82a5"). InnerVolumeSpecName "kube-api-access-t4t6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:53:25.430395 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.430272 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0082bcdc-04ae-43b5-82ad-9a5052f535f2-kube-api-access-slprv" pod "0082bcdc-04ae-43b5-82ad-9a5052f535f2" (UID: "0082bcdc-04ae-43b5-82ad-9a5052f535f2"). InnerVolumeSpecName "kube-api-access-slprv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:53:25.490186 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.490150 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw" Oct 31 13:53:25.530112 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.529776 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0082bcdc-04ae-43b5-82ad-9a5052f535f2-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:25.530112 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.529814 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed78ec8c-6582-4454-b810-9d37124a82a5-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:25.530112 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.529834 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4t6h\" (UniqueName: \"kubernetes.io/projected/ed78ec8c-6582-4454-b810-9d37124a82a5-kube-api-access-t4t6h\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:25.530112 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.529849 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slprv\" (UniqueName: \"kubernetes.io/projected/0082bcdc-04ae-43b5-82ad-9a5052f535f2-kube-api-access-slprv\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:25.767214 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.767188 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw"] Oct 31 13:53:25.769947 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:53:25.769918 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5bca485f_16b7_43e2_b4b3_f81997be011b.slice/crio-fe1098d3abc7fe334be5621034e157d69e695075d3aebf7bb8b3b19653c47548 WatchSource:0}: Error finding container fe1098d3abc7fe334be5621034e157d69e695075d3aebf7bb8b3b19653c47548: Status 404 returned error can't find the container with id fe1098d3abc7fe334be5621034e157d69e695075d3aebf7bb8b3b19653c47548 Oct 31 13:53:25.826175 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.826150 6841 generic.go:356] "Generic (PLEG): container finished" podID="0082bcdc-04ae-43b5-82ad-9a5052f535f2" containerID="5125a50bf41b442abbebdbb0c6f244871c48d5bdde99dcd2b915d48b498c80df" exitCode=0 Oct 31 13:53:25.826303 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.826189 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495qd7jn" Oct 31 13:53:25.826303 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.826230 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495qd7jn" event={"ID":"0082bcdc-04ae-43b5-82ad-9a5052f535f2","Type":"ContainerDied","Data":"5125a50bf41b442abbebdbb0c6f244871c48d5bdde99dcd2b915d48b498c80df"} Oct 31 13:53:25.826303 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.826285 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495qd7jn" event={"ID":"0082bcdc-04ae-43b5-82ad-9a5052f535f2","Type":"ContainerDied","Data":"27fe5738cc4fc3db8508ef18dd46641785979a5e3efda1558b8ace0f022b92dd"} Oct 31 13:53:25.826485 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.826310 6841 scope.go:122] "RemoveContainer" containerID="5125a50bf41b442abbebdbb0c6f244871c48d5bdde99dcd2b915d48b498c80df" Oct 31 13:53:25.828186 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.828157 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw" event={"ID":"5bca485f-16b7-43e2-b4b3-f81997be011b","Type":"ContainerStarted","Data":"fe1098d3abc7fe334be5621034e157d69e695075d3aebf7bb8b3b19653c47548"} Oct 31 13:53:25.829774 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.829752 6841 generic.go:356] "Generic (PLEG): container finished" podID="ed78ec8c-6582-4454-b810-9d37124a82a5" containerID="93999161036a460fb0e315d4b9bc9c3b139ff45c337cb8573210ff6a951df1f4" exitCode=0 Oct 31 13:53:25.829895 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.829781 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4dmj44f" Oct 31 13:53:25.829895 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.829847 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4dmj44f" event={"ID":"ed78ec8c-6582-4454-b810-9d37124a82a5","Type":"ContainerDied","Data":"93999161036a460fb0e315d4b9bc9c3b139ff45c337cb8573210ff6a951df1f4"} Oct 31 13:53:25.830042 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.829897 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4dmj44f" event={"ID":"ed78ec8c-6582-4454-b810-9d37124a82a5","Type":"ContainerDied","Data":"a4657af5277b9bc6991aaa43d0c0156b1fb683dbd3376d0999e14173627ba5d1"} Oct 31 13:53:25.837203 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.837178 6841 scope.go:122] "RemoveContainer" containerID="5125a50bf41b442abbebdbb0c6f244871c48d5bdde99dcd2b915d48b498c80df" Oct 31 13:53:25.837489 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:53:25.837466 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5125a50bf41b442abbebdbb0c6f244871c48d5bdde99dcd2b915d48b498c80df\": container with ID starting with 5125a50bf41b442abbebdbb0c6f244871c48d5bdde99dcd2b915d48b498c80df not found: ID does not exist" containerID="5125a50bf41b442abbebdbb0c6f244871c48d5bdde99dcd2b915d48b498c80df" Oct 31 13:53:25.837594 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.837493 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5125a50bf41b442abbebdbb0c6f244871c48d5bdde99dcd2b915d48b498c80df"} err="failed to get container status \"5125a50bf41b442abbebdbb0c6f244871c48d5bdde99dcd2b915d48b498c80df\": rpc error: code = NotFound desc = could not find container \"5125a50bf41b442abbebdbb0c6f244871c48d5bdde99dcd2b915d48b498c80df\": container with ID starting with 5125a50bf41b442abbebdbb0c6f244871c48d5bdde99dcd2b915d48b498c80df not found: ID does not exist" Oct 31 13:53:25.837594 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.837509 6841 scope.go:122] "RemoveContainer" containerID="93999161036a460fb0e315d4b9bc9c3b139ff45c337cb8573210ff6a951df1f4" Oct 31 13:53:25.847108 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.847089 6841 scope.go:122] "RemoveContainer" containerID="93999161036a460fb0e315d4b9bc9c3b139ff45c337cb8573210ff6a951df1f4" Oct 31 13:53:25.847432 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:53:25.847406 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93999161036a460fb0e315d4b9bc9c3b139ff45c337cb8573210ff6a951df1f4\": container with ID starting with 93999161036a460fb0e315d4b9bc9c3b139ff45c337cb8573210ff6a951df1f4 not found: ID does not exist" containerID="93999161036a460fb0e315d4b9bc9c3b139ff45c337cb8573210ff6a951df1f4" Oct 31 13:53:25.847513 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.847445 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93999161036a460fb0e315d4b9bc9c3b139ff45c337cb8573210ff6a951df1f4"} err="failed to get container status \"93999161036a460fb0e315d4b9bc9c3b139ff45c337cb8573210ff6a951df1f4\": rpc error: code = NotFound desc = could not find container \"93999161036a460fb0e315d4b9bc9c3b139ff45c337cb8573210ff6a951df1f4\": container with ID starting with 93999161036a460fb0e315d4b9bc9c3b139ff45c337cb8573210ff6a951df1f4 not found: ID does not exist" Oct 31 13:53:25.848994 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.848960 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495qd7jn"] Oct 31 13:53:25.853921 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.853899 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495qd7jn"] Oct 31 13:53:25.863347 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.863326 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4dmj44f"] Oct 31 13:53:25.868407 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:25.868382 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4dmj44f"] Oct 31 13:53:26.023293 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:26.023262 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wj64z"] Oct 31 13:53:26.028251 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:26.028225 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wj64z"] Oct 31 13:53:26.837818 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:26.837781 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw" event={"ID":"5bca485f-16b7-43e2-b4b3-f81997be011b","Type":"ContainerStarted","Data":"3c354aaf4f2c6eb0e15d262b1280da053be929a9b560b58d9219a98e9ed71799"} Oct 31 13:53:26.860627 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:26.860576 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw" podStartSLOduration=1.860563942 podStartE2EDuration="1.860563942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:53:26.857564197 +0000 UTC m=+4853.340687100" watchObservedRunningTime="2026-10-31 13:53:26.860563942 +0000 UTC m=+4853.343686842" Oct 31 13:53:27.490903 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:27.490867 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw" Oct 31 13:53:27.645248 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:27.645208 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0082bcdc-04ae-43b5-82ad-9a5052f535f2" path="/var/lib/kubelet/pods/0082bcdc-04ae-43b5-82ad-9a5052f535f2/volumes" Oct 31 13:53:27.645870 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:27.645843 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31c6b47f-a737-4c7e-a4c0-81746942fb89" path="/var/lib/kubelet/pods/31c6b47f-a737-4c7e-a4c0-81746942fb89/volumes" Oct 31 13:53:27.646401 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:27.646377 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed78ec8c-6582-4454-b810-9d37124a82a5" path="/var/lib/kubelet/pods/ed78ec8c-6582-4454-b810-9d37124a82a5/volumes" Oct 31 13:53:35.018239 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:35.018203 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:35.491031 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:35.490992 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw" Oct 31 13:53:35.503779 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:35.503749 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw" Oct 31 13:53:35.934873 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:35.934789 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw" Oct 31 13:53:41.513435 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:41.513400 6841 scope.go:122] "RemoveContainer" containerID="0927e7fad0afff766a5b5d0d531b8a68b17e6c944a71a9b94c2d5b1248ed5f33" Oct 31 13:53:41.522715 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:41.522697 6841 scope.go:122] "RemoveContainer" containerID="45c03f04514571518ded9f50f0e079892c8b684a54916580d613b67f92f8f82c" Oct 31 13:53:41.608000 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:41.607916 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:43.828537 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:43.828495 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg"] Oct 31 13:53:43.829250 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:43.829231 6841 cpu_manager.go:414] "containerMap: removing container" podUID="0082bcdc-04ae-43b5-82ad-9a5052f535f2" containerName="payload-processing" Oct 31 13:53:43.829305 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:43.829251 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0082bcdc-04ae-43b5-82ad-9a5052f535f2" containerName="payload-processing" Oct 31 13:53:43.829305 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:43.829294 6841 cpu_manager.go:414] "containerMap: removing container" podUID="ed78ec8c-6582-4454-b810-9d37124a82a5" containerName="payload-pre-processing" Oct 31 13:53:43.829305 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:43.829300 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed78ec8c-6582-4454-b810-9d37124a82a5" containerName="payload-pre-processing" Oct 31 13:53:43.829410 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:43.829369 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="0082bcdc-04ae-43b5-82ad-9a5052f535f2" containerName="payload-processing" Oct 31 13:53:43.829410 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:43.829381 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed78ec8c-6582-4454-b810-9d37124a82a5" containerName="payload-pre-processing" Oct 31 13:53:43.834434 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:43.834414 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg" Oct 31 13:53:43.839627 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:43.839583 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-9ebe6601-kserve-self-signed-certs\"" Oct 31 13:53:43.843139 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:43.843115 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg"] Oct 31 13:53:43.936735 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:43.936704 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99d41763-e41c-417a-9bbe-b59bc95cf39b-dshm\") pod \"e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg\" (UID: \"99d41763-e41c-417a-9bbe-b59bc95cf39b\") " pod="llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg" Oct 31 13:53:43.936925 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:43.936741 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99d41763-e41c-417a-9bbe-b59bc95cf39b-home\") pod \"e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg\" (UID: \"99d41763-e41c-417a-9bbe-b59bc95cf39b\") " pod="llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg" Oct 31 13:53:43.936925 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:43.936769 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp22r\" (UniqueName: \"kubernetes.io/projected/99d41763-e41c-417a-9bbe-b59bc95cf39b-kube-api-access-zp22r\") pod \"e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg\" (UID: \"99d41763-e41c-417a-9bbe-b59bc95cf39b\") " pod="llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg" Oct 31 13:53:43.936925 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:43.936911 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99d41763-e41c-417a-9bbe-b59bc95cf39b-model-cache\") pod \"e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg\" (UID: \"99d41763-e41c-417a-9bbe-b59bc95cf39b\") " pod="llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg" Oct 31 13:53:43.937089 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:43.936947 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99d41763-e41c-417a-9bbe-b59bc95cf39b-tmp-dir\") pod \"e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg\" (UID: \"99d41763-e41c-417a-9bbe-b59bc95cf39b\") " pod="llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg" Oct 31 13:53:43.937089 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:43.937007 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99d41763-e41c-417a-9bbe-b59bc95cf39b-tls-certs\") pod \"e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg\" (UID: \"99d41763-e41c-417a-9bbe-b59bc95cf39b\") " pod="llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg" Oct 31 13:53:43.974136 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:43.974094 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:53:44.038403 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:44.038367 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99d41763-e41c-417a-9bbe-b59bc95cf39b-dshm\") pod \"e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg\" (UID: \"99d41763-e41c-417a-9bbe-b59bc95cf39b\") " pod="llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg" Oct 31 13:53:44.038403 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:44.038406 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99d41763-e41c-417a-9bbe-b59bc95cf39b-home\") pod \"e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg\" (UID: \"99d41763-e41c-417a-9bbe-b59bc95cf39b\") " pod="llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg" Oct 31 13:53:44.038628 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:44.038433 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp22r\" (UniqueName: \"kubernetes.io/projected/99d41763-e41c-417a-9bbe-b59bc95cf39b-kube-api-access-zp22r\") pod \"e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg\" (UID: \"99d41763-e41c-417a-9bbe-b59bc95cf39b\") " pod="llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg" Oct 31 13:53:44.038628 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:44.038485 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99d41763-e41c-417a-9bbe-b59bc95cf39b-model-cache\") pod \"e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg\" (UID: \"99d41763-e41c-417a-9bbe-b59bc95cf39b\") " pod="llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg" Oct 31 13:53:44.038628 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:44.038516 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99d41763-e41c-417a-9bbe-b59bc95cf39b-tmp-dir\") pod \"e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg\" (UID: \"99d41763-e41c-417a-9bbe-b59bc95cf39b\") " pod="llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg" Oct 31 13:53:44.038628 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:44.038557 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99d41763-e41c-417a-9bbe-b59bc95cf39b-tls-certs\") pod \"e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg\" (UID: \"99d41763-e41c-417a-9bbe-b59bc95cf39b\") " pod="llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg" Oct 31 13:53:44.038851 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:44.038831 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99d41763-e41c-417a-9bbe-b59bc95cf39b-home\") pod \"e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg\" (UID: \"99d41763-e41c-417a-9bbe-b59bc95cf39b\") " pod="llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg" Oct 31 13:53:44.038923 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:44.038897 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99d41763-e41c-417a-9bbe-b59bc95cf39b-model-cache\") pod \"e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg\" (UID: \"99d41763-e41c-417a-9bbe-b59bc95cf39b\") " pod="llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg" Oct 31 13:53:44.038978 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:44.038963 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99d41763-e41c-417a-9bbe-b59bc95cf39b-tmp-dir\") pod \"e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg\" (UID: \"99d41763-e41c-417a-9bbe-b59bc95cf39b\") " pod="llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg" Oct 31 13:53:44.040808 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:44.040782 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99d41763-e41c-417a-9bbe-b59bc95cf39b-dshm\") pod \"e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg\" (UID: \"99d41763-e41c-417a-9bbe-b59bc95cf39b\") " pod="llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg" Oct 31 13:53:44.041012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:44.040976 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99d41763-e41c-417a-9bbe-b59bc95cf39b-tls-certs\") pod \"e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg\" (UID: \"99d41763-e41c-417a-9bbe-b59bc95cf39b\") " pod="llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg" Oct 31 13:53:44.051349 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:44.051324 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp22r\" (UniqueName: \"kubernetes.io/projected/99d41763-e41c-417a-9bbe-b59bc95cf39b-kube-api-access-zp22r\") pod \"e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg\" (UID: \"99d41763-e41c-417a-9bbe-b59bc95cf39b\") " pod="llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg" Oct 31 13:53:44.151420 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:44.151344 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg" Oct 31 13:53:44.427326 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:44.427300 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg"] Oct 31 13:53:44.428885 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:53:44.428855 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99d41763_e41c_417a_9bbe_b59bc95cf39b.slice/crio-18cbb96afef113c4c63da47fc36ff9356ffda4e2a1a8b5411e656438e19f41c2 WatchSource:0}: Error finding container 18cbb96afef113c4c63da47fc36ff9356ffda4e2a1a8b5411e656438e19f41c2: Status 404 returned error can't find the container with id 18cbb96afef113c4c63da47fc36ff9356ffda4e2a1a8b5411e656438e19f41c2 Oct 31 13:53:44.978917 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:44.978880 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg" event={"ID":"99d41763-e41c-417a-9bbe-b59bc95cf39b","Type":"ContainerStarted","Data":"b1c70bd947e2501e01686606a7eb777eeab61629176eeb9adf4fab3d78c1cf2a"} Oct 31 13:53:44.979373 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:44.978930 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg" event={"ID":"99d41763-e41c-417a-9bbe-b59bc95cf39b","Type":"ContainerStarted","Data":"18cbb96afef113c4c63da47fc36ff9356ffda4e2a1a8b5411e656438e19f41c2"} Oct 31 13:53:44.999375 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:44.999329 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg" podStartSLOduration=1.9993154560000002 podStartE2EDuration="1.999315456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:53:44.998439245 +0000 UTC m=+4871.481562158" watchObservedRunningTime="2026-10-31 13:53:44.999315456 +0000 UTC m=+4871.482438357" Oct 31 13:53:45.152081 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:45.152044 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg" Oct 31 13:53:48.620191 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:48.620092 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-xn5wx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:53:48.620928 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:48.620868 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" podUID="2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:53:48.862006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:48.861081 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-bp2d4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:53:48.862006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:48.861145 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" podUID="51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:53:50.034680 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:53:50.034644 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:53:54.151475 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:54.151439 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg" Oct 31 13:53:54.168682 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:54.168648 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg" Oct 31 13:53:55.083579 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:55.083545 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg" Oct 31 13:53:59.125677 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:59.125640 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xfsql"] Oct 31 13:53:59.129975 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:59.129952 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xfsql" Oct 31 13:53:59.135722 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:59.135672 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:53:59.137135 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:59.137082 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:53:59.137305 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:59.137280 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:53:59.138639 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:59.137883 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 13:53:59.140793 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:59.140771 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xfsql"] Oct 31 13:53:59.241892 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:59.241853 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b759a18-e9b9-444a-84fa-d38a5201cdd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-xfsql\" (UID: \"0b759a18-e9b9-444a-84fa-d38a5201cdd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xfsql" Oct 31 13:53:59.242112 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:59.242022 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6nvs\" (UniqueName: \"kubernetes.io/projected/0b759a18-e9b9-444a-84fa-d38a5201cdd7-kube-api-access-m6nvs\") pod \"maas-api-revoke-sub-452b6d7b-xfsql\" (UID: \"0b759a18-e9b9-444a-84fa-d38a5201cdd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xfsql" Oct 31 13:53:59.242112 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:59.242060 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b759a18-e9b9-444a-84fa-d38a5201cdd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-xfsql\" (UID: \"0b759a18-e9b9-444a-84fa-d38a5201cdd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xfsql" Oct 31 13:53:59.343445 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:59.343408 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b759a18-e9b9-444a-84fa-d38a5201cdd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-xfsql\" (UID: \"0b759a18-e9b9-444a-84fa-d38a5201cdd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xfsql" Oct 31 13:53:59.343624 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:59.343509 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6nvs\" (UniqueName: \"kubernetes.io/projected/0b759a18-e9b9-444a-84fa-d38a5201cdd7-kube-api-access-m6nvs\") pod \"maas-api-revoke-sub-452b6d7b-xfsql\" (UID: \"0b759a18-e9b9-444a-84fa-d38a5201cdd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xfsql" Oct 31 13:53:59.343624 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:59.343537 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b759a18-e9b9-444a-84fa-d38a5201cdd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-xfsql\" (UID: \"0b759a18-e9b9-444a-84fa-d38a5201cdd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xfsql" Oct 31 13:53:59.344193 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:59.344166 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b759a18-e9b9-444a-84fa-d38a5201cdd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-xfsql\" (UID: \"0b759a18-e9b9-444a-84fa-d38a5201cdd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xfsql" Oct 31 13:53:59.345976 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:59.345953 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b759a18-e9b9-444a-84fa-d38a5201cdd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-xfsql\" (UID: \"0b759a18-e9b9-444a-84fa-d38a5201cdd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xfsql" Oct 31 13:53:59.356293 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:59.356271 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6nvs\" (UniqueName: \"kubernetes.io/projected/0b759a18-e9b9-444a-84fa-d38a5201cdd7-kube-api-access-m6nvs\") pod \"maas-api-revoke-sub-452b6d7b-xfsql\" (UID: \"0b759a18-e9b9-444a-84fa-d38a5201cdd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xfsql" Oct 31 13:53:59.445871 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:59.445846 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xfsql" Oct 31 13:53:59.724367 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:53:59.724335 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xfsql"] Oct 31 13:53:59.725734 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:53:59.725701 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b759a18_e9b9_444a_84fa_d38a5201cdd7.slice/crio-2973f485dbd96891e1711bf81af163722f620c976593efd2663aef4e0628c0fc WatchSource:0}: Error finding container 2973f485dbd96891e1711bf81af163722f620c976593efd2663aef4e0628c0fc: Status 404 returned error can't find the container with id 2973f485dbd96891e1711bf81af163722f620c976593efd2663aef4e0628c0fc Oct 31 13:54:00.103052 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:00.103015 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xfsql" event={"ID":"0b759a18-e9b9-444a-84fa-d38a5201cdd7","Type":"ContainerStarted","Data":"437b7ef0f2a363f2bb98cb8365fe2bcca7a98a6e08060dbe4655587943d443a0"} Oct 31 13:54:00.103234 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:00.103080 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xfsql" event={"ID":"0b759a18-e9b9-444a-84fa-d38a5201cdd7","Type":"ContainerStarted","Data":"2973f485dbd96891e1711bf81af163722f620c976593efd2663aef4e0628c0fc"} Oct 31 13:54:00.123036 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:00.122972 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xfsql" podStartSLOduration=1.122958591 podStartE2EDuration="1.122958591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:53:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:54:00.122178391 +0000 UTC m=+4886.605301295" watchObservedRunningTime="2026-10-31 13:54:00.122958591 +0000 UTC m=+4886.606081491" Oct 31 13:54:01.113844 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:01.113807 6841 generic.go:356] "Generic (PLEG): container finished" podID="0b759a18-e9b9-444a-84fa-d38a5201cdd7" containerID="437b7ef0f2a363f2bb98cb8365fe2bcca7a98a6e08060dbe4655587943d443a0" exitCode=0 Oct 31 13:54:01.114420 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:01.113964 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xfsql" event={"ID":"0b759a18-e9b9-444a-84fa-d38a5201cdd7","Type":"ContainerDied","Data":"437b7ef0f2a363f2bb98cb8365fe2bcca7a98a6e08060dbe4655587943d443a0"} Oct 31 13:54:02.983955 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:02.983919 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xfsql" Oct 31 13:54:03.056006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:03.040540 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b759a18-e9b9-444a-84fa-d38a5201cdd7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b759a18-e9b9-444a-84fa-d38a5201cdd7-maas-api-xks-ca\") pod \"0b759a18-e9b9-444a-84fa-d38a5201cdd7\" (UID: \"0b759a18-e9b9-444a-84fa-d38a5201cdd7\") " Oct 31 13:54:03.056006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:03.040631 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b759a18-e9b9-444a-84fa-d38a5201cdd7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b759a18-e9b9-444a-84fa-d38a5201cdd7-maas-api-openshift-ca\") pod \"0b759a18-e9b9-444a-84fa-d38a5201cdd7\" (UID: \"0b759a18-e9b9-444a-84fa-d38a5201cdd7\") " Oct 31 13:54:03.056006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:03.040681 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b759a18-e9b9-444a-84fa-d38a5201cdd7-kube-api-access-m6nvs\" (UniqueName: \"kubernetes.io/projected/0b759a18-e9b9-444a-84fa-d38a5201cdd7-kube-api-access-m6nvs\") pod \"0b759a18-e9b9-444a-84fa-d38a5201cdd7\" (UID: \"0b759a18-e9b9-444a-84fa-d38a5201cdd7\") " Oct 31 13:54:03.056006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:03.053189 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b759a18-e9b9-444a-84fa-d38a5201cdd7-maas-api-openshift-ca" pod "0b759a18-e9b9-444a-84fa-d38a5201cdd7" (UID: "0b759a18-e9b9-444a-84fa-d38a5201cdd7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:54:03.128594 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:03.128556 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b759a18-e9b9-444a-84fa-d38a5201cdd7-kube-api-access-m6nvs" pod "0b759a18-e9b9-444a-84fa-d38a5201cdd7" (UID: "0b759a18-e9b9-444a-84fa-d38a5201cdd7"). InnerVolumeSpecName "kube-api-access-m6nvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:54:03.159752 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:03.159663 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b759a18-e9b9-444a-84fa-d38a5201cdd7-maas-api-openshift-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:54:03.160154 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:03.160133 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6nvs\" (UniqueName: \"kubernetes.io/projected/0b759a18-e9b9-444a-84fa-d38a5201cdd7-kube-api-access-m6nvs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:54:03.173399 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:03.173080 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xfsql" event={"ID":"0b759a18-e9b9-444a-84fa-d38a5201cdd7","Type":"ContainerDied","Data":"2973f485dbd96891e1711bf81af163722f620c976593efd2663aef4e0628c0fc"} Oct 31 13:54:03.173399 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:03.173145 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2973f485dbd96891e1711bf81af163722f620c976593efd2663aef4e0628c0fc" Oct 31 13:54:03.173399 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:03.173213 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xfsql" Oct 31 13:54:03.174722 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:03.174682 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b759a18-e9b9-444a-84fa-d38a5201cdd7-maas-api-xks-ca" pod "0b759a18-e9b9-444a-84fa-d38a5201cdd7" (UID: "0b759a18-e9b9-444a-84fa-d38a5201cdd7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:54:03.261625 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:03.261565 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b759a18-e9b9-444a-84fa-d38a5201cdd7-maas-api-xks-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:54:05.023307 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:05.023268 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-tk8j6"] Oct 31 13:54:05.025540 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:05.025510 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-tk8j6"] Oct 31 13:54:05.641291 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:05.641257 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54802583-fbdc-48a9-90de-262d5aecb544" path="/var/lib/kubelet/pods/54802583-fbdc-48a9-90de-262d5aecb544/volumes" Oct 31 13:54:09.998093 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:09.998049 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw"] Oct 31 13:54:10.144667 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:10.144629 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:54:11.582749 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:11.582715 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg"] Oct 31 13:54:11.715649 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:11.715603 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:54:15.319691 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:15.319654 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tn6ng"] Oct 31 13:54:15.320268 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:15.320249 6841 cpu_manager.go:414] "containerMap: removing container" podUID="0b759a18-e9b9-444a-84fa-d38a5201cdd7" containerName="revoke-keys" Oct 31 13:54:15.320336 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:15.320269 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b759a18-e9b9-444a-84fa-d38a5201cdd7" containerName="revoke-keys" Oct 31 13:54:15.320374 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:15.320357 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b759a18-e9b9-444a-84fa-d38a5201cdd7" containerName="revoke-keys" Oct 31 13:54:15.324661 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:15.324644 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tn6ng" Oct 31 13:54:15.329705 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:15.329683 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:54:15.330971 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:15.330928 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:54:15.331110 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:15.331092 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 13:54:15.331172 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:15.331110 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:54:15.334050 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:15.334030 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tn6ng"] Oct 31 13:54:15.450680 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:15.450638 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/803b53b4-4ae4-493c-ae16-9cac9f6025c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-tn6ng\" (UID: \"803b53b4-4ae4-493c-ae16-9cac9f6025c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tn6ng" Oct 31 13:54:15.450889 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:15.450688 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/803b53b4-4ae4-493c-ae16-9cac9f6025c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-tn6ng\" (UID: \"803b53b4-4ae4-493c-ae16-9cac9f6025c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tn6ng" Oct 31 13:54:15.450968 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:15.450881 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl4m6\" (UniqueName: \"kubernetes.io/projected/803b53b4-4ae4-493c-ae16-9cac9f6025c1-kube-api-access-dl4m6\") pod \"maas-api-revoke-sub-3e31a506-tn6ng\" (UID: \"803b53b4-4ae4-493c-ae16-9cac9f6025c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tn6ng" Oct 31 13:54:15.552238 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:15.552176 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl4m6\" (UniqueName: \"kubernetes.io/projected/803b53b4-4ae4-493c-ae16-9cac9f6025c1-kube-api-access-dl4m6\") pod \"maas-api-revoke-sub-3e31a506-tn6ng\" (UID: \"803b53b4-4ae4-493c-ae16-9cac9f6025c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tn6ng" Oct 31 13:54:15.552500 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:15.552375 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/803b53b4-4ae4-493c-ae16-9cac9f6025c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-tn6ng\" (UID: \"803b53b4-4ae4-493c-ae16-9cac9f6025c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tn6ng" Oct 31 13:54:15.552500 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:15.552409 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/803b53b4-4ae4-493c-ae16-9cac9f6025c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-tn6ng\" (UID: \"803b53b4-4ae4-493c-ae16-9cac9f6025c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tn6ng" Oct 31 13:54:15.553360 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:15.553328 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/803b53b4-4ae4-493c-ae16-9cac9f6025c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-tn6ng\" (UID: \"803b53b4-4ae4-493c-ae16-9cac9f6025c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tn6ng" Oct 31 13:54:15.555769 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:15.555740 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/803b53b4-4ae4-493c-ae16-9cac9f6025c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-tn6ng\" (UID: \"803b53b4-4ae4-493c-ae16-9cac9f6025c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tn6ng" Oct 31 13:54:15.565157 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:15.565127 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl4m6\" (UniqueName: \"kubernetes.io/projected/803b53b4-4ae4-493c-ae16-9cac9f6025c1-kube-api-access-dl4m6\") pod \"maas-api-revoke-sub-3e31a506-tn6ng\" (UID: \"803b53b4-4ae4-493c-ae16-9cac9f6025c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tn6ng" Oct 31 13:54:15.641662 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:15.641584 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tn6ng" Oct 31 13:54:15.960480 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:15.960441 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tn6ng"] Oct 31 13:54:15.974575 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:54:15.974541 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod803b53b4_4ae4_493c_ae16_9cac9f6025c1.slice/crio-746a8f901f89ce66843d7fc0d158daa95142246ef8f4abd7d8ca00cbf06adfb4 WatchSource:0}: Error finding container 746a8f901f89ce66843d7fc0d158daa95142246ef8f4abd7d8ca00cbf06adfb4: Status 404 returned error can't find the container with id 746a8f901f89ce66843d7fc0d158daa95142246ef8f4abd7d8ca00cbf06adfb4 Oct 31 13:54:16.276462 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:16.276412 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tn6ng" event={"ID":"803b53b4-4ae4-493c-ae16-9cac9f6025c1","Type":"ContainerStarted","Data":"1e4a188ecd9d1c4a76c35134e1e00ae1699701d8bad8eb95ffb67d465ef0226c"} Oct 31 13:54:16.276624 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:16.276482 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tn6ng" event={"ID":"803b53b4-4ae4-493c-ae16-9cac9f6025c1","Type":"ContainerStarted","Data":"746a8f901f89ce66843d7fc0d158daa95142246ef8f4abd7d8ca00cbf06adfb4"} Oct 31 13:54:16.296871 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:16.296811 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tn6ng" podStartSLOduration=1.296793374 podStartE2EDuration="1.296793374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:54:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:54:16.295076307 +0000 UTC m=+4902.778199208" watchObservedRunningTime="2026-10-31 13:54:16.296793374 +0000 UTC m=+4902.779916276" Oct 31 13:54:17.369684 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:17.369037 6841 generic.go:356] "Generic (PLEG): container finished" podID="803b53b4-4ae4-493c-ae16-9cac9f6025c1" containerID="1e4a188ecd9d1c4a76c35134e1e00ae1699701d8bad8eb95ffb67d465ef0226c" exitCode=0 Oct 31 13:54:17.369684 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:17.369260 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tn6ng" event={"ID":"803b53b4-4ae4-493c-ae16-9cac9f6025c1","Type":"ContainerDied","Data":"1e4a188ecd9d1c4a76c35134e1e00ae1699701d8bad8eb95ffb67d465ef0226c"} Oct 31 13:54:18.514890 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:18.514862 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tn6ng" Oct 31 13:54:18.611566 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:18.611534 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/803b53b4-4ae4-493c-ae16-9cac9f6025c1-kube-api-access-dl4m6\" (UniqueName: \"kubernetes.io/projected/803b53b4-4ae4-493c-ae16-9cac9f6025c1-kube-api-access-dl4m6\") pod \"803b53b4-4ae4-493c-ae16-9cac9f6025c1\" (UID: \"803b53b4-4ae4-493c-ae16-9cac9f6025c1\") " Oct 31 13:54:18.611739 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:18.611665 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/803b53b4-4ae4-493c-ae16-9cac9f6025c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/803b53b4-4ae4-493c-ae16-9cac9f6025c1-maas-api-openshift-ca\") pod \"803b53b4-4ae4-493c-ae16-9cac9f6025c1\" (UID: \"803b53b4-4ae4-493c-ae16-9cac9f6025c1\") " Oct 31 13:54:18.611739 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:18.611699 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/803b53b4-4ae4-493c-ae16-9cac9f6025c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/803b53b4-4ae4-493c-ae16-9cac9f6025c1-maas-api-xks-ca\") pod \"803b53b4-4ae4-493c-ae16-9cac9f6025c1\" (UID: \"803b53b4-4ae4-493c-ae16-9cac9f6025c1\") " Oct 31 13:54:18.612136 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:18.612106 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/803b53b4-4ae4-493c-ae16-9cac9f6025c1-maas-api-openshift-ca" pod "803b53b4-4ae4-493c-ae16-9cac9f6025c1" (UID: "803b53b4-4ae4-493c-ae16-9cac9f6025c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:54:18.613829 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:18.613803 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/803b53b4-4ae4-493c-ae16-9cac9f6025c1-kube-api-access-dl4m6" pod "803b53b4-4ae4-493c-ae16-9cac9f6025c1" (UID: "803b53b4-4ae4-493c-ae16-9cac9f6025c1"). InnerVolumeSpecName "kube-api-access-dl4m6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:54:18.613972 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:18.613873 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/803b53b4-4ae4-493c-ae16-9cac9f6025c1-maas-api-xks-ca" pod "803b53b4-4ae4-493c-ae16-9cac9f6025c1" (UID: "803b53b4-4ae4-493c-ae16-9cac9f6025c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:54:18.713431 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:18.713396 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl4m6\" (UniqueName: \"kubernetes.io/projected/803b53b4-4ae4-493c-ae16-9cac9f6025c1-kube-api-access-dl4m6\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:54:18.713431 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:18.713428 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/803b53b4-4ae4-493c-ae16-9cac9f6025c1-maas-api-openshift-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:54:18.713651 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:18.713440 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/803b53b4-4ae4-493c-ae16-9cac9f6025c1-maas-api-xks-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:54:19.382530 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:19.382498 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tn6ng" Oct 31 13:54:19.382707 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:19.382529 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tn6ng" event={"ID":"803b53b4-4ae4-493c-ae16-9cac9f6025c1","Type":"ContainerDied","Data":"746a8f901f89ce66843d7fc0d158daa95142246ef8f4abd7d8ca00cbf06adfb4"} Oct 31 13:54:19.382707 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:19.382608 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="746a8f901f89ce66843d7fc0d158daa95142246ef8f4abd7d8ca00cbf06adfb4" Oct 31 13:54:19.709940 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:19.709899 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:54:19.903892 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:19.903851 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:54:20.035246 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:54:20.035160 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:54:25.019878 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:25.019836 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw" podUID="5bca485f-16b7-43e2-b4b3-f81997be011b" containerName="main" containerID="cri-o://3c354aaf4f2c6eb0e15d262b1280da053be929a9b560b58d9219a98e9ed71799" gracePeriod=45 Oct 31 13:54:25.287609 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:25.287585 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw" Oct 31 13:54:25.378976 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:25.378940 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bca485f-16b7-43e2-b4b3-f81997be011b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bca485f-16b7-43e2-b4b3-f81997be011b-dshm\") pod \"5bca485f-16b7-43e2-b4b3-f81997be011b\" (UID: \"5bca485f-16b7-43e2-b4b3-f81997be011b\") " Oct 31 13:54:25.379159 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:25.379039 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bca485f-16b7-43e2-b4b3-f81997be011b-kube-api-access-chjvt\" (UniqueName: \"kubernetes.io/projected/5bca485f-16b7-43e2-b4b3-f81997be011b-kube-api-access-chjvt\") pod \"5bca485f-16b7-43e2-b4b3-f81997be011b\" (UID: \"5bca485f-16b7-43e2-b4b3-f81997be011b\") " Oct 31 13:54:25.379159 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:25.379108 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bca485f-16b7-43e2-b4b3-f81997be011b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bca485f-16b7-43e2-b4b3-f81997be011b-model-cache\") pod \"5bca485f-16b7-43e2-b4b3-f81997be011b\" (UID: \"5bca485f-16b7-43e2-b4b3-f81997be011b\") " Oct 31 13:54:25.379242 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:25.379162 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bca485f-16b7-43e2-b4b3-f81997be011b-tls-certs\" (UniqueName: \"kubernetes.io/secret/5bca485f-16b7-43e2-b4b3-f81997be011b-tls-certs\") pod \"5bca485f-16b7-43e2-b4b3-f81997be011b\" (UID: \"5bca485f-16b7-43e2-b4b3-f81997be011b\") " Oct 31 13:54:25.379242 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:25.379215 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bca485f-16b7-43e2-b4b3-f81997be011b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bca485f-16b7-43e2-b4b3-f81997be011b-tmp-dir\") pod \"5bca485f-16b7-43e2-b4b3-f81997be011b\" (UID: \"5bca485f-16b7-43e2-b4b3-f81997be011b\") " Oct 31 13:54:25.379343 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:25.379261 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bca485f-16b7-43e2-b4b3-f81997be011b-home\" (UniqueName: \"kubernetes.io/empty-dir/5bca485f-16b7-43e2-b4b3-f81997be011b-home\") pod \"5bca485f-16b7-43e2-b4b3-f81997be011b\" (UID: \"5bca485f-16b7-43e2-b4b3-f81997be011b\") " Oct 31 13:54:25.379403 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:25.379359 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bca485f-16b7-43e2-b4b3-f81997be011b-model-cache" pod "5bca485f-16b7-43e2-b4b3-f81997be011b" (UID: "5bca485f-16b7-43e2-b4b3-f81997be011b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:54:25.379574 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:25.379548 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bca485f-16b7-43e2-b4b3-f81997be011b-tmp-dir" pod "5bca485f-16b7-43e2-b4b3-f81997be011b" (UID: "5bca485f-16b7-43e2-b4b3-f81997be011b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:54:25.379653 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:25.379572 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bca485f-16b7-43e2-b4b3-f81997be011b-home" pod "5bca485f-16b7-43e2-b4b3-f81997be011b" (UID: "5bca485f-16b7-43e2-b4b3-f81997be011b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:54:25.379703 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:25.379674 6841 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bca485f-16b7-43e2-b4b3-f81997be011b-home\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:54:25.379703 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:25.379693 6841 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bca485f-16b7-43e2-b4b3-f81997be011b-model-cache\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:54:25.379781 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:25.379732 6841 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bca485f-16b7-43e2-b4b3-f81997be011b-tmp-dir\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:54:25.381176 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:25.381146 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bca485f-16b7-43e2-b4b3-f81997be011b-dshm" pod "5bca485f-16b7-43e2-b4b3-f81997be011b" (UID: "5bca485f-16b7-43e2-b4b3-f81997be011b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:54:25.381331 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:25.381311 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bca485f-16b7-43e2-b4b3-f81997be011b-tls-certs" pod "5bca485f-16b7-43e2-b4b3-f81997be011b" (UID: "5bca485f-16b7-43e2-b4b3-f81997be011b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:54:25.381386 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:25.381339 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bca485f-16b7-43e2-b4b3-f81997be011b-kube-api-access-chjvt" pod "5bca485f-16b7-43e2-b4b3-f81997be011b" (UID: "5bca485f-16b7-43e2-b4b3-f81997be011b"). InnerVolumeSpecName "kube-api-access-chjvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:54:25.416684 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:25.416655 6841 generic.go:356] "Generic (PLEG): container finished" podID="5bca485f-16b7-43e2-b4b3-f81997be011b" containerID="3c354aaf4f2c6eb0e15d262b1280da053be929a9b560b58d9219a98e9ed71799" exitCode=0 Oct 31 13:54:25.416833 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:25.416722 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw" Oct 31 13:54:25.416833 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:25.416736 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw" event={"ID":"5bca485f-16b7-43e2-b4b3-f81997be011b","Type":"ContainerDied","Data":"3c354aaf4f2c6eb0e15d262b1280da053be929a9b560b58d9219a98e9ed71799"} Oct 31 13:54:25.416833 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:25.416781 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw" event={"ID":"5bca485f-16b7-43e2-b4b3-f81997be011b","Type":"ContainerDied","Data":"fe1098d3abc7fe334be5621034e157d69e695075d3aebf7bb8b3b19653c47548"} Oct 31 13:54:25.416833 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:25.416798 6841 scope.go:122] "RemoveContainer" containerID="3c354aaf4f2c6eb0e15d262b1280da053be929a9b560b58d9219a98e9ed71799" Oct 31 13:54:25.426733 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:25.426714 6841 scope.go:122] "RemoveContainer" containerID="3c354aaf4f2c6eb0e15d262b1280da053be929a9b560b58d9219a98e9ed71799" Oct 31 13:54:25.427060 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:54:25.427035 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c354aaf4f2c6eb0e15d262b1280da053be929a9b560b58d9219a98e9ed71799\": container with ID starting with 3c354aaf4f2c6eb0e15d262b1280da053be929a9b560b58d9219a98e9ed71799 not found: ID does not exist" containerID="3c354aaf4f2c6eb0e15d262b1280da053be929a9b560b58d9219a98e9ed71799" Oct 31 13:54:25.427164 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:25.427067 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c354aaf4f2c6eb0e15d262b1280da053be929a9b560b58d9219a98e9ed71799"} err="failed to get container status \"3c354aaf4f2c6eb0e15d262b1280da053be929a9b560b58d9219a98e9ed71799\": rpc error: code = NotFound desc = could not find container \"3c354aaf4f2c6eb0e15d262b1280da053be929a9b560b58d9219a98e9ed71799\": container with ID starting with 3c354aaf4f2c6eb0e15d262b1280da053be929a9b560b58d9219a98e9ed71799 not found: ID does not exist" Oct 31 13:54:25.439863 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:25.439838 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw"] Oct 31 13:54:25.443322 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:25.443302 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-9ebe6601-kserve-7c6fc698c7-ftbtw"] Oct 31 13:54:25.480453 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:25.480431 6841 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bca485f-16b7-43e2-b4b3-f81997be011b-dshm\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:54:25.480453 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:25.480455 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chjvt\" (UniqueName: \"kubernetes.io/projected/5bca485f-16b7-43e2-b4b3-f81997be011b-kube-api-access-chjvt\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:54:25.480621 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:25.480466 6841 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bca485f-16b7-43e2-b4b3-f81997be011b-tls-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:54:25.641786 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:25.641712 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bca485f-16b7-43e2-b4b3-f81997be011b" path="/var/lib/kubelet/pods/5bca485f-16b7-43e2-b4b3-f81997be011b/volumes" Oct 31 13:54:26.603618 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:26.603579 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg" podUID="99d41763-e41c-417a-9bbe-b59bc95cf39b" containerName="main" containerID="cri-o://b1c70bd947e2501e01686606a7eb777eeab61629176eeb9adf4fab3d78c1cf2a" gracePeriod=45 Oct 31 13:54:27.496404 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:27.496361 6841 generic.go:356] "Generic (PLEG): container finished" podID="99d41763-e41c-417a-9bbe-b59bc95cf39b" containerID="b1c70bd947e2501e01686606a7eb777eeab61629176eeb9adf4fab3d78c1cf2a" exitCode=0 Oct 31 13:54:27.496726 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:27.496639 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg" event={"ID":"99d41763-e41c-417a-9bbe-b59bc95cf39b","Type":"ContainerDied","Data":"b1c70bd947e2501e01686606a7eb777eeab61629176eeb9adf4fab3d78c1cf2a"} Oct 31 13:54:27.496941 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:27.496919 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg" event={"ID":"99d41763-e41c-417a-9bbe-b59bc95cf39b","Type":"ContainerDied","Data":"18cbb96afef113c4c63da47fc36ff9356ffda4e2a1a8b5411e656438e19f41c2"} Oct 31 13:54:27.504175 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:27.504151 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18cbb96afef113c4c63da47fc36ff9356ffda4e2a1a8b5411e656438e19f41c2" Oct 31 13:54:27.508177 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:27.508155 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg" Oct 31 13:54:27.680149 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:27.631507 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99d41763-e41c-417a-9bbe-b59bc95cf39b-home\" (UniqueName: \"kubernetes.io/empty-dir/99d41763-e41c-417a-9bbe-b59bc95cf39b-home\") pod \"99d41763-e41c-417a-9bbe-b59bc95cf39b\" (UID: \"99d41763-e41c-417a-9bbe-b59bc95cf39b\") " Oct 31 13:54:27.680749 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:27.680722 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99d41763-e41c-417a-9bbe-b59bc95cf39b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99d41763-e41c-417a-9bbe-b59bc95cf39b-model-cache\") pod \"99d41763-e41c-417a-9bbe-b59bc95cf39b\" (UID: \"99d41763-e41c-417a-9bbe-b59bc95cf39b\") " Oct 31 13:54:27.681221 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:27.681200 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99d41763-e41c-417a-9bbe-b59bc95cf39b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/99d41763-e41c-417a-9bbe-b59bc95cf39b-dshm\") pod \"99d41763-e41c-417a-9bbe-b59bc95cf39b\" (UID: \"99d41763-e41c-417a-9bbe-b59bc95cf39b\") " Oct 31 13:54:27.681734 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:27.681713 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99d41763-e41c-417a-9bbe-b59bc95cf39b-kube-api-access-zp22r\" (UniqueName: \"kubernetes.io/projected/99d41763-e41c-417a-9bbe-b59bc95cf39b-kube-api-access-zp22r\") pod \"99d41763-e41c-417a-9bbe-b59bc95cf39b\" (UID: \"99d41763-e41c-417a-9bbe-b59bc95cf39b\") " Oct 31 13:54:27.682248 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:27.682229 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99d41763-e41c-417a-9bbe-b59bc95cf39b-tls-certs\" (UniqueName: \"kubernetes.io/secret/99d41763-e41c-417a-9bbe-b59bc95cf39b-tls-certs\") pod \"99d41763-e41c-417a-9bbe-b59bc95cf39b\" (UID: \"99d41763-e41c-417a-9bbe-b59bc95cf39b\") " Oct 31 13:54:27.682430 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:27.682413 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99d41763-e41c-417a-9bbe-b59bc95cf39b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99d41763-e41c-417a-9bbe-b59bc95cf39b-tmp-dir\") pod \"99d41763-e41c-417a-9bbe-b59bc95cf39b\" (UID: \"99d41763-e41c-417a-9bbe-b59bc95cf39b\") " Oct 31 13:54:27.683323 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:27.638078 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d41763-e41c-417a-9bbe-b59bc95cf39b-home" pod "99d41763-e41c-417a-9bbe-b59bc95cf39b" (UID: "99d41763-e41c-417a-9bbe-b59bc95cf39b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:54:27.683458 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:27.681140 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d41763-e41c-417a-9bbe-b59bc95cf39b-model-cache" pod "99d41763-e41c-417a-9bbe-b59bc95cf39b" (UID: "99d41763-e41c-417a-9bbe-b59bc95cf39b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:54:27.684279 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:27.684257 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d41763-e41c-417a-9bbe-b59bc95cf39b-tmp-dir" pod "99d41763-e41c-417a-9bbe-b59bc95cf39b" (UID: "99d41763-e41c-417a-9bbe-b59bc95cf39b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:54:27.727005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:27.723165 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d41763-e41c-417a-9bbe-b59bc95cf39b-dshm" pod "99d41763-e41c-417a-9bbe-b59bc95cf39b" (UID: "99d41763-e41c-417a-9bbe-b59bc95cf39b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:54:27.736223 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:27.736165 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d41763-e41c-417a-9bbe-b59bc95cf39b-kube-api-access-zp22r" pod "99d41763-e41c-417a-9bbe-b59bc95cf39b" (UID: "99d41763-e41c-417a-9bbe-b59bc95cf39b"). InnerVolumeSpecName "kube-api-access-zp22r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:54:27.736486 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:27.736453 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99d41763-e41c-417a-9bbe-b59bc95cf39b-tls-certs" pod "99d41763-e41c-417a-9bbe-b59bc95cf39b" (UID: "99d41763-e41c-417a-9bbe-b59bc95cf39b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:54:27.784807 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:27.784616 6841 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99d41763-e41c-417a-9bbe-b59bc95cf39b-home\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:54:27.784807 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:27.784664 6841 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99d41763-e41c-417a-9bbe-b59bc95cf39b-model-cache\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:54:27.784807 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:27.784679 6841 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99d41763-e41c-417a-9bbe-b59bc95cf39b-dshm\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:54:27.784807 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:27.784694 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp22r\" (UniqueName: \"kubernetes.io/projected/99d41763-e41c-417a-9bbe-b59bc95cf39b-kube-api-access-zp22r\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:54:27.784807 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:27.784709 6841 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99d41763-e41c-417a-9bbe-b59bc95cf39b-tls-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:54:27.784807 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:27.784723 6841 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99d41763-e41c-417a-9bbe-b59bc95cf39b-tmp-dir\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:54:28.229121 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:28.229072 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg" podUID="99d41763-e41c-417a-9bbe-b59bc95cf39b" containerName="main" probeResult="failure" output="Get \"https://10.134.1.8:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:54:28.516290 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:28.516190 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg" Oct 31 13:54:28.551926 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:28.551883 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg"] Oct 31 13:54:28.555852 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:28.555823 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-9ebe6601-kserve-796b9dd7c7-lzrpg"] Oct 31 13:54:29.034209 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:29.034171 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5kjcn"] Oct 31 13:54:29.037536 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:29.037497 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5kjcn"] Oct 31 13:54:29.644612 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:29.644567 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6604b972-8235-4362-8ac7-5698f25099a1" path="/var/lib/kubelet/pods/6604b972-8235-4362-8ac7-5698f25099a1/volumes" Oct 31 13:54:29.645272 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:29.645246 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99d41763-e41c-417a-9bbe-b59bc95cf39b" path="/var/lib/kubelet/pods/99d41763-e41c-417a-9bbe-b59bc95cf39b/volumes" Oct 31 13:54:32.836506 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:32.836465 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-953a6bca-kmqwr"] Oct 31 13:54:32.836940 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:32.836925 6841 cpu_manager.go:414] "containerMap: removing container" podUID="5bca485f-16b7-43e2-b4b3-f81997be011b" containerName="main" Oct 31 13:54:32.837037 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:32.836941 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bca485f-16b7-43e2-b4b3-f81997be011b" containerName="main" Oct 31 13:54:32.837037 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:32.836958 6841 cpu_manager.go:414] "containerMap: removing container" podUID="99d41763-e41c-417a-9bbe-b59bc95cf39b" containerName="main" Oct 31 13:54:32.837037 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:32.836964 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99d41763-e41c-417a-9bbe-b59bc95cf39b" containerName="main" Oct 31 13:54:32.837037 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:32.837000 6841 cpu_manager.go:414] "containerMap: removing container" podUID="803b53b4-4ae4-493c-ae16-9cac9f6025c1" containerName="revoke-keys" Oct 31 13:54:32.837037 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:32.837007 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="803b53b4-4ae4-493c-ae16-9cac9f6025c1" containerName="revoke-keys" Oct 31 13:54:32.837229 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:32.837086 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bca485f-16b7-43e2-b4b3-f81997be011b" containerName="main" Oct 31 13:54:32.837229 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:32.837097 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="99d41763-e41c-417a-9bbe-b59bc95cf39b" containerName="main" Oct 31 13:54:32.837229 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:32.837110 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="803b53b4-4ae4-493c-ae16-9cac9f6025c1" containerName="revoke-keys" Oct 31 13:54:32.841619 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:32.841603 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-953a6bca-kmqwr" Oct 31 13:54:32.847767 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:32.847739 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:54:32.848377 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:32.848349 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 13:54:32.848869 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:32.848438 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:54:32.848869 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:32.848561 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:54:32.851049 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:32.851026 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-953a6bca-kmqwr"] Oct 31 13:54:32.921088 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:32.921060 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14c1666f-e6d5-4f5e-bf05-43253e9239eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-953a6bca-kmqwr\" (UID: \"14c1666f-e6d5-4f5e-bf05-43253e9239eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-953a6bca-kmqwr" Oct 31 13:54:32.921244 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:32.921102 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14c1666f-e6d5-4f5e-bf05-43253e9239eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-953a6bca-kmqwr\" (UID: \"14c1666f-e6d5-4f5e-bf05-43253e9239eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-953a6bca-kmqwr" Oct 31 13:54:32.921244 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:32.921120 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57jp5\" (UniqueName: \"kubernetes.io/projected/14c1666f-e6d5-4f5e-bf05-43253e9239eb-kube-api-access-57jp5\") pod \"maas-api-revoke-sub-953a6bca-kmqwr\" (UID: \"14c1666f-e6d5-4f5e-bf05-43253e9239eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-953a6bca-kmqwr" Oct 31 13:54:33.022349 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:33.022316 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14c1666f-e6d5-4f5e-bf05-43253e9239eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-953a6bca-kmqwr\" (UID: \"14c1666f-e6d5-4f5e-bf05-43253e9239eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-953a6bca-kmqwr" Oct 31 13:54:33.022521 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:33.022356 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14c1666f-e6d5-4f5e-bf05-43253e9239eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-953a6bca-kmqwr\" (UID: \"14c1666f-e6d5-4f5e-bf05-43253e9239eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-953a6bca-kmqwr" Oct 31 13:54:33.022521 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:33.022375 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57jp5\" (UniqueName: \"kubernetes.io/projected/14c1666f-e6d5-4f5e-bf05-43253e9239eb-kube-api-access-57jp5\") pod \"maas-api-revoke-sub-953a6bca-kmqwr\" (UID: \"14c1666f-e6d5-4f5e-bf05-43253e9239eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-953a6bca-kmqwr" Oct 31 13:54:33.023026 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:33.022970 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14c1666f-e6d5-4f5e-bf05-43253e9239eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-953a6bca-kmqwr\" (UID: \"14c1666f-e6d5-4f5e-bf05-43253e9239eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-953a6bca-kmqwr" Oct 31 13:54:33.024733 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:33.024709 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14c1666f-e6d5-4f5e-bf05-43253e9239eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-953a6bca-kmqwr\" (UID: \"14c1666f-e6d5-4f5e-bf05-43253e9239eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-953a6bca-kmqwr" Oct 31 13:54:33.034547 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:33.034526 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57jp5\" (UniqueName: \"kubernetes.io/projected/14c1666f-e6d5-4f5e-bf05-43253e9239eb-kube-api-access-57jp5\") pod \"maas-api-revoke-sub-953a6bca-kmqwr\" (UID: \"14c1666f-e6d5-4f5e-bf05-43253e9239eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-953a6bca-kmqwr" Oct 31 13:54:33.157693 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:33.157609 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-953a6bca-kmqwr" Oct 31 13:54:33.432401 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:33.432369 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-953a6bca-kmqwr"] Oct 31 13:54:33.434042 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:54:33.434014 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14c1666f_e6d5_4f5e_bf05_43253e9239eb.slice/crio-ec959c1adebaadb08134716a65ef9943751b8c50ae27ae8d04eb543dea728f09 WatchSource:0}: Error finding container ec959c1adebaadb08134716a65ef9943751b8c50ae27ae8d04eb543dea728f09: Status 404 returned error can't find the container with id ec959c1adebaadb08134716a65ef9943751b8c50ae27ae8d04eb543dea728f09 Oct 31 13:54:33.572418 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:33.572383 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-953a6bca-kmqwr" event={"ID":"14c1666f-e6d5-4f5e-bf05-43253e9239eb","Type":"ContainerStarted","Data":"ec959c1adebaadb08134716a65ef9943751b8c50ae27ae8d04eb543dea728f09"} Oct 31 13:54:34.579238 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:34.579144 6841 generic.go:356] "Generic (PLEG): container finished" podID="14c1666f-e6d5-4f5e-bf05-43253e9239eb" containerID="d14d35a1c9c4023a5cfd3f4ec5e2839c71b448f4efc723a2e887f756662baae9" exitCode=0 Oct 31 13:54:34.579238 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:34.579212 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-953a6bca-kmqwr" event={"ID":"14c1666f-e6d5-4f5e-bf05-43253e9239eb","Type":"ContainerDied","Data":"d14d35a1c9c4023a5cfd3f4ec5e2839c71b448f4efc723a2e887f756662baae9"} Oct 31 13:54:35.718934 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:35.718909 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-953a6bca-kmqwr" Oct 31 13:54:35.850713 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:35.850629 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14c1666f-e6d5-4f5e-bf05-43253e9239eb-kube-api-access-57jp5\" (UniqueName: \"kubernetes.io/projected/14c1666f-e6d5-4f5e-bf05-43253e9239eb-kube-api-access-57jp5\") pod \"14c1666f-e6d5-4f5e-bf05-43253e9239eb\" (UID: \"14c1666f-e6d5-4f5e-bf05-43253e9239eb\") " Oct 31 13:54:35.850713 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:35.850706 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14c1666f-e6d5-4f5e-bf05-43253e9239eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14c1666f-e6d5-4f5e-bf05-43253e9239eb-maas-api-openshift-ca\") pod \"14c1666f-e6d5-4f5e-bf05-43253e9239eb\" (UID: \"14c1666f-e6d5-4f5e-bf05-43253e9239eb\") " Oct 31 13:54:35.850952 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:35.850731 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14c1666f-e6d5-4f5e-bf05-43253e9239eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14c1666f-e6d5-4f5e-bf05-43253e9239eb-maas-api-xks-ca\") pod \"14c1666f-e6d5-4f5e-bf05-43253e9239eb\" (UID: \"14c1666f-e6d5-4f5e-bf05-43253e9239eb\") " Oct 31 13:54:35.851103 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:35.851078 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14c1666f-e6d5-4f5e-bf05-43253e9239eb-maas-api-openshift-ca" pod "14c1666f-e6d5-4f5e-bf05-43253e9239eb" (UID: "14c1666f-e6d5-4f5e-bf05-43253e9239eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:54:35.852837 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:35.852812 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14c1666f-e6d5-4f5e-bf05-43253e9239eb-maas-api-xks-ca" pod "14c1666f-e6d5-4f5e-bf05-43253e9239eb" (UID: "14c1666f-e6d5-4f5e-bf05-43253e9239eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:54:35.852928 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:35.852888 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14c1666f-e6d5-4f5e-bf05-43253e9239eb-kube-api-access-57jp5" pod "14c1666f-e6d5-4f5e-bf05-43253e9239eb" (UID: "14c1666f-e6d5-4f5e-bf05-43253e9239eb"). InnerVolumeSpecName "kube-api-access-57jp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:54:35.951873 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:35.951841 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14c1666f-e6d5-4f5e-bf05-43253e9239eb-maas-api-xks-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:54:35.951873 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:35.951873 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57jp5\" (UniqueName: \"kubernetes.io/projected/14c1666f-e6d5-4f5e-bf05-43253e9239eb-kube-api-access-57jp5\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:54:35.952078 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:35.951891 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14c1666f-e6d5-4f5e-bf05-43253e9239eb-maas-api-openshift-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:54:36.592140 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:36.592112 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-953a6bca-kmqwr" Oct 31 13:54:36.592296 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:36.592151 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-953a6bca-kmqwr" event={"ID":"14c1666f-e6d5-4f5e-bf05-43253e9239eb","Type":"ContainerDied","Data":"ec959c1adebaadb08134716a65ef9943751b8c50ae27ae8d04eb543dea728f09"} Oct 31 13:54:36.592296 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:36.592235 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec959c1adebaadb08134716a65ef9943751b8c50ae27ae8d04eb543dea728f09" Oct 31 13:54:41.558909 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:41.558875 6841 scope.go:122] "RemoveContainer" containerID="a7abd4228da7fc8f8882c3fe8ef9d1b78a096dc62b002a0e996630d647045480" Oct 31 13:54:41.673846 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:41.673802 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:54:45.021936 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:45.021894 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w9dcl"] Oct 31 13:54:45.024042 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:45.024014 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w9dcl"] Oct 31 13:54:45.640874 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:45.640836 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9a4c1cd-e46c-41b3-b201-a62f51880a3f" path="/var/lib/kubelet/pods/c9a4c1cd-e46c-41b3-b201-a62f51880a3f/volumes" Oct 31 13:54:46.186080 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:46.186039 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-76c647798b-m46mv"] Oct 31 13:54:46.186579 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:46.186562 6841 cpu_manager.go:414] "containerMap: removing container" podUID="14c1666f-e6d5-4f5e-bf05-43253e9239eb" containerName="revoke-keys" Oct 31 13:54:46.186579 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:46.186579 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14c1666f-e6d5-4f5e-bf05-43253e9239eb" containerName="revoke-keys" Oct 31 13:54:46.186717 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:46.186654 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="14c1666f-e6d5-4f5e-bf05-43253e9239eb" containerName="revoke-keys" Oct 31 13:54:46.191301 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:46.191285 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-76c647798b-m46mv" Oct 31 13:54:46.199482 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:46.199209 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-76c647798b-m46mv"] Oct 31 13:54:46.199879 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:46.199858 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 31 13:54:46.200020 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:46.199944 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-z7fks\"" Oct 31 13:54:46.250274 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:46.250244 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7jmq\" (UniqueName: \"kubernetes.io/projected/32124ac7-9f3c-449e-84a0-691c8b108ead-kube-api-access-r7jmq\") pod \"payload-pre-processing-76c647798b-m46mv\" (UID: \"32124ac7-9f3c-449e-84a0-691c8b108ead\") " pod="openshift-ingress/payload-pre-processing-76c647798b-m46mv" Oct 31 13:54:46.250433 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:46.250301 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32124ac7-9f3c-449e-84a0-691c8b108ead-plugins-config-volume\") pod \"payload-pre-processing-76c647798b-m46mv\" (UID: \"32124ac7-9f3c-449e-84a0-691c8b108ead\") " pod="openshift-ingress/payload-pre-processing-76c647798b-m46mv" Oct 31 13:54:46.351591 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:46.351551 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7jmq\" (UniqueName: \"kubernetes.io/projected/32124ac7-9f3c-449e-84a0-691c8b108ead-kube-api-access-r7jmq\") pod \"payload-pre-processing-76c647798b-m46mv\" (UID: \"32124ac7-9f3c-449e-84a0-691c8b108ead\") " pod="openshift-ingress/payload-pre-processing-76c647798b-m46mv" Oct 31 13:54:46.351778 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:46.351646 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32124ac7-9f3c-449e-84a0-691c8b108ead-plugins-config-volume\") pod \"payload-pre-processing-76c647798b-m46mv\" (UID: \"32124ac7-9f3c-449e-84a0-691c8b108ead\") " pod="openshift-ingress/payload-pre-processing-76c647798b-m46mv" Oct 31 13:54:46.352361 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:46.352338 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32124ac7-9f3c-449e-84a0-691c8b108ead-plugins-config-volume\") pod \"payload-pre-processing-76c647798b-m46mv\" (UID: \"32124ac7-9f3c-449e-84a0-691c8b108ead\") " pod="openshift-ingress/payload-pre-processing-76c647798b-m46mv" Oct 31 13:54:46.365616 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:46.365590 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7jmq\" (UniqueName: \"kubernetes.io/projected/32124ac7-9f3c-449e-84a0-691c8b108ead-kube-api-access-r7jmq\") pod \"payload-pre-processing-76c647798b-m46mv\" (UID: \"32124ac7-9f3c-449e-84a0-691c8b108ead\") " pod="openshift-ingress/payload-pre-processing-76c647798b-m46mv" Oct 31 13:54:46.507849 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:46.507815 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-76c647798b-m46mv" Oct 31 13:54:46.781723 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:46.781694 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-76c647798b-m46mv"] Oct 31 13:54:46.783609 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:54:46.783571 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32124ac7_9f3c_449e_84a0_691c8b108ead.slice/crio-dfc61245a88d4a43ba573e6cae9b7f9862f376ba24412877e9d08c4e7724fe35 WatchSource:0}: Error finding container dfc61245a88d4a43ba573e6cae9b7f9862f376ba24412877e9d08c4e7724fe35: Status 404 returned error can't find the container with id dfc61245a88d4a43ba573e6cae9b7f9862f376ba24412877e9d08c4e7724fe35 Oct 31 13:54:47.651296 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:47.651262 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-76c647798b-m46mv" event={"ID":"32124ac7-9f3c-449e-84a0-691c8b108ead","Type":"ContainerStarted","Data":"eecc3155a5a4d6cfa6ce789523a931da639748a8af7fd5882059867092ef7531"} Oct 31 13:54:47.651701 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:47.651318 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-76c647798b-m46mv" event={"ID":"32124ac7-9f3c-449e-84a0-691c8b108ead","Type":"ContainerStarted","Data":"dfc61245a88d4a43ba573e6cae9b7f9862f376ba24412877e9d08c4e7724fe35"} Oct 31 13:54:47.651701 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:47.651349 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-76c647798b-m46mv" Oct 31 13:54:47.672076 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:47.672029 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-76c647798b-m46mv" podStartSLOduration=1.672015623 podStartE2EDuration="1.672015623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:54:47.670615888 +0000 UTC m=+4934.153738790" watchObservedRunningTime="2026-10-31 13:54:47.672015623 +0000 UTC m=+4934.155138524" Oct 31 13:54:50.033889 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:54:50.033857 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:54:55.023037 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:55.022993 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8nm9f"] Oct 31 13:54:55.025888 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:55.025857 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8nm9f"] Oct 31 13:54:55.641463 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:55.641426 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fc9c711-807a-4378-98d8-8e39feda77d5" path="/var/lib/kubelet/pods/6fc9c711-807a-4378-98d8-8e39feda77d5/volumes" Oct 31 13:54:58.660417 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:54:58.660382 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-76c647798b-m46mv" Oct 31 13:55:00.469853 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:00.469813 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-74c58f9c5b-xvm2h"] Oct 31 13:55:00.474564 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:00.474535 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-74c58f9c5b-xvm2h" Oct 31 13:55:00.486674 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:00.486642 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-74c58f9c5b-xvm2h"] Oct 31 13:55:00.588433 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:00.588398 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c619b77-1620-4d5a-ab6b-44413826f8a0-plugins-config-volume\") pod \"payload-processing-74c58f9c5b-xvm2h\" (UID: \"7c619b77-1620-4d5a-ab6b-44413826f8a0\") " pod="openshift-ingress/payload-processing-74c58f9c5b-xvm2h" Oct 31 13:55:00.588614 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:00.588576 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl7ks\" (UniqueName: \"kubernetes.io/projected/7c619b77-1620-4d5a-ab6b-44413826f8a0-kube-api-access-cl7ks\") pod \"payload-processing-74c58f9c5b-xvm2h\" (UID: \"7c619b77-1620-4d5a-ab6b-44413826f8a0\") " pod="openshift-ingress/payload-processing-74c58f9c5b-xvm2h" Oct 31 13:55:00.690074 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:00.690038 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl7ks\" (UniqueName: \"kubernetes.io/projected/7c619b77-1620-4d5a-ab6b-44413826f8a0-kube-api-access-cl7ks\") pod \"payload-processing-74c58f9c5b-xvm2h\" (UID: \"7c619b77-1620-4d5a-ab6b-44413826f8a0\") " pod="openshift-ingress/payload-processing-74c58f9c5b-xvm2h" Oct 31 13:55:00.690238 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:00.690090 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c619b77-1620-4d5a-ab6b-44413826f8a0-plugins-config-volume\") pod \"payload-processing-74c58f9c5b-xvm2h\" (UID: \"7c619b77-1620-4d5a-ab6b-44413826f8a0\") " pod="openshift-ingress/payload-processing-74c58f9c5b-xvm2h" Oct 31 13:55:00.690791 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:00.690769 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c619b77-1620-4d5a-ab6b-44413826f8a0-plugins-config-volume\") pod \"payload-processing-74c58f9c5b-xvm2h\" (UID: \"7c619b77-1620-4d5a-ab6b-44413826f8a0\") " pod="openshift-ingress/payload-processing-74c58f9c5b-xvm2h" Oct 31 13:55:00.702822 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:00.702789 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl7ks\" (UniqueName: \"kubernetes.io/projected/7c619b77-1620-4d5a-ab6b-44413826f8a0-kube-api-access-cl7ks\") pod \"payload-processing-74c58f9c5b-xvm2h\" (UID: \"7c619b77-1620-4d5a-ab6b-44413826f8a0\") " pod="openshift-ingress/payload-processing-74c58f9c5b-xvm2h" Oct 31 13:55:00.790359 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:00.790270 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-74c58f9c5b-xvm2h" Oct 31 13:55:01.070793 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:01.070767 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-74c58f9c5b-xvm2h"] Oct 31 13:55:01.072932 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:55:01.072901 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c619b77_1620_4d5a_ab6b_44413826f8a0.slice/crio-b4ab872d9f0a072e62022d1b12882391be07e177ff4afba3a7811b7eecb2347a WatchSource:0}: Error finding container b4ab872d9f0a072e62022d1b12882391be07e177ff4afba3a7811b7eecb2347a: Status 404 returned error can't find the container with id b4ab872d9f0a072e62022d1b12882391be07e177ff4afba3a7811b7eecb2347a Oct 31 13:55:01.736708 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:01.736671 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-74c58f9c5b-xvm2h" event={"ID":"7c619b77-1620-4d5a-ab6b-44413826f8a0","Type":"ContainerStarted","Data":"0d3b71489b64504f92528423fdc87b5253ced6530f2e36eb7509d26bc0e9a244"} Oct 31 13:55:01.737137 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:01.736732 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-74c58f9c5b-xvm2h" event={"ID":"7c619b77-1620-4d5a-ab6b-44413826f8a0","Type":"ContainerStarted","Data":"b4ab872d9f0a072e62022d1b12882391be07e177ff4afba3a7811b7eecb2347a"} Oct 31 13:55:01.737137 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:01.736892 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-74c58f9c5b-xvm2h" Oct 31 13:55:01.755517 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:01.755443 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-74c58f9c5b-xvm2h" podStartSLOduration=1.7554278760000002 podStartE2EDuration="1.755427876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:55:01.755243703 +0000 UTC m=+4948.238366605" watchObservedRunningTime="2026-10-31 13:55:01.755427876 +0000 UTC m=+4948.238550778" Oct 31 13:55:12.746498 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:12.746465 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-74c58f9c5b-xvm2h" Oct 31 13:55:14.733773 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:14.733740 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-686b67694c-t4nfb"] Oct 31 13:55:14.737790 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:14.737772 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-686b67694c-t4nfb" Oct 31 13:55:14.747291 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:14.747266 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-686b67694c-t4nfb"] Oct 31 13:55:14.837736 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:14.837697 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbkx5\" (UniqueName: \"kubernetes.io/projected/f6b4b035-480f-4d5b-8336-8aac9a0c2be3-kube-api-access-lbkx5\") pod \"payload-processing-686b67694c-t4nfb\" (UID: \"f6b4b035-480f-4d5b-8336-8aac9a0c2be3\") " pod="openshift-ingress/payload-processing-686b67694c-t4nfb" Oct 31 13:55:14.837929 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:14.837857 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6b4b035-480f-4d5b-8336-8aac9a0c2be3-plugins-config-volume\") pod \"payload-processing-686b67694c-t4nfb\" (UID: \"f6b4b035-480f-4d5b-8336-8aac9a0c2be3\") " pod="openshift-ingress/payload-processing-686b67694c-t4nfb" Oct 31 13:55:14.910587 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:14.910551 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-79b6547d4d-548dj"] Oct 31 13:55:14.914354 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:14.914329 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-79b6547d4d-548dj" Oct 31 13:55:14.924228 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:14.924202 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-79b6547d4d-548dj"] Oct 31 13:55:14.939299 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:14.939269 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6b4b035-480f-4d5b-8336-8aac9a0c2be3-plugins-config-volume\") pod \"payload-processing-686b67694c-t4nfb\" (UID: \"f6b4b035-480f-4d5b-8336-8aac9a0c2be3\") " pod="openshift-ingress/payload-processing-686b67694c-t4nfb" Oct 31 13:55:14.939466 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:14.939326 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbkx5\" (UniqueName: \"kubernetes.io/projected/f6b4b035-480f-4d5b-8336-8aac9a0c2be3-kube-api-access-lbkx5\") pod \"payload-processing-686b67694c-t4nfb\" (UID: \"f6b4b035-480f-4d5b-8336-8aac9a0c2be3\") " pod="openshift-ingress/payload-processing-686b67694c-t4nfb" Oct 31 13:55:14.940019 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:14.939970 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6b4b035-480f-4d5b-8336-8aac9a0c2be3-plugins-config-volume\") pod \"payload-processing-686b67694c-t4nfb\" (UID: \"f6b4b035-480f-4d5b-8336-8aac9a0c2be3\") " pod="openshift-ingress/payload-processing-686b67694c-t4nfb" Oct 31 13:55:14.952373 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:14.952342 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbkx5\" (UniqueName: \"kubernetes.io/projected/f6b4b035-480f-4d5b-8336-8aac9a0c2be3-kube-api-access-lbkx5\") pod \"payload-processing-686b67694c-t4nfb\" (UID: \"f6b4b035-480f-4d5b-8336-8aac9a0c2be3\") " pod="openshift-ingress/payload-processing-686b67694c-t4nfb" Oct 31 13:55:15.040196 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:15.040097 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g82m4\" (UniqueName: \"kubernetes.io/projected/2185522e-f5f4-4936-ab62-90b428398664-kube-api-access-g82m4\") pod \"payload-pre-processing-79b6547d4d-548dj\" (UID: \"2185522e-f5f4-4936-ab62-90b428398664\") " pod="openshift-ingress/payload-pre-processing-79b6547d4d-548dj" Oct 31 13:55:15.040196 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:15.040169 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2185522e-f5f4-4936-ab62-90b428398664-plugins-config-volume\") pod \"payload-pre-processing-79b6547d4d-548dj\" (UID: \"2185522e-f5f4-4936-ab62-90b428398664\") " pod="openshift-ingress/payload-pre-processing-79b6547d4d-548dj" Oct 31 13:55:15.051942 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:15.051916 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-686b67694c-t4nfb" Oct 31 13:55:15.141469 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:15.141436 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2185522e-f5f4-4936-ab62-90b428398664-plugins-config-volume\") pod \"payload-pre-processing-79b6547d4d-548dj\" (UID: \"2185522e-f5f4-4936-ab62-90b428398664\") " pod="openshift-ingress/payload-pre-processing-79b6547d4d-548dj" Oct 31 13:55:15.141650 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:15.141552 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g82m4\" (UniqueName: \"kubernetes.io/projected/2185522e-f5f4-4936-ab62-90b428398664-kube-api-access-g82m4\") pod \"payload-pre-processing-79b6547d4d-548dj\" (UID: \"2185522e-f5f4-4936-ab62-90b428398664\") " pod="openshift-ingress/payload-pre-processing-79b6547d4d-548dj" Oct 31 13:55:15.142267 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:15.142246 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2185522e-f5f4-4936-ab62-90b428398664-plugins-config-volume\") pod \"payload-pre-processing-79b6547d4d-548dj\" (UID: \"2185522e-f5f4-4936-ab62-90b428398664\") " pod="openshift-ingress/payload-pre-processing-79b6547d4d-548dj" Oct 31 13:55:15.155089 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:15.155058 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g82m4\" (UniqueName: \"kubernetes.io/projected/2185522e-f5f4-4936-ab62-90b428398664-kube-api-access-g82m4\") pod \"payload-pre-processing-79b6547d4d-548dj\" (UID: \"2185522e-f5f4-4936-ab62-90b428398664\") " pod="openshift-ingress/payload-pre-processing-79b6547d4d-548dj" Oct 31 13:55:15.229870 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:15.229831 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-79b6547d4d-548dj" Oct 31 13:55:15.328109 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:15.328086 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-686b67694c-t4nfb"] Oct 31 13:55:15.331112 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:55:15.331084 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6b4b035_480f_4d5b_8336_8aac9a0c2be3.slice/crio-e08d33cd446dec3d063848042d4589250fabc16f07bba0095654b8d99e4f6516 WatchSource:0}: Error finding container e08d33cd446dec3d063848042d4589250fabc16f07bba0095654b8d99e4f6516: Status 404 returned error can't find the container with id e08d33cd446dec3d063848042d4589250fabc16f07bba0095654b8d99e4f6516 Oct 31 13:55:15.504627 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:15.504600 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-79b6547d4d-548dj"] Oct 31 13:55:15.506230 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:55:15.506201 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2185522e_f5f4_4936_ab62_90b428398664.slice/crio-10b081ee13a2ade90373b3477ae22b89ed489d2003235111ef65f4510097eb10 WatchSource:0}: Error finding container 10b081ee13a2ade90373b3477ae22b89ed489d2003235111ef65f4510097eb10: Status 404 returned error can't find the container with id 10b081ee13a2ade90373b3477ae22b89ed489d2003235111ef65f4510097eb10 Oct 31 13:55:15.819748 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:15.819659 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-686b67694c-t4nfb" event={"ID":"f6b4b035-480f-4d5b-8336-8aac9a0c2be3","Type":"ContainerStarted","Data":"4eb76d4da70046b565b1b9564b1745446687e4b81e92daf0398c0995095ab530"} Oct 31 13:55:15.819748 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:15.819722 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-686b67694c-t4nfb" event={"ID":"f6b4b035-480f-4d5b-8336-8aac9a0c2be3","Type":"ContainerStarted","Data":"e08d33cd446dec3d063848042d4589250fabc16f07bba0095654b8d99e4f6516"} Oct 31 13:55:15.820314 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:15.819858 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-686b67694c-t4nfb" Oct 31 13:55:15.821261 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:15.821237 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-79b6547d4d-548dj" event={"ID":"2185522e-f5f4-4936-ab62-90b428398664","Type":"ContainerStarted","Data":"804f1179e8c71d444ea4140b244b3df1a56c06bf3277ce9e692cbadf0553d32a"} Oct 31 13:55:15.821376 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:15.821277 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-79b6547d4d-548dj" event={"ID":"2185522e-f5f4-4936-ab62-90b428398664","Type":"ContainerStarted","Data":"10b081ee13a2ade90373b3477ae22b89ed489d2003235111ef65f4510097eb10"} Oct 31 13:55:15.821376 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:15.821298 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-79b6547d4d-548dj" Oct 31 13:55:15.838518 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:15.838479 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-686b67694c-t4nfb" podStartSLOduration=1.838467069 podStartE2EDuration="1.838467069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:55:15.837514844 +0000 UTC m=+4962.320637749" watchObservedRunningTime="2026-10-31 13:55:15.838467069 +0000 UTC m=+4962.321589974" Oct 31 13:55:15.855653 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:15.855598 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-79b6547d4d-548dj" podStartSLOduration=1.855582977 podStartE2EDuration="1.855582977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:55:15.855527831 +0000 UTC m=+4962.338650734" watchObservedRunningTime="2026-10-31 13:55:15.855582977 +0000 UTC m=+4962.338705881" Oct 31 13:55:20.034157 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:55:20.034120 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:55:26.831059 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:26.831029 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-79b6547d4d-548dj" Oct 31 13:55:26.831480 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:26.831100 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-686b67694c-t4nfb" Oct 31 13:55:26.877875 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:26.877838 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-74c58f9c5b-xvm2h"] Oct 31 13:55:26.909385 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:26.909354 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-76c647798b-m46mv"] Oct 31 13:55:29.614144 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:29.614104 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-sxnmn"] Oct 31 13:55:29.618275 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:29.618257 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-sxnmn" Oct 31 13:55:29.647157 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:29.647130 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-sxnmn"] Oct 31 13:55:29.648498 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:29.648469 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\"" Oct 31 13:55:29.648920 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:29.648895 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 31 13:55:29.649892 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:29.649845 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 31 13:55:29.649892 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:29.649866 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 31 13:55:29.650105 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:29.649903 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 31 13:55:29.650105 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:29.649968 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-4q8mf\"" Oct 31 13:55:29.675812 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:29.675788 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc568\" (UniqueName: \"kubernetes.io/projected/696e6f49-45c1-4504-875e-86b899b405c4-kube-api-access-pc568\") pod \"payload-processing-external-model-5bcd7997-sxnmn\" (UID: \"696e6f49-45c1-4504-875e-86b899b405c4\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-sxnmn" Oct 31 13:55:29.675940 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:29.675837 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/696e6f49-45c1-4504-875e-86b899b405c4-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-sxnmn\" (UID: \"696e6f49-45c1-4504-875e-86b899b405c4\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-sxnmn" Oct 31 13:55:29.676040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:29.676027 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/696e6f49-45c1-4504-875e-86b899b405c4-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-sxnmn\" (UID: \"696e6f49-45c1-4504-875e-86b899b405c4\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-sxnmn" Oct 31 13:55:29.676086 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:29.676055 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/696e6f49-45c1-4504-875e-86b899b405c4-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-sxnmn\" (UID: \"696e6f49-45c1-4504-875e-86b899b405c4\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-sxnmn" Oct 31 13:55:29.777174 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:29.777135 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/696e6f49-45c1-4504-875e-86b899b405c4-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-sxnmn\" (UID: \"696e6f49-45c1-4504-875e-86b899b405c4\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-sxnmn" Oct 31 13:55:29.777495 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:29.777318 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/696e6f49-45c1-4504-875e-86b899b405c4-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-sxnmn\" (UID: \"696e6f49-45c1-4504-875e-86b899b405c4\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-sxnmn" Oct 31 13:55:29.777495 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:29.777353 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/696e6f49-45c1-4504-875e-86b899b405c4-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-sxnmn\" (UID: \"696e6f49-45c1-4504-875e-86b899b405c4\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-sxnmn" Oct 31 13:55:29.777495 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:29.777400 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc568\" (UniqueName: \"kubernetes.io/projected/696e6f49-45c1-4504-875e-86b899b405c4-kube-api-access-pc568\") pod \"payload-processing-external-model-5bcd7997-sxnmn\" (UID: \"696e6f49-45c1-4504-875e-86b899b405c4\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-sxnmn" Oct 31 13:55:29.778478 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:29.778445 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/696e6f49-45c1-4504-875e-86b899b405c4-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-sxnmn\" (UID: \"696e6f49-45c1-4504-875e-86b899b405c4\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-sxnmn" Oct 31 13:55:29.778720 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:29.778692 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/696e6f49-45c1-4504-875e-86b899b405c4-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-sxnmn\" (UID: \"696e6f49-45c1-4504-875e-86b899b405c4\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-sxnmn" Oct 31 13:55:29.780297 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:29.780272 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/696e6f49-45c1-4504-875e-86b899b405c4-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-sxnmn\" (UID: \"696e6f49-45c1-4504-875e-86b899b405c4\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-sxnmn" Oct 31 13:55:29.791505 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:29.791478 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc568\" (UniqueName: \"kubernetes.io/projected/696e6f49-45c1-4504-875e-86b899b405c4-kube-api-access-pc568\") pod \"payload-processing-external-model-5bcd7997-sxnmn\" (UID: \"696e6f49-45c1-4504-875e-86b899b405c4\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-sxnmn" Oct 31 13:55:29.931638 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:29.931565 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-sxnmn" Oct 31 13:55:30.210747 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:30.210723 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-sxnmn"] Oct 31 13:55:30.212259 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:55:30.212220 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod696e6f49_45c1_4504_875e_86b899b405c4.slice/crio-24df9819e5fd198bf55ae1b4bdbf110a0d62d5cef88e9c608c3fc94ade08e0bd WatchSource:0}: Error finding container 24df9819e5fd198bf55ae1b4bdbf110a0d62d5cef88e9c608c3fc94ade08e0bd: Status 404 returned error can't find the container with id 24df9819e5fd198bf55ae1b4bdbf110a0d62d5cef88e9c608c3fc94ade08e0bd Oct 31 13:55:30.644489 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:30.644399 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:55:30.846285 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:30.846246 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:55:30.911618 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:30.911535 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-sxnmn" event={"ID":"696e6f49-45c1-4504-875e-86b899b405c4","Type":"ContainerStarted","Data":"a7a71064b7df02c2344e5b16b7f0aff54a589f5d69c4b5833dbc48ecad0ae1ea"} Oct 31 13:55:30.911618 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:30.911594 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-sxnmn" Oct 31 13:55:30.911618 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:30.911610 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-sxnmn" event={"ID":"696e6f49-45c1-4504-875e-86b899b405c4","Type":"ContainerStarted","Data":"24df9819e5fd198bf55ae1b4bdbf110a0d62d5cef88e9c608c3fc94ade08e0bd"} Oct 31 13:55:30.933531 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:30.933482 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-sxnmn" podStartSLOduration=1.933467472 podStartE2EDuration="1.933467472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:55:30.932332649 +0000 UTC m=+4977.415455552" watchObservedRunningTime="2026-10-31 13:55:30.933467472 +0000 UTC m=+4977.416590374" Oct 31 13:55:31.879150 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:31.879102 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-74c58f9c5b-xvm2h" podUID="7c619b77-1620-4d5a-ab6b-44413826f8a0" containerName="payload-processing" containerID="cri-o://0d3b71489b64504f92528423fdc87b5253ced6530f2e36eb7509d26bc0e9a244" gracePeriod=25 Oct 31 13:55:31.909735 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:31.909662 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-76c647798b-m46mv" podUID="32124ac7-9f3c-449e-84a0-691c8b108ead" containerName="payload-pre-processing" containerID="cri-o://eecc3155a5a4d6cfa6ce789523a931da639748a8af7fd5882059867092ef7531" gracePeriod=25 Oct 31 13:55:32.244496 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:32.244131 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-76c647798b-m46mv" Oct 31 13:55:32.277020 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:32.276684 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-74c58f9c5b-xvm2h" Oct 31 13:55:32.302916 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:32.299264 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32124ac7-9f3c-449e-84a0-691c8b108ead-kube-api-access-r7jmq\" (UniqueName: \"kubernetes.io/projected/32124ac7-9f3c-449e-84a0-691c8b108ead-kube-api-access-r7jmq\") pod \"32124ac7-9f3c-449e-84a0-691c8b108ead\" (UID: \"32124ac7-9f3c-449e-84a0-691c8b108ead\") " Oct 31 13:55:32.302916 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:32.299364 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32124ac7-9f3c-449e-84a0-691c8b108ead-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32124ac7-9f3c-449e-84a0-691c8b108ead-plugins-config-volume\") pod \"32124ac7-9f3c-449e-84a0-691c8b108ead\" (UID: \"32124ac7-9f3c-449e-84a0-691c8b108ead\") " Oct 31 13:55:32.302916 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:32.299398 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c619b77-1620-4d5a-ab6b-44413826f8a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c619b77-1620-4d5a-ab6b-44413826f8a0-plugins-config-volume\") pod \"7c619b77-1620-4d5a-ab6b-44413826f8a0\" (UID: \"7c619b77-1620-4d5a-ab6b-44413826f8a0\") " Oct 31 13:55:32.302916 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:32.299509 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c619b77-1620-4d5a-ab6b-44413826f8a0-kube-api-access-cl7ks\" (UniqueName: \"kubernetes.io/projected/7c619b77-1620-4d5a-ab6b-44413826f8a0-kube-api-access-cl7ks\") pod \"7c619b77-1620-4d5a-ab6b-44413826f8a0\" (UID: \"7c619b77-1620-4d5a-ab6b-44413826f8a0\") " Oct 31 13:55:32.302916 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:32.300638 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32124ac7-9f3c-449e-84a0-691c8b108ead-plugins-config-volume" pod "32124ac7-9f3c-449e-84a0-691c8b108ead" (UID: "32124ac7-9f3c-449e-84a0-691c8b108ead"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:55:32.302916 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:32.302758 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c619b77-1620-4d5a-ab6b-44413826f8a0-plugins-config-volume" pod "7c619b77-1620-4d5a-ab6b-44413826f8a0" (UID: "7c619b77-1620-4d5a-ab6b-44413826f8a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:55:32.318030 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:32.317965 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c619b77-1620-4d5a-ab6b-44413826f8a0-kube-api-access-cl7ks" pod "7c619b77-1620-4d5a-ab6b-44413826f8a0" (UID: "7c619b77-1620-4d5a-ab6b-44413826f8a0"). InnerVolumeSpecName "kube-api-access-cl7ks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:55:32.318590 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:32.318561 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32124ac7-9f3c-449e-84a0-691c8b108ead-kube-api-access-r7jmq" pod "32124ac7-9f3c-449e-84a0-691c8b108ead" (UID: "32124ac7-9f3c-449e-84a0-691c8b108ead"). InnerVolumeSpecName "kube-api-access-r7jmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:55:32.405094 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:32.404677 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl7ks\" (UniqueName: \"kubernetes.io/projected/7c619b77-1620-4d5a-ab6b-44413826f8a0-kube-api-access-cl7ks\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:32.405094 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:32.404736 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7jmq\" (UniqueName: \"kubernetes.io/projected/32124ac7-9f3c-449e-84a0-691c8b108ead-kube-api-access-r7jmq\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:32.405094 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:32.404753 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32124ac7-9f3c-449e-84a0-691c8b108ead-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:32.405094 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:32.404772 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c619b77-1620-4d5a-ab6b-44413826f8a0-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:33.071006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:33.070719 6841 generic.go:356] "Generic (PLEG): container finished" podID="32124ac7-9f3c-449e-84a0-691c8b108ead" containerID="eecc3155a5a4d6cfa6ce789523a931da639748a8af7fd5882059867092ef7531" exitCode=0 Oct 31 13:55:33.071006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:33.070940 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-76c647798b-m46mv" event={"ID":"32124ac7-9f3c-449e-84a0-691c8b108ead","Type":"ContainerDied","Data":"eecc3155a5a4d6cfa6ce789523a931da639748a8af7fd5882059867092ef7531"} Oct 31 13:55:33.079004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:33.075097 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-76c647798b-m46mv" event={"ID":"32124ac7-9f3c-449e-84a0-691c8b108ead","Type":"ContainerDied","Data":"dfc61245a88d4a43ba573e6cae9b7f9862f376ba24412877e9d08c4e7724fe35"} Oct 31 13:55:33.079004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:33.075156 6841 scope.go:122] "RemoveContainer" containerID="eecc3155a5a4d6cfa6ce789523a931da639748a8af7fd5882059867092ef7531" Oct 31 13:55:33.079004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:33.075331 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-76c647798b-m46mv" Oct 31 13:55:33.123003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:33.119735 6841 generic.go:356] "Generic (PLEG): container finished" podID="7c619b77-1620-4d5a-ab6b-44413826f8a0" containerID="0d3b71489b64504f92528423fdc87b5253ced6530f2e36eb7509d26bc0e9a244" exitCode=0 Oct 31 13:55:33.123003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:33.119808 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-74c58f9c5b-xvm2h" event={"ID":"7c619b77-1620-4d5a-ab6b-44413826f8a0","Type":"ContainerDied","Data":"0d3b71489b64504f92528423fdc87b5253ced6530f2e36eb7509d26bc0e9a244"} Oct 31 13:55:33.123003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:33.119879 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-74c58f9c5b-xvm2h" event={"ID":"7c619b77-1620-4d5a-ab6b-44413826f8a0","Type":"ContainerDied","Data":"b4ab872d9f0a072e62022d1b12882391be07e177ff4afba3a7811b7eecb2347a"} Oct 31 13:55:33.123003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:33.120023 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-74c58f9c5b-xvm2h" Oct 31 13:55:33.181004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:33.171709 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-76c647798b-m46mv"] Oct 31 13:55:33.181004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:33.171893 6841 scope.go:122] "RemoveContainer" containerID="eecc3155a5a4d6cfa6ce789523a931da639748a8af7fd5882059867092ef7531" Oct 31 13:55:33.182350 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:55:33.181931 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eecc3155a5a4d6cfa6ce789523a931da639748a8af7fd5882059867092ef7531\": container with ID starting with eecc3155a5a4d6cfa6ce789523a931da639748a8af7fd5882059867092ef7531 not found: ID does not exist" containerID="eecc3155a5a4d6cfa6ce789523a931da639748a8af7fd5882059867092ef7531" Oct 31 13:55:33.182350 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:33.181999 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eecc3155a5a4d6cfa6ce789523a931da639748a8af7fd5882059867092ef7531"} err="failed to get container status \"eecc3155a5a4d6cfa6ce789523a931da639748a8af7fd5882059867092ef7531\": rpc error: code = NotFound desc = could not find container \"eecc3155a5a4d6cfa6ce789523a931da639748a8af7fd5882059867092ef7531\": container with ID starting with eecc3155a5a4d6cfa6ce789523a931da639748a8af7fd5882059867092ef7531 not found: ID does not exist" Oct 31 13:55:33.182350 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:33.182028 6841 scope.go:122] "RemoveContainer" containerID="0d3b71489b64504f92528423fdc87b5253ced6530f2e36eb7509d26bc0e9a244" Oct 31 13:55:33.186127 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:33.186073 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-76c647798b-m46mv"] Oct 31 13:55:33.218064 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:33.218036 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-74c58f9c5b-xvm2h"] Oct 31 13:55:33.221531 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:33.221503 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-74c58f9c5b-xvm2h"] Oct 31 13:55:33.247002 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:33.243483 6841 scope.go:122] "RemoveContainer" containerID="0d3b71489b64504f92528423fdc87b5253ced6530f2e36eb7509d26bc0e9a244" Oct 31 13:55:33.247002 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:55:33.244131 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d3b71489b64504f92528423fdc87b5253ced6530f2e36eb7509d26bc0e9a244\": container with ID starting with 0d3b71489b64504f92528423fdc87b5253ced6530f2e36eb7509d26bc0e9a244 not found: ID does not exist" containerID="0d3b71489b64504f92528423fdc87b5253ced6530f2e36eb7509d26bc0e9a244" Oct 31 13:55:33.247002 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:33.244182 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d3b71489b64504f92528423fdc87b5253ced6530f2e36eb7509d26bc0e9a244"} err="failed to get container status \"0d3b71489b64504f92528423fdc87b5253ced6530f2e36eb7509d26bc0e9a244\": rpc error: code = NotFound desc = could not find container \"0d3b71489b64504f92528423fdc87b5253ced6530f2e36eb7509d26bc0e9a244\": container with ID starting with 0d3b71489b64504f92528423fdc87b5253ced6530f2e36eb7509d26bc0e9a244 not found: ID does not exist" Oct 31 13:55:33.633006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:33.629109 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-xn5wx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:55:33.633006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:33.629220 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-xn5wx" podUID="2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:33.693006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:33.692849 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32124ac7-9f3c-449e-84a0-691c8b108ead" path="/var/lib/kubelet/pods/32124ac7-9f3c-449e-84a0-691c8b108ead/volumes" Oct 31 13:55:33.698025 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:33.693477 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c619b77-1620-4d5a-ab6b-44413826f8a0" path="/var/lib/kubelet/pods/7c619b77-1620-4d5a-ab6b-44413826f8a0/volumes" Oct 31 13:55:33.868006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:33.867080 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-bp2d4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:55:33.868006 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:33.867149 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" podUID="51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:41.588342 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:41.588310 6841 scope.go:122] "RemoveContainer" containerID="cec2bd58920f98715aebd07dc7385f4451c5798911c59f95600b1665e75309d9" Oct 31 13:55:41.597523 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:41.597502 6841 scope.go:122] "RemoveContainer" containerID="1158f2a756832676f4b89c597c48b565de6a5035be08653799d7d788c61b963a" Oct 31 13:55:41.605526 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:41.605492 6841 scope.go:122] "RemoveContainer" containerID="bed11a3c1258f98e70f2a968df7febe322fc01e66213922bb07f2cdcf80d2d5d" Oct 31 13:55:41.715456 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:41.715428 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:55:41.924114 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:55:41.924041 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-sxnmn" Oct 31 13:55:50.034172 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:55:50.034132 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:56:20.033524 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:56:20.033491 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:56:41.641864 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:41.641832 6841 scope.go:122] "RemoveContainer" containerID="90c52c8f4490aca0374e7a6231aebf69b42b4035e0bd2a083c315b358d7da8fd" Oct 31 13:56:41.777088 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:41.777054 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:56:50.033743 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:56:50.033654 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:56:57.060933 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:57.060884 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-w724m"] Oct 31 13:56:57.061445 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:57.061429 6841 cpu_manager.go:414] "containerMap: removing container" podUID="32124ac7-9f3c-449e-84a0-691c8b108ead" containerName="payload-pre-processing" Oct 31 13:56:57.061494 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:57.061446 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32124ac7-9f3c-449e-84a0-691c8b108ead" containerName="payload-pre-processing" Oct 31 13:56:57.061494 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:57.061467 6841 cpu_manager.go:414] "containerMap: removing container" podUID="7c619b77-1620-4d5a-ab6b-44413826f8a0" containerName="payload-processing" Oct 31 13:56:57.061494 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:57.061473 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c619b77-1620-4d5a-ab6b-44413826f8a0" containerName="payload-processing" Oct 31 13:56:57.061598 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:57.061586 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c619b77-1620-4d5a-ab6b-44413826f8a0" containerName="payload-processing" Oct 31 13:56:57.061633 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:57.061604 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="32124ac7-9f3c-449e-84a0-691c8b108ead" containerName="payload-pre-processing" Oct 31 13:56:57.066421 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:57.066395 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-w724m" Oct 31 13:56:57.072053 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:57.072025 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:56:57.075626 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:57.072283 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:56:57.075626 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:57.072387 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 13:56:57.075626 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:57.072463 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:56:57.076877 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:57.076850 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-w724m"] Oct 31 13:56:57.105946 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:57.105916 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7756303-b1fa-4e6b-9ecb-dea12e211565-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-w724m\" (UID: \"e7756303-b1fa-4e6b-9ecb-dea12e211565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-w724m" Oct 31 13:56:57.106119 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:57.105952 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7756303-b1fa-4e6b-9ecb-dea12e211565-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-w724m\" (UID: \"e7756303-b1fa-4e6b-9ecb-dea12e211565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-w724m" Oct 31 13:56:57.106119 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:57.106078 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc7qj\" (UniqueName: \"kubernetes.io/projected/e7756303-b1fa-4e6b-9ecb-dea12e211565-kube-api-access-pc7qj\") pod \"maas-api-revoke-sub-e2547622-w724m\" (UID: \"e7756303-b1fa-4e6b-9ecb-dea12e211565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-w724m" Oct 31 13:56:57.207549 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:57.207515 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7756303-b1fa-4e6b-9ecb-dea12e211565-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-w724m\" (UID: \"e7756303-b1fa-4e6b-9ecb-dea12e211565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-w724m" Oct 31 13:56:57.207748 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:57.207560 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7756303-b1fa-4e6b-9ecb-dea12e211565-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-w724m\" (UID: \"e7756303-b1fa-4e6b-9ecb-dea12e211565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-w724m" Oct 31 13:56:57.207748 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:57.207594 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc7qj\" (UniqueName: \"kubernetes.io/projected/e7756303-b1fa-4e6b-9ecb-dea12e211565-kube-api-access-pc7qj\") pod \"maas-api-revoke-sub-e2547622-w724m\" (UID: \"e7756303-b1fa-4e6b-9ecb-dea12e211565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-w724m" Oct 31 13:56:57.208252 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:57.208224 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7756303-b1fa-4e6b-9ecb-dea12e211565-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-w724m\" (UID: \"e7756303-b1fa-4e6b-9ecb-dea12e211565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-w724m" Oct 31 13:56:57.210089 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:57.210058 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7756303-b1fa-4e6b-9ecb-dea12e211565-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-w724m\" (UID: \"e7756303-b1fa-4e6b-9ecb-dea12e211565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-w724m" Oct 31 13:56:57.220336 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:57.220313 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc7qj\" (UniqueName: \"kubernetes.io/projected/e7756303-b1fa-4e6b-9ecb-dea12e211565-kube-api-access-pc7qj\") pod \"maas-api-revoke-sub-e2547622-w724m\" (UID: \"e7756303-b1fa-4e6b-9ecb-dea12e211565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-w724m" Oct 31 13:56:57.384767 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:57.384683 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-w724m" Oct 31 13:56:57.661902 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:57.661878 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-w724m"] Oct 31 13:56:57.664090 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:56:57.664063 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7756303_b1fa_4e6b_9ecb_dea12e211565.slice/crio-e70f7564235860a079fab0f04af72a0c12c4c1a8adc329dc9b28e52e16a8c9f0 WatchSource:0}: Error finding container e70f7564235860a079fab0f04af72a0c12c4c1a8adc329dc9b28e52e16a8c9f0: Status 404 returned error can't find the container with id e70f7564235860a079fab0f04af72a0c12c4c1a8adc329dc9b28e52e16a8c9f0 Oct 31 13:56:58.600887 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:58.600854 6841 generic.go:356] "Generic (PLEG): container finished" podID="e7756303-b1fa-4e6b-9ecb-dea12e211565" containerID="46616dfced959209a6a572c0daf3fcec76f39e82bdaebe7dae224fad85d19845" exitCode=0 Oct 31 13:56:58.601301 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:58.600909 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-w724m" event={"ID":"e7756303-b1fa-4e6b-9ecb-dea12e211565","Type":"ContainerDied","Data":"46616dfced959209a6a572c0daf3fcec76f39e82bdaebe7dae224fad85d19845"} Oct 31 13:56:58.601301 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:58.600957 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-w724m" event={"ID":"e7756303-b1fa-4e6b-9ecb-dea12e211565","Type":"ContainerStarted","Data":"e70f7564235860a079fab0f04af72a0c12c4c1a8adc329dc9b28e52e16a8c9f0"} Oct 31 13:56:59.743751 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:59.743726 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-w724m" Oct 31 13:56:59.835462 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:59.835427 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7756303-b1fa-4e6b-9ecb-dea12e211565-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7756303-b1fa-4e6b-9ecb-dea12e211565-maas-api-openshift-ca\") pod \"e7756303-b1fa-4e6b-9ecb-dea12e211565\" (UID: \"e7756303-b1fa-4e6b-9ecb-dea12e211565\") " Oct 31 13:56:59.835628 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:59.835516 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7756303-b1fa-4e6b-9ecb-dea12e211565-kube-api-access-pc7qj\" (UniqueName: \"kubernetes.io/projected/e7756303-b1fa-4e6b-9ecb-dea12e211565-kube-api-access-pc7qj\") pod \"e7756303-b1fa-4e6b-9ecb-dea12e211565\" (UID: \"e7756303-b1fa-4e6b-9ecb-dea12e211565\") " Oct 31 13:56:59.835628 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:59.835557 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7756303-b1fa-4e6b-9ecb-dea12e211565-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7756303-b1fa-4e6b-9ecb-dea12e211565-maas-api-xks-ca\") pod \"e7756303-b1fa-4e6b-9ecb-dea12e211565\" (UID: \"e7756303-b1fa-4e6b-9ecb-dea12e211565\") " Oct 31 13:56:59.835886 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:59.835848 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7756303-b1fa-4e6b-9ecb-dea12e211565-maas-api-openshift-ca" pod "e7756303-b1fa-4e6b-9ecb-dea12e211565" (UID: "e7756303-b1fa-4e6b-9ecb-dea12e211565"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:56:59.837667 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:59.837642 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7756303-b1fa-4e6b-9ecb-dea12e211565-maas-api-xks-ca" pod "e7756303-b1fa-4e6b-9ecb-dea12e211565" (UID: "e7756303-b1fa-4e6b-9ecb-dea12e211565"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:56:59.837777 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:59.837766 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7756303-b1fa-4e6b-9ecb-dea12e211565-kube-api-access-pc7qj" pod "e7756303-b1fa-4e6b-9ecb-dea12e211565" (UID: "e7756303-b1fa-4e6b-9ecb-dea12e211565"). InnerVolumeSpecName "kube-api-access-pc7qj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:56:59.937416 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:59.937350 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc7qj\" (UniqueName: \"kubernetes.io/projected/e7756303-b1fa-4e6b-9ecb-dea12e211565-kube-api-access-pc7qj\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:59.937416 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:59.937376 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7756303-b1fa-4e6b-9ecb-dea12e211565-maas-api-xks-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:59.937416 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:56:59.937386 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7756303-b1fa-4e6b-9ecb-dea12e211565-maas-api-openshift-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:00.612772 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:00.612732 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-w724m" event={"ID":"e7756303-b1fa-4e6b-9ecb-dea12e211565","Type":"ContainerDied","Data":"e70f7564235860a079fab0f04af72a0c12c4c1a8adc329dc9b28e52e16a8c9f0"} Oct 31 13:57:00.612772 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:00.612766 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-w724m" Oct 31 13:57:00.613040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:00.612792 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e70f7564235860a079fab0f04af72a0c12c4c1a8adc329dc9b28e52e16a8c9f0" Oct 31 13:57:07.655124 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:07.655089 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-sxnmn"] Oct 31 13:57:07.793720 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:07.793684 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:57:09.591417 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:09.591314 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:57:12.054004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:12.053186 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fk2tx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:57:12.054004 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:12.053328 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" podUID="aa7e0a48-9dc5-4735-b1ae-cafd28ff982f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:57:12.658089 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:12.658044 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-sxnmn" podUID="696e6f49-45c1-4504-875e-86b899b405c4" containerName="payload-processing" containerID="cri-o://a7a71064b7df02c2344e5b16b7f0aff54a589f5d69c4b5833dbc48ecad0ae1ea" gracePeriod=25 Oct 31 13:57:13.293993 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:13.293955 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-sxnmn" Oct 31 13:57:13.300127 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:13.300094 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/696e6f49-45c1-4504-875e-86b899b405c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/696e6f49-45c1-4504-875e-86b899b405c4-plugins-config-volume\") pod \"696e6f49-45c1-4504-875e-86b899b405c4\" (UID: \"696e6f49-45c1-4504-875e-86b899b405c4\") " Oct 31 13:57:13.300324 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:13.300162 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/696e6f49-45c1-4504-875e-86b899b405c4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/696e6f49-45c1-4504-875e-86b899b405c4-routing-overlay-volume\") pod \"696e6f49-45c1-4504-875e-86b899b405c4\" (UID: \"696e6f49-45c1-4504-875e-86b899b405c4\") " Oct 31 13:57:13.300324 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:13.300222 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/696e6f49-45c1-4504-875e-86b899b405c4-kube-api-access-pc568\" (UniqueName: \"kubernetes.io/projected/696e6f49-45c1-4504-875e-86b899b405c4-kube-api-access-pc568\") pod \"696e6f49-45c1-4504-875e-86b899b405c4\" (UID: \"696e6f49-45c1-4504-875e-86b899b405c4\") " Oct 31 13:57:13.300324 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:13.300243 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/696e6f49-45c1-4504-875e-86b899b405c4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/696e6f49-45c1-4504-875e-86b899b405c4-provider-credentials\") pod \"696e6f49-45c1-4504-875e-86b899b405c4\" (UID: \"696e6f49-45c1-4504-875e-86b899b405c4\") " Oct 31 13:57:13.300583 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:13.300550 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/696e6f49-45c1-4504-875e-86b899b405c4-plugins-config-volume" pod "696e6f49-45c1-4504-875e-86b899b405c4" (UID: "696e6f49-45c1-4504-875e-86b899b405c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:57:13.300659 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:13.300595 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/696e6f49-45c1-4504-875e-86b899b405c4-routing-overlay-volume" pod "696e6f49-45c1-4504-875e-86b899b405c4" (UID: "696e6f49-45c1-4504-875e-86b899b405c4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:57:13.302646 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:13.302607 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/696e6f49-45c1-4504-875e-86b899b405c4-provider-credentials" pod "696e6f49-45c1-4504-875e-86b899b405c4" (UID: "696e6f49-45c1-4504-875e-86b899b405c4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:57:13.302967 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:13.302946 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/696e6f49-45c1-4504-875e-86b899b405c4-kube-api-access-pc568" pod "696e6f49-45c1-4504-875e-86b899b405c4" (UID: "696e6f49-45c1-4504-875e-86b899b405c4"). InnerVolumeSpecName "kube-api-access-pc568". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:57:13.401540 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:13.401450 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/696e6f49-45c1-4504-875e-86b899b405c4-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:13.401540 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:13.401482 6841 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/696e6f49-45c1-4504-875e-86b899b405c4-routing-overlay-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:13.401540 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:13.401491 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc568\" (UniqueName: \"kubernetes.io/projected/696e6f49-45c1-4504-875e-86b899b405c4-kube-api-access-pc568\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:13.401540 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:13.401500 6841 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/696e6f49-45c1-4504-875e-86b899b405c4-provider-credentials\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:13.809074 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:13.809035 6841 generic.go:356] "Generic (PLEG): container finished" podID="696e6f49-45c1-4504-875e-86b899b405c4" containerID="a7a71064b7df02c2344e5b16b7f0aff54a589f5d69c4b5833dbc48ecad0ae1ea" exitCode=0 Oct 31 13:57:13.809301 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:13.809103 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-sxnmn" Oct 31 13:57:13.809301 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:13.809225 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-sxnmn" event={"ID":"696e6f49-45c1-4504-875e-86b899b405c4","Type":"ContainerDied","Data":"a7a71064b7df02c2344e5b16b7f0aff54a589f5d69c4b5833dbc48ecad0ae1ea"} Oct 31 13:57:13.809301 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:13.809275 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-sxnmn" event={"ID":"696e6f49-45c1-4504-875e-86b899b405c4","Type":"ContainerDied","Data":"24df9819e5fd198bf55ae1b4bdbf110a0d62d5cef88e9c608c3fc94ade08e0bd"} Oct 31 13:57:13.809500 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:13.809306 6841 scope.go:122] "RemoveContainer" containerID="a7a71064b7df02c2344e5b16b7f0aff54a589f5d69c4b5833dbc48ecad0ae1ea" Oct 31 13:57:13.823713 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:13.823689 6841 scope.go:122] "RemoveContainer" containerID="a7a71064b7df02c2344e5b16b7f0aff54a589f5d69c4b5833dbc48ecad0ae1ea" Oct 31 13:57:13.824100 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:57:13.824059 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7a71064b7df02c2344e5b16b7f0aff54a589f5d69c4b5833dbc48ecad0ae1ea\": container with ID starting with a7a71064b7df02c2344e5b16b7f0aff54a589f5d69c4b5833dbc48ecad0ae1ea not found: ID does not exist" containerID="a7a71064b7df02c2344e5b16b7f0aff54a589f5d69c4b5833dbc48ecad0ae1ea" Oct 31 13:57:13.824269 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:13.824244 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7a71064b7df02c2344e5b16b7f0aff54a589f5d69c4b5833dbc48ecad0ae1ea"} err="failed to get container status \"a7a71064b7df02c2344e5b16b7f0aff54a589f5d69c4b5833dbc48ecad0ae1ea\": rpc error: code = NotFound desc = could not find container \"a7a71064b7df02c2344e5b16b7f0aff54a589f5d69c4b5833dbc48ecad0ae1ea\": container with ID starting with a7a71064b7df02c2344e5b16b7f0aff54a589f5d69c4b5833dbc48ecad0ae1ea not found: ID does not exist" Oct 31 13:57:13.830683 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:13.830655 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-sxnmn"] Oct 31 13:57:13.835637 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:13.835617 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-sxnmn"] Oct 31 13:57:15.659412 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:15.659358 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="696e6f49-45c1-4504-875e-86b899b405c4" path="/var/lib/kubelet/pods/696e6f49-45c1-4504-875e-86b899b405c4/volumes" Oct 31 13:57:16.024015 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:16.023951 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cc05b22-hp424"] Oct 31 13:57:16.027396 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:16.027367 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cc05b22-hp424"] Oct 31 13:57:17.478020 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:17.477969 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cxsxg"] Oct 31 13:57:17.478522 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:17.478502 6841 cpu_manager.go:414] "containerMap: removing container" podUID="696e6f49-45c1-4504-875e-86b899b405c4" containerName="payload-processing" Oct 31 13:57:17.478606 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:17.478525 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="696e6f49-45c1-4504-875e-86b899b405c4" containerName="payload-processing" Oct 31 13:57:17.478606 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:17.478547 6841 cpu_manager.go:414] "containerMap: removing container" podUID="e7756303-b1fa-4e6b-9ecb-dea12e211565" containerName="revoke-keys" Oct 31 13:57:17.478606 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:17.478555 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7756303-b1fa-4e6b-9ecb-dea12e211565" containerName="revoke-keys" Oct 31 13:57:17.478739 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:17.478672 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="696e6f49-45c1-4504-875e-86b899b405c4" containerName="payload-processing" Oct 31 13:57:17.478739 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:17.478687 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7756303-b1fa-4e6b-9ecb-dea12e211565" containerName="revoke-keys" Oct 31 13:57:17.482940 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:17.482921 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cxsxg" Oct 31 13:57:17.487691 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:17.487664 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:57:17.488689 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:17.488654 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:57:17.488808 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:17.488692 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:57:17.488864 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:17.488838 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 13:57:17.491707 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:17.491683 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cxsxg"] Oct 31 13:57:17.641800 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:17.641764 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a162f846-3c4e-4905-9172-fab1fcbac33f" path="/var/lib/kubelet/pods/a162f846-3c4e-4905-9172-fab1fcbac33f/volumes" Oct 31 13:57:17.663450 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:17.663418 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cb24607-1513-4cd1-b3d1-ec45ec1f26b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-cxsxg\" (UID: \"2cb24607-1513-4cd1-b3d1-ec45ec1f26b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cxsxg" Oct 31 13:57:17.663450 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:17.663451 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cb24607-1513-4cd1-b3d1-ec45ec1f26b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-cxsxg\" (UID: \"2cb24607-1513-4cd1-b3d1-ec45ec1f26b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cxsxg" Oct 31 13:57:17.663675 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:17.663501 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4z4s\" (UniqueName: \"kubernetes.io/projected/2cb24607-1513-4cd1-b3d1-ec45ec1f26b5-kube-api-access-t4z4s\") pod \"maas-api-revoke-sub-c1e271a9-cxsxg\" (UID: \"2cb24607-1513-4cd1-b3d1-ec45ec1f26b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cxsxg" Oct 31 13:57:17.764714 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:17.764609 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cb24607-1513-4cd1-b3d1-ec45ec1f26b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-cxsxg\" (UID: \"2cb24607-1513-4cd1-b3d1-ec45ec1f26b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cxsxg" Oct 31 13:57:17.764714 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:17.764663 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cb24607-1513-4cd1-b3d1-ec45ec1f26b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-cxsxg\" (UID: \"2cb24607-1513-4cd1-b3d1-ec45ec1f26b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cxsxg" Oct 31 13:57:17.765032 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:17.764736 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4z4s\" (UniqueName: \"kubernetes.io/projected/2cb24607-1513-4cd1-b3d1-ec45ec1f26b5-kube-api-access-t4z4s\") pod \"maas-api-revoke-sub-c1e271a9-cxsxg\" (UID: \"2cb24607-1513-4cd1-b3d1-ec45ec1f26b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cxsxg" Oct 31 13:57:17.765496 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:17.765473 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cb24607-1513-4cd1-b3d1-ec45ec1f26b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-cxsxg\" (UID: \"2cb24607-1513-4cd1-b3d1-ec45ec1f26b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cxsxg" Oct 31 13:57:17.767216 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:17.767189 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cb24607-1513-4cd1-b3d1-ec45ec1f26b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-cxsxg\" (UID: \"2cb24607-1513-4cd1-b3d1-ec45ec1f26b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cxsxg" Oct 31 13:57:17.778069 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:17.778042 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4z4s\" (UniqueName: \"kubernetes.io/projected/2cb24607-1513-4cd1-b3d1-ec45ec1f26b5-kube-api-access-t4z4s\") pod \"maas-api-revoke-sub-c1e271a9-cxsxg\" (UID: \"2cb24607-1513-4cd1-b3d1-ec45ec1f26b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cxsxg" Oct 31 13:57:17.799711 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:17.799690 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cxsxg" Oct 31 13:57:18.078217 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:18.078184 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cxsxg"] Oct 31 13:57:18.079837 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:57:18.079809 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb24607_1513_4cd1_b3d1_ec45ec1f26b5.slice/crio-08232cfafaa08ad03724ccdf78d426790228773c859f8dfc80a435721d38b9f9 WatchSource:0}: Error finding container 08232cfafaa08ad03724ccdf78d426790228773c859f8dfc80a435721d38b9f9: Status 404 returned error can't find the container with id 08232cfafaa08ad03724ccdf78d426790228773c859f8dfc80a435721d38b9f9 Oct 31 13:57:18.855516 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:18.855483 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cxsxg" event={"ID":"2cb24607-1513-4cd1-b3d1-ec45ec1f26b5","Type":"ContainerStarted","Data":"5cc6033e95ea8a00ac5f2a9b59fb74697b87895efb53008605ce8d374b0cc0f1"} Oct 31 13:57:18.855911 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:18.855540 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cxsxg" event={"ID":"2cb24607-1513-4cd1-b3d1-ec45ec1f26b5","Type":"ContainerStarted","Data":"08232cfafaa08ad03724ccdf78d426790228773c859f8dfc80a435721d38b9f9"} Oct 31 13:57:18.875554 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:18.875504 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cxsxg" podStartSLOduration=1.875490932 podStartE2EDuration="1.875490932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:57:18.872915984 +0000 UTC m=+5085.356038882" watchObservedRunningTime="2026-10-31 13:57:18.875490932 +0000 UTC m=+5085.358613833" Oct 31 13:57:19.862746 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:19.862713 6841 generic.go:356] "Generic (PLEG): container finished" podID="2cb24607-1513-4cd1-b3d1-ec45ec1f26b5" containerID="5cc6033e95ea8a00ac5f2a9b59fb74697b87895efb53008605ce8d374b0cc0f1" exitCode=0 Oct 31 13:57:19.863200 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:19.862797 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cxsxg" event={"ID":"2cb24607-1513-4cd1-b3d1-ec45ec1f26b5","Type":"ContainerDied","Data":"5cc6033e95ea8a00ac5f2a9b59fb74697b87895efb53008605ce8d374b0cc0f1"} Oct 31 13:57:20.034051 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:57:20.034018 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:57:21.016002 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:21.015969 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cxsxg" Oct 31 13:57:21.096706 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:21.096671 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cb24607-1513-4cd1-b3d1-ec45ec1f26b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cb24607-1513-4cd1-b3d1-ec45ec1f26b5-maas-api-xks-ca\") pod \"2cb24607-1513-4cd1-b3d1-ec45ec1f26b5\" (UID: \"2cb24607-1513-4cd1-b3d1-ec45ec1f26b5\") " Oct 31 13:57:21.096916 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:21.096718 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cb24607-1513-4cd1-b3d1-ec45ec1f26b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cb24607-1513-4cd1-b3d1-ec45ec1f26b5-maas-api-openshift-ca\") pod \"2cb24607-1513-4cd1-b3d1-ec45ec1f26b5\" (UID: \"2cb24607-1513-4cd1-b3d1-ec45ec1f26b5\") " Oct 31 13:57:21.096916 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:21.096775 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cb24607-1513-4cd1-b3d1-ec45ec1f26b5-kube-api-access-t4z4s\" (UniqueName: \"kubernetes.io/projected/2cb24607-1513-4cd1-b3d1-ec45ec1f26b5-kube-api-access-t4z4s\") pod \"2cb24607-1513-4cd1-b3d1-ec45ec1f26b5\" (UID: \"2cb24607-1513-4cd1-b3d1-ec45ec1f26b5\") " Oct 31 13:57:21.097153 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:21.097126 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cb24607-1513-4cd1-b3d1-ec45ec1f26b5-maas-api-openshift-ca" pod "2cb24607-1513-4cd1-b3d1-ec45ec1f26b5" (UID: "2cb24607-1513-4cd1-b3d1-ec45ec1f26b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:57:21.098941 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:21.098912 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cb24607-1513-4cd1-b3d1-ec45ec1f26b5-maas-api-xks-ca" pod "2cb24607-1513-4cd1-b3d1-ec45ec1f26b5" (UID: "2cb24607-1513-4cd1-b3d1-ec45ec1f26b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:57:21.099070 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:21.099005 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb24607-1513-4cd1-b3d1-ec45ec1f26b5-kube-api-access-t4z4s" pod "2cb24607-1513-4cd1-b3d1-ec45ec1f26b5" (UID: "2cb24607-1513-4cd1-b3d1-ec45ec1f26b5"). InnerVolumeSpecName "kube-api-access-t4z4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:57:21.197623 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:21.197594 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cb24607-1513-4cd1-b3d1-ec45ec1f26b5-maas-api-xks-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:21.197782 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:21.197634 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cb24607-1513-4cd1-b3d1-ec45ec1f26b5-maas-api-openshift-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:21.197782 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:21.197645 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4z4s\" (UniqueName: \"kubernetes.io/projected/2cb24607-1513-4cd1-b3d1-ec45ec1f26b5-kube-api-access-t4z4s\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:21.875109 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:21.875078 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cxsxg" Oct 31 13:57:21.875320 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:21.875112 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cxsxg" event={"ID":"2cb24607-1513-4cd1-b3d1-ec45ec1f26b5","Type":"ContainerDied","Data":"08232cfafaa08ad03724ccdf78d426790228773c859f8dfc80a435721d38b9f9"} Oct 31 13:57:21.875320 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:21.875190 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08232cfafaa08ad03724ccdf78d426790228773c859f8dfc80a435721d38b9f9" Oct 31 13:57:23.995589 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:23.995556 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:57:24.195834 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:24.195802 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:57:33.711385 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:33.711351 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ctnd7"] Oct 31 13:57:33.712052 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:33.712032 6841 cpu_manager.go:414] "containerMap: removing container" podUID="2cb24607-1513-4cd1-b3d1-ec45ec1f26b5" containerName="revoke-keys" Oct 31 13:57:33.712154 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:33.712054 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cb24607-1513-4cd1-b3d1-ec45ec1f26b5" containerName="revoke-keys" Oct 31 13:57:33.712222 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:33.712196 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cb24607-1513-4cd1-b3d1-ec45ec1f26b5" containerName="revoke-keys" Oct 31 13:57:33.715709 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:33.715688 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ctnd7" Oct 31 13:57:33.720284 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:33.720257 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:57:33.720974 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:33.720949 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 13:57:33.724144 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:33.723760 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ctnd7"] Oct 31 13:57:33.740689 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:33.740658 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:57:33.740919 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:33.740826 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:57:33.762616 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:33.762578 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fczqg\" (UniqueName: \"kubernetes.io/projected/4867ee48-7c82-47ab-9e23-93f4e2f87ae3-kube-api-access-fczqg\") pod \"maas-api-revoke-sub-2e1f5b5f-ctnd7\" (UID: \"4867ee48-7c82-47ab-9e23-93f4e2f87ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ctnd7" Oct 31 13:57:33.762816 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:33.762725 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4867ee48-7c82-47ab-9e23-93f4e2f87ae3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-ctnd7\" (UID: \"4867ee48-7c82-47ab-9e23-93f4e2f87ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ctnd7" Oct 31 13:57:33.762816 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:33.762758 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4867ee48-7c82-47ab-9e23-93f4e2f87ae3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-ctnd7\" (UID: \"4867ee48-7c82-47ab-9e23-93f4e2f87ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ctnd7" Oct 31 13:57:33.863735 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:33.863701 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4867ee48-7c82-47ab-9e23-93f4e2f87ae3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-ctnd7\" (UID: \"4867ee48-7c82-47ab-9e23-93f4e2f87ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ctnd7" Oct 31 13:57:33.863735 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:33.863737 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4867ee48-7c82-47ab-9e23-93f4e2f87ae3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-ctnd7\" (UID: \"4867ee48-7c82-47ab-9e23-93f4e2f87ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ctnd7" Oct 31 13:57:33.863930 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:33.863803 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fczqg\" (UniqueName: \"kubernetes.io/projected/4867ee48-7c82-47ab-9e23-93f4e2f87ae3-kube-api-access-fczqg\") pod \"maas-api-revoke-sub-2e1f5b5f-ctnd7\" (UID: \"4867ee48-7c82-47ab-9e23-93f4e2f87ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ctnd7" Oct 31 13:57:33.866724 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:33.866705 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:57:33.866787 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:33.866712 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:57:33.874585 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:33.874564 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4867ee48-7c82-47ab-9e23-93f4e2f87ae3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-ctnd7\" (UID: \"4867ee48-7c82-47ab-9e23-93f4e2f87ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ctnd7" Oct 31 13:57:33.876273 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:33.876256 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4867ee48-7c82-47ab-9e23-93f4e2f87ae3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-ctnd7\" (UID: \"4867ee48-7c82-47ab-9e23-93f4e2f87ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ctnd7" Oct 31 13:57:33.876589 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:33.876574 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:57:33.887001 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:33.886948 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fczqg\" (UniqueName: \"kubernetes.io/projected/4867ee48-7c82-47ab-9e23-93f4e2f87ae3-kube-api-access-fczqg\") pod \"maas-api-revoke-sub-2e1f5b5f-ctnd7\" (UID: \"4867ee48-7c82-47ab-9e23-93f4e2f87ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ctnd7" Oct 31 13:57:34.034060 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:34.033973 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 13:57:34.042034 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:34.042007 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ctnd7" Oct 31 13:57:34.318736 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:34.318707 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ctnd7"] Oct 31 13:57:34.320531 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:57:34.320497 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4867ee48_7c82_47ab_9e23_93f4e2f87ae3.slice/crio-74cbb1b00fc800e2bbc691b0ee9e38ec72bb866e289fb47520238aded3e14399 WatchSource:0}: Error finding container 74cbb1b00fc800e2bbc691b0ee9e38ec72bb866e289fb47520238aded3e14399: Status 404 returned error can't find the container with id 74cbb1b00fc800e2bbc691b0ee9e38ec72bb866e289fb47520238aded3e14399 Oct 31 13:57:34.998870 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:34.998834 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ctnd7" event={"ID":"4867ee48-7c82-47ab-9e23-93f4e2f87ae3","Type":"ContainerStarted","Data":"d67e100292d3b5594bcc3c085c2ffa01abe6fc32b4be362f8903a14928a51567"} Oct 31 13:57:34.999383 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:34.998894 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ctnd7" event={"ID":"4867ee48-7c82-47ab-9e23-93f4e2f87ae3","Type":"ContainerStarted","Data":"74cbb1b00fc800e2bbc691b0ee9e38ec72bb866e289fb47520238aded3e14399"} Oct 31 13:57:35.018673 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:35.018620 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ctnd7" podStartSLOduration=2.018608521 podStartE2EDuration="2.018608521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:57:35.016662057 +0000 UTC m=+5101.499784956" watchObservedRunningTime="2026-10-31 13:57:35.018608521 +0000 UTC m=+5101.501731422" Oct 31 13:57:36.005667 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:36.005632 6841 generic.go:356] "Generic (PLEG): container finished" podID="4867ee48-7c82-47ab-9e23-93f4e2f87ae3" containerID="d67e100292d3b5594bcc3c085c2ffa01abe6fc32b4be362f8903a14928a51567" exitCode=0 Oct 31 13:57:36.006087 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:36.005685 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ctnd7" event={"ID":"4867ee48-7c82-47ab-9e23-93f4e2f87ae3","Type":"ContainerDied","Data":"d67e100292d3b5594bcc3c085c2ffa01abe6fc32b4be362f8903a14928a51567"} Oct 31 13:57:37.156963 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:37.156931 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ctnd7" Oct 31 13:57:37.195341 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:37.195310 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4867ee48-7c82-47ab-9e23-93f4e2f87ae3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4867ee48-7c82-47ab-9e23-93f4e2f87ae3-maas-api-openshift-ca\") pod \"4867ee48-7c82-47ab-9e23-93f4e2f87ae3\" (UID: \"4867ee48-7c82-47ab-9e23-93f4e2f87ae3\") " Oct 31 13:57:37.195534 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:37.195384 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4867ee48-7c82-47ab-9e23-93f4e2f87ae3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4867ee48-7c82-47ab-9e23-93f4e2f87ae3-maas-api-xks-ca\") pod \"4867ee48-7c82-47ab-9e23-93f4e2f87ae3\" (UID: \"4867ee48-7c82-47ab-9e23-93f4e2f87ae3\") " Oct 31 13:57:37.195534 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:37.195421 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4867ee48-7c82-47ab-9e23-93f4e2f87ae3-kube-api-access-fczqg\" (UniqueName: \"kubernetes.io/projected/4867ee48-7c82-47ab-9e23-93f4e2f87ae3-kube-api-access-fczqg\") pod \"4867ee48-7c82-47ab-9e23-93f4e2f87ae3\" (UID: \"4867ee48-7c82-47ab-9e23-93f4e2f87ae3\") " Oct 31 13:57:37.195687 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:37.195664 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4867ee48-7c82-47ab-9e23-93f4e2f87ae3-maas-api-openshift-ca" pod "4867ee48-7c82-47ab-9e23-93f4e2f87ae3" (UID: "4867ee48-7c82-47ab-9e23-93f4e2f87ae3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:57:37.195847 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:37.195827 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4867ee48-7c82-47ab-9e23-93f4e2f87ae3-maas-api-openshift-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:37.197498 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:37.197471 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4867ee48-7c82-47ab-9e23-93f4e2f87ae3-maas-api-xks-ca" pod "4867ee48-7c82-47ab-9e23-93f4e2f87ae3" (UID: "4867ee48-7c82-47ab-9e23-93f4e2f87ae3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:57:37.197700 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:37.197683 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4867ee48-7c82-47ab-9e23-93f4e2f87ae3-kube-api-access-fczqg" pod "4867ee48-7c82-47ab-9e23-93f4e2f87ae3" (UID: "4867ee48-7c82-47ab-9e23-93f4e2f87ae3"). InnerVolumeSpecName "kube-api-access-fczqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:57:37.296929 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:37.296847 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4867ee48-7c82-47ab-9e23-93f4e2f87ae3-maas-api-xks-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:37.296929 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:37.296875 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fczqg\" (UniqueName: \"kubernetes.io/projected/4867ee48-7c82-47ab-9e23-93f4e2f87ae3-kube-api-access-fczqg\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:38.017731 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:38.017697 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ctnd7" Oct 31 13:57:38.017731 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:38.017711 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ctnd7" event={"ID":"4867ee48-7c82-47ab-9e23-93f4e2f87ae3","Type":"ContainerDied","Data":"74cbb1b00fc800e2bbc691b0ee9e38ec72bb866e289fb47520238aded3e14399"} Oct 31 13:57:38.017958 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:38.017765 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74cbb1b00fc800e2bbc691b0ee9e38ec72bb866e289fb47520238aded3e14399" Oct 31 13:57:41.830475 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:41.830437 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:57:44.693267 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:44.693228 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4rjgt"] Oct 31 13:57:44.693702 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:44.693685 6841 cpu_manager.go:414] "containerMap: removing container" podUID="4867ee48-7c82-47ab-9e23-93f4e2f87ae3" containerName="revoke-keys" Oct 31 13:57:44.693755 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:44.693704 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4867ee48-7c82-47ab-9e23-93f4e2f87ae3" containerName="revoke-keys" Oct 31 13:57:44.693804 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:44.693794 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="4867ee48-7c82-47ab-9e23-93f4e2f87ae3" containerName="revoke-keys" Oct 31 13:57:44.696932 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:44.696915 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4rjgt" Oct 31 13:57:44.707591 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:44.707529 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4rjgt"] Oct 31 13:57:44.708367 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:44.708342 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:57:44.708478 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:44.708342 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:57:44.708672 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:44.708648 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:57:44.708804 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:44.708787 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 13:57:44.769594 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:44.769558 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4qs6\" (UniqueName: \"kubernetes.io/projected/c7f0fe4e-47a4-43f3-a65b-e69b97144544-kube-api-access-g4qs6\") pod \"maas-api-revoke-sub-ec843018-4rjgt\" (UID: \"c7f0fe4e-47a4-43f3-a65b-e69b97144544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4rjgt" Oct 31 13:57:44.769802 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:44.769773 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7f0fe4e-47a4-43f3-a65b-e69b97144544-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-4rjgt\" (UID: \"c7f0fe4e-47a4-43f3-a65b-e69b97144544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4rjgt" Oct 31 13:57:44.769886 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:44.769869 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7f0fe4e-47a4-43f3-a65b-e69b97144544-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-4rjgt\" (UID: \"c7f0fe4e-47a4-43f3-a65b-e69b97144544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4rjgt" Oct 31 13:57:44.871110 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:44.871075 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4qs6\" (UniqueName: \"kubernetes.io/projected/c7f0fe4e-47a4-43f3-a65b-e69b97144544-kube-api-access-g4qs6\") pod \"maas-api-revoke-sub-ec843018-4rjgt\" (UID: \"c7f0fe4e-47a4-43f3-a65b-e69b97144544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4rjgt" Oct 31 13:57:44.871297 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:44.871180 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7f0fe4e-47a4-43f3-a65b-e69b97144544-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-4rjgt\" (UID: \"c7f0fe4e-47a4-43f3-a65b-e69b97144544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4rjgt" Oct 31 13:57:44.871297 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:44.871213 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7f0fe4e-47a4-43f3-a65b-e69b97144544-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-4rjgt\" (UID: \"c7f0fe4e-47a4-43f3-a65b-e69b97144544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4rjgt" Oct 31 13:57:44.871801 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:44.871774 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7f0fe4e-47a4-43f3-a65b-e69b97144544-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-4rjgt\" (UID: \"c7f0fe4e-47a4-43f3-a65b-e69b97144544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4rjgt" Oct 31 13:57:44.873519 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:44.873495 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7f0fe4e-47a4-43f3-a65b-e69b97144544-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-4rjgt\" (UID: \"c7f0fe4e-47a4-43f3-a65b-e69b97144544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4rjgt" Oct 31 13:57:44.898854 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:44.898820 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4qs6\" (UniqueName: \"kubernetes.io/projected/c7f0fe4e-47a4-43f3-a65b-e69b97144544-kube-api-access-g4qs6\") pod \"maas-api-revoke-sub-ec843018-4rjgt\" (UID: \"c7f0fe4e-47a4-43f3-a65b-e69b97144544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4rjgt" Oct 31 13:57:45.012822 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:45.012734 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4rjgt" Oct 31 13:57:45.294008 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:45.293967 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4rjgt"] Oct 31 13:57:46.063896 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:46.063863 6841 generic.go:356] "Generic (PLEG): container finished" podID="c7f0fe4e-47a4-43f3-a65b-e69b97144544" containerID="233e9dd8de307e70f9831a6126631a95b5892afc20f9cadf08b24adcec5542bb" exitCode=0 Oct 31 13:57:46.064305 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:46.063945 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4rjgt" event={"ID":"c7f0fe4e-47a4-43f3-a65b-e69b97144544","Type":"ContainerDied","Data":"233e9dd8de307e70f9831a6126631a95b5892afc20f9cadf08b24adcec5542bb"} Oct 31 13:57:46.064305 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:46.064014 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4rjgt" event={"ID":"c7f0fe4e-47a4-43f3-a65b-e69b97144544","Type":"ContainerStarted","Data":"bb0d27b72f055e8da7f7e4c6d508b680eea160a9522f47d670b9c2f277558552"} Oct 31 13:57:47.269147 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:47.268810 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4rjgt" Oct 31 13:57:47.400414 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:47.399620 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7f0fe4e-47a4-43f3-a65b-e69b97144544-kube-api-access-g4qs6\" (UniqueName: \"kubernetes.io/projected/c7f0fe4e-47a4-43f3-a65b-e69b97144544-kube-api-access-g4qs6\") pod \"c7f0fe4e-47a4-43f3-a65b-e69b97144544\" (UID: \"c7f0fe4e-47a4-43f3-a65b-e69b97144544\") " Oct 31 13:57:47.400414 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:47.399700 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7f0fe4e-47a4-43f3-a65b-e69b97144544-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7f0fe4e-47a4-43f3-a65b-e69b97144544-maas-api-openshift-ca\") pod \"c7f0fe4e-47a4-43f3-a65b-e69b97144544\" (UID: \"c7f0fe4e-47a4-43f3-a65b-e69b97144544\") " Oct 31 13:57:47.400414 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:47.399807 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7f0fe4e-47a4-43f3-a65b-e69b97144544-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7f0fe4e-47a4-43f3-a65b-e69b97144544-maas-api-xks-ca\") pod \"c7f0fe4e-47a4-43f3-a65b-e69b97144544\" (UID: \"c7f0fe4e-47a4-43f3-a65b-e69b97144544\") " Oct 31 13:57:47.400414 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:47.400373 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7f0fe4e-47a4-43f3-a65b-e69b97144544-maas-api-openshift-ca" pod "c7f0fe4e-47a4-43f3-a65b-e69b97144544" (UID: "c7f0fe4e-47a4-43f3-a65b-e69b97144544"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:57:47.402577 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:47.402540 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7f0fe4e-47a4-43f3-a65b-e69b97144544-maas-api-xks-ca" pod "c7f0fe4e-47a4-43f3-a65b-e69b97144544" (UID: "c7f0fe4e-47a4-43f3-a65b-e69b97144544"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:57:47.402723 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:47.402653 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f0fe4e-47a4-43f3-a65b-e69b97144544-kube-api-access-g4qs6" pod "c7f0fe4e-47a4-43f3-a65b-e69b97144544" (UID: "c7f0fe4e-47a4-43f3-a65b-e69b97144544"). InnerVolumeSpecName "kube-api-access-g4qs6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:57:47.501415 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:47.501346 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4qs6\" (UniqueName: \"kubernetes.io/projected/c7f0fe4e-47a4-43f3-a65b-e69b97144544-kube-api-access-g4qs6\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:47.501415 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:47.501407 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7f0fe4e-47a4-43f3-a65b-e69b97144544-maas-api-openshift-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:47.501415 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:47.501425 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7f0fe4e-47a4-43f3-a65b-e69b97144544-maas-api-xks-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:48.129843 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:48.129744 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4rjgt" event={"ID":"c7f0fe4e-47a4-43f3-a65b-e69b97144544","Type":"ContainerDied","Data":"bb0d27b72f055e8da7f7e4c6d508b680eea160a9522f47d670b9c2f277558552"} Oct 31 13:57:48.130128 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:48.130102 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4rjgt" Oct 31 13:57:48.130329 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:48.130100 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb0d27b72f055e8da7f7e4c6d508b680eea160a9522f47d670b9c2f277558552" Oct 31 13:57:48.882141 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:48.882099 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-bp2d4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:57:48.882790 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:57:48.882741 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bp2d4" podUID="51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:57:50.032732 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:57:50.032700 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:58:05.958596 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:05.958548 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:58:06.156576 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:06.156541 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:58:10.969415 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:10.969367 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sftzl"] Oct 31 13:58:10.969850 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:10.969834 6841 cpu_manager.go:414] "containerMap: removing container" podUID="c7f0fe4e-47a4-43f3-a65b-e69b97144544" containerName="revoke-keys" Oct 31 13:58:10.969897 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:10.969852 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7f0fe4e-47a4-43f3-a65b-e69b97144544" containerName="revoke-keys" Oct 31 13:58:10.970010 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:10.969974 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7f0fe4e-47a4-43f3-a65b-e69b97144544" containerName="revoke-keys" Oct 31 13:58:10.973480 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:10.973461 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sftzl" Oct 31 13:58:10.978695 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:10.978670 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:58:10.980362 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:10.980065 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:58:10.980362 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:10.980081 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:58:10.981341 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:10.981298 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 13:58:10.984584 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:10.984555 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sftzl"] Oct 31 13:58:11.014758 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:11.014719 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-sftzl\" (UID: \"6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sftzl" Oct 31 13:58:11.014930 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:11.014905 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-sftzl\" (UID: \"6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sftzl" Oct 31 13:58:11.015051 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:11.015031 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmbj9\" (UniqueName: \"kubernetes.io/projected/6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa-kube-api-access-mmbj9\") pod \"maas-api-revoke-sub-046e1343-sftzl\" (UID: \"6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sftzl" Oct 31 13:58:11.116218 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:11.116185 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-sftzl\" (UID: \"6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sftzl" Oct 31 13:58:11.116397 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:11.116245 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-sftzl\" (UID: \"6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sftzl" Oct 31 13:58:11.116397 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:11.116274 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmbj9\" (UniqueName: \"kubernetes.io/projected/6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa-kube-api-access-mmbj9\") pod \"maas-api-revoke-sub-046e1343-sftzl\" (UID: \"6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sftzl" Oct 31 13:58:11.116848 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:11.116826 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-sftzl\" (UID: \"6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sftzl" Oct 31 13:58:11.118583 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:11.118564 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-sftzl\" (UID: \"6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sftzl" Oct 31 13:58:11.128530 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:11.128506 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmbj9\" (UniqueName: \"kubernetes.io/projected/6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa-kube-api-access-mmbj9\") pod \"maas-api-revoke-sub-046e1343-sftzl\" (UID: \"6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sftzl" Oct 31 13:58:11.290261 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:11.290185 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sftzl" Oct 31 13:58:11.564958 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:11.564929 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sftzl"] Oct 31 13:58:11.566475 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:58:11.566436 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc1d5e6_8eb5_42e2_8349_06e66eddd4fa.slice/crio-4d3f3294b081a3093eb5f4bdc9bc49a7431019304f2ffae47bdf9938af1dd076 WatchSource:0}: Error finding container 4d3f3294b081a3093eb5f4bdc9bc49a7431019304f2ffae47bdf9938af1dd076: Status 404 returned error can't find the container with id 4d3f3294b081a3093eb5f4bdc9bc49a7431019304f2ffae47bdf9938af1dd076 Oct 31 13:58:12.302834 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:12.302798 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sftzl" event={"ID":"6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa","Type":"ContainerStarted","Data":"958e5ce033cad86a832a7f2efabc8fc8d2eec87a2399448617c72f86933f2c9d"} Oct 31 13:58:12.303292 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:12.302850 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sftzl" event={"ID":"6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa","Type":"ContainerStarted","Data":"4d3f3294b081a3093eb5f4bdc9bc49a7431019304f2ffae47bdf9938af1dd076"} Oct 31 13:58:12.323363 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:12.323320 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sftzl" podStartSLOduration=2.32330749 podStartE2EDuration="2.32330749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:58:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:58:12.322019429 +0000 UTC m=+5138.805142327" watchObservedRunningTime="2026-10-31 13:58:12.32330749 +0000 UTC m=+5138.806430403" Oct 31 13:58:13.311464 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:13.311427 6841 generic.go:356] "Generic (PLEG): container finished" podID="6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa" containerID="958e5ce033cad86a832a7f2efabc8fc8d2eec87a2399448617c72f86933f2c9d" exitCode=0 Oct 31 13:58:13.311866 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:13.311510 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sftzl" event={"ID":"6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa","Type":"ContainerDied","Data":"958e5ce033cad86a832a7f2efabc8fc8d2eec87a2399448617c72f86933f2c9d"} Oct 31 13:58:14.446281 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:14.446252 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sftzl" Oct 31 13:58:14.550566 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:14.550532 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa-maas-api-xks-ca\") pod \"6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa\" (UID: \"6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa\") " Oct 31 13:58:14.550566 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:14.550578 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa-maas-api-openshift-ca\") pod \"6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa\" (UID: \"6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa\") " Oct 31 13:58:14.550793 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:14.550631 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa-kube-api-access-mmbj9\" (UniqueName: \"kubernetes.io/projected/6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa-kube-api-access-mmbj9\") pod \"6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa\" (UID: \"6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa\") " Oct 31 13:58:14.551000 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:14.550958 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa-maas-api-openshift-ca" pod "6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa" (UID: "6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:58:14.552700 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:14.552672 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa-maas-api-xks-ca" pod "6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa" (UID: "6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:58:14.552808 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:14.552723 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa-kube-api-access-mmbj9" pod "6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa" (UID: "6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa"). InnerVolumeSpecName "kube-api-access-mmbj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:58:14.652304 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:14.652211 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa-maas-api-xks-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:58:14.652304 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:14.652249 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa-maas-api-openshift-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:58:14.652304 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:14.652264 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmbj9\" (UniqueName: \"kubernetes.io/projected/6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa-kube-api-access-mmbj9\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:58:15.325553 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:15.325525 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sftzl" Oct 31 13:58:15.325756 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:15.325522 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sftzl" event={"ID":"6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa","Type":"ContainerDied","Data":"4d3f3294b081a3093eb5f4bdc9bc49a7431019304f2ffae47bdf9938af1dd076"} Oct 31 13:58:15.325756 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:15.325651 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d3f3294b081a3093eb5f4bdc9bc49a7431019304f2ffae47bdf9938af1dd076" Oct 31 13:58:20.033827 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:58:20.033791 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:58:21.150425 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:21.150389 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:58:21.199195 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:21.199153 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8gpjs"] Oct 31 13:58:21.199897 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:21.199869 6841 cpu_manager.go:414] "containerMap: removing container" podUID="6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa" containerName="revoke-keys" Oct 31 13:58:21.199897 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:21.199893 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa" containerName="revoke-keys" Oct 31 13:58:21.200137 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:21.200074 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa" containerName="revoke-keys" Oct 31 13:58:21.203291 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:21.203265 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8gpjs" Oct 31 13:58:21.208727 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:21.208703 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:58:21.208866 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:21.208747 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 13:58:21.208866 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:21.208770 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:58:21.208866 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:21.208852 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:58:21.212521 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:21.212450 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8gpjs"] Oct 31 13:58:21.317517 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:21.317478 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4be917f1-8a0f-4e31-bb92-be147421d3dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-8gpjs\" (UID: \"4be917f1-8a0f-4e31-bb92-be147421d3dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8gpjs" Oct 31 13:58:21.317725 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:21.317543 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4be917f1-8a0f-4e31-bb92-be147421d3dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-8gpjs\" (UID: \"4be917f1-8a0f-4e31-bb92-be147421d3dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8gpjs" Oct 31 13:58:21.317725 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:21.317708 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhz7f\" (UniqueName: \"kubernetes.io/projected/4be917f1-8a0f-4e31-bb92-be147421d3dd-kube-api-access-xhz7f\") pod \"maas-api-revoke-sub-d3fba056-8gpjs\" (UID: \"4be917f1-8a0f-4e31-bb92-be147421d3dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8gpjs" Oct 31 13:58:21.418838 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:21.418739 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhz7f\" (UniqueName: \"kubernetes.io/projected/4be917f1-8a0f-4e31-bb92-be147421d3dd-kube-api-access-xhz7f\") pod \"maas-api-revoke-sub-d3fba056-8gpjs\" (UID: \"4be917f1-8a0f-4e31-bb92-be147421d3dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8gpjs" Oct 31 13:58:21.419067 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:21.418873 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4be917f1-8a0f-4e31-bb92-be147421d3dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-8gpjs\" (UID: \"4be917f1-8a0f-4e31-bb92-be147421d3dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8gpjs" Oct 31 13:58:21.419067 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:21.418912 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4be917f1-8a0f-4e31-bb92-be147421d3dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-8gpjs\" (UID: \"4be917f1-8a0f-4e31-bb92-be147421d3dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8gpjs" Oct 31 13:58:21.419627 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:21.419605 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4be917f1-8a0f-4e31-bb92-be147421d3dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-8gpjs\" (UID: \"4be917f1-8a0f-4e31-bb92-be147421d3dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8gpjs" Oct 31 13:58:21.421328 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:21.421304 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4be917f1-8a0f-4e31-bb92-be147421d3dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-8gpjs\" (UID: \"4be917f1-8a0f-4e31-bb92-be147421d3dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8gpjs" Oct 31 13:58:21.431015 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:21.430975 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhz7f\" (UniqueName: \"kubernetes.io/projected/4be917f1-8a0f-4e31-bb92-be147421d3dd-kube-api-access-xhz7f\") pod \"maas-api-revoke-sub-d3fba056-8gpjs\" (UID: \"4be917f1-8a0f-4e31-bb92-be147421d3dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8gpjs" Oct 31 13:58:21.525780 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:21.525735 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8gpjs" Oct 31 13:58:21.801042 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:21.800967 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8gpjs"] Oct 31 13:58:21.802792 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:58:21.802766 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be917f1_8a0f_4e31_bb92_be147421d3dd.slice/crio-ef357d24e8c4a295a484ec3457a71f75b67aa5a84e698b6c7e534a415ab49bf0 WatchSource:0}: Error finding container ef357d24e8c4a295a484ec3457a71f75b67aa5a84e698b6c7e534a415ab49bf0: Status 404 returned error can't find the container with id ef357d24e8c4a295a484ec3457a71f75b67aa5a84e698b6c7e534a415ab49bf0 Oct 31 13:58:22.369444 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:22.369342 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8gpjs" event={"ID":"4be917f1-8a0f-4e31-bb92-be147421d3dd","Type":"ContainerStarted","Data":"b431d3c8fa1e9ce3eb76b49b451af8436c9100dec9aad0157cb157b157c90109"} Oct 31 13:58:22.369444 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:22.369420 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8gpjs" event={"ID":"4be917f1-8a0f-4e31-bb92-be147421d3dd","Type":"ContainerStarted","Data":"ef357d24e8c4a295a484ec3457a71f75b67aa5a84e698b6c7e534a415ab49bf0"} Oct 31 13:58:22.392505 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:22.392443 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8gpjs" podStartSLOduration=1.392423983 podStartE2EDuration="1.392423983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:58:22.390726267 +0000 UTC m=+5148.873849192" watchObservedRunningTime="2026-10-31 13:58:22.392423983 +0000 UTC m=+5148.875546884" Oct 31 13:58:23.384794 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:23.384757 6841 generic.go:356] "Generic (PLEG): container finished" podID="4be917f1-8a0f-4e31-bb92-be147421d3dd" containerID="b431d3c8fa1e9ce3eb76b49b451af8436c9100dec9aad0157cb157b157c90109" exitCode=0 Oct 31 13:58:23.389625 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:23.389594 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8gpjs" event={"ID":"4be917f1-8a0f-4e31-bb92-be147421d3dd","Type":"ContainerDied","Data":"b431d3c8fa1e9ce3eb76b49b451af8436c9100dec9aad0157cb157b157c90109"} Oct 31 13:58:24.910156 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:24.910133 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8gpjs" Oct 31 13:58:25.043323 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:25.043286 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4be917f1-8a0f-4e31-bb92-be147421d3dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4be917f1-8a0f-4e31-bb92-be147421d3dd-maas-api-openshift-ca\") pod \"4be917f1-8a0f-4e31-bb92-be147421d3dd\" (UID: \"4be917f1-8a0f-4e31-bb92-be147421d3dd\") " Oct 31 13:58:25.043533 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:25.043355 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4be917f1-8a0f-4e31-bb92-be147421d3dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4be917f1-8a0f-4e31-bb92-be147421d3dd-maas-api-xks-ca\") pod \"4be917f1-8a0f-4e31-bb92-be147421d3dd\" (UID: \"4be917f1-8a0f-4e31-bb92-be147421d3dd\") " Oct 31 13:58:25.043533 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:25.043392 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4be917f1-8a0f-4e31-bb92-be147421d3dd-kube-api-access-xhz7f\" (UniqueName: \"kubernetes.io/projected/4be917f1-8a0f-4e31-bb92-be147421d3dd-kube-api-access-xhz7f\") pod \"4be917f1-8a0f-4e31-bb92-be147421d3dd\" (UID: \"4be917f1-8a0f-4e31-bb92-be147421d3dd\") " Oct 31 13:58:25.043687 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:25.043668 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4be917f1-8a0f-4e31-bb92-be147421d3dd-maas-api-openshift-ca" pod "4be917f1-8a0f-4e31-bb92-be147421d3dd" (UID: "4be917f1-8a0f-4e31-bb92-be147421d3dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:58:25.043875 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:25.043846 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4be917f1-8a0f-4e31-bb92-be147421d3dd-maas-api-openshift-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:58:25.045586 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:25.045559 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be917f1-8a0f-4e31-bb92-be147421d3dd-kube-api-access-xhz7f" pod "4be917f1-8a0f-4e31-bb92-be147421d3dd" (UID: "4be917f1-8a0f-4e31-bb92-be147421d3dd"). InnerVolumeSpecName "kube-api-access-xhz7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:58:25.045674 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:25.045564 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4be917f1-8a0f-4e31-bb92-be147421d3dd-maas-api-xks-ca" pod "4be917f1-8a0f-4e31-bb92-be147421d3dd" (UID: "4be917f1-8a0f-4e31-bb92-be147421d3dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:58:25.144567 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:25.144466 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhz7f\" (UniqueName: \"kubernetes.io/projected/4be917f1-8a0f-4e31-bb92-be147421d3dd-kube-api-access-xhz7f\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:58:25.144567 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:25.144516 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4be917f1-8a0f-4e31-bb92-be147421d3dd-maas-api-xks-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:58:25.412149 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:25.412063 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8gpjs" Oct 31 13:58:25.412149 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:25.412073 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8gpjs" event={"ID":"4be917f1-8a0f-4e31-bb92-be147421d3dd","Type":"ContainerDied","Data":"ef357d24e8c4a295a484ec3457a71f75b67aa5a84e698b6c7e534a415ab49bf0"} Oct 31 13:58:25.412149 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:25.412149 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef357d24e8c4a295a484ec3457a71f75b67aa5a84e698b6c7e534a415ab49bf0" Oct 31 13:58:41.118098 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:41.118062 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ptl2l"] Oct 31 13:58:41.118582 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:41.118499 6841 cpu_manager.go:414] "containerMap: removing container" podUID="4be917f1-8a0f-4e31-bb92-be147421d3dd" containerName="revoke-keys" Oct 31 13:58:41.118582 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:41.118513 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4be917f1-8a0f-4e31-bb92-be147421d3dd" containerName="revoke-keys" Oct 31 13:58:41.118681 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:41.118665 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="4be917f1-8a0f-4e31-bb92-be147421d3dd" containerName="revoke-keys" Oct 31 13:58:41.122820 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:41.122795 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ptl2l" Oct 31 13:58:41.133039 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:41.133010 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:58:41.133182 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:41.133040 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 13:58:41.133182 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:41.133019 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:58:41.133182 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:41.133090 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:58:41.134203 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:41.134177 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ptl2l"] Oct 31 13:58:41.275203 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:41.275164 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4d65\" (UniqueName: \"kubernetes.io/projected/1839bfc3-00a9-436e-bc8d-e4f20b0dea30-kube-api-access-s4d65\") pod \"maas-api-revoke-sub-ae594979-ptl2l\" (UID: \"1839bfc3-00a9-436e-bc8d-e4f20b0dea30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ptl2l" Oct 31 13:58:41.275203 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:41.275208 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1839bfc3-00a9-436e-bc8d-e4f20b0dea30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-ptl2l\" (UID: \"1839bfc3-00a9-436e-bc8d-e4f20b0dea30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ptl2l" Oct 31 13:58:41.275433 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:41.275240 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1839bfc3-00a9-436e-bc8d-e4f20b0dea30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-ptl2l\" (UID: \"1839bfc3-00a9-436e-bc8d-e4f20b0dea30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ptl2l" Oct 31 13:58:41.376707 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:41.376614 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1839bfc3-00a9-436e-bc8d-e4f20b0dea30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-ptl2l\" (UID: \"1839bfc3-00a9-436e-bc8d-e4f20b0dea30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ptl2l" Oct 31 13:58:41.376889 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:41.376737 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4d65\" (UniqueName: \"kubernetes.io/projected/1839bfc3-00a9-436e-bc8d-e4f20b0dea30-kube-api-access-s4d65\") pod \"maas-api-revoke-sub-ae594979-ptl2l\" (UID: \"1839bfc3-00a9-436e-bc8d-e4f20b0dea30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ptl2l" Oct 31 13:58:41.376889 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:41.376763 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1839bfc3-00a9-436e-bc8d-e4f20b0dea30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-ptl2l\" (UID: \"1839bfc3-00a9-436e-bc8d-e4f20b0dea30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ptl2l" Oct 31 13:58:41.377427 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:41.377400 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1839bfc3-00a9-436e-bc8d-e4f20b0dea30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-ptl2l\" (UID: \"1839bfc3-00a9-436e-bc8d-e4f20b0dea30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ptl2l" Oct 31 13:58:41.379274 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:41.379246 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1839bfc3-00a9-436e-bc8d-e4f20b0dea30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-ptl2l\" (UID: \"1839bfc3-00a9-436e-bc8d-e4f20b0dea30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ptl2l" Oct 31 13:58:41.389921 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:41.389895 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4d65\" (UniqueName: \"kubernetes.io/projected/1839bfc3-00a9-436e-bc8d-e4f20b0dea30-kube-api-access-s4d65\") pod \"maas-api-revoke-sub-ae594979-ptl2l\" (UID: \"1839bfc3-00a9-436e-bc8d-e4f20b0dea30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ptl2l" Oct 31 13:58:41.438425 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:41.438391 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ptl2l" Oct 31 13:58:41.675521 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:41.675440 6841 scope.go:122] "RemoveContainer" containerID="9609bf5546de3650a6c60b61bc6fef6735d4e10ae278359bbad7c82a58326da7" Oct 31 13:58:41.719150 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:41.719123 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ptl2l"] Oct 31 13:58:41.720460 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:58:41.720436 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1839bfc3_00a9_436e_bc8d_e4f20b0dea30.slice/crio-4a897263948cf5f9dd6e93fec800821ab0e141f3690bd23cf94248eb7f3966ca WatchSource:0}: Error finding container 4a897263948cf5f9dd6e93fec800821ab0e141f3690bd23cf94248eb7f3966ca: Status 404 returned error can't find the container with id 4a897263948cf5f9dd6e93fec800821ab0e141f3690bd23cf94248eb7f3966ca Oct 31 13:58:41.877522 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:41.877493 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:58:42.549296 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:42.549262 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ptl2l" event={"ID":"1839bfc3-00a9-436e-bc8d-e4f20b0dea30","Type":"ContainerStarted","Data":"6b2bf70da5f88104d6a4ef0b4560eba313afae195ba4a25666ff9738f6e43acc"} Oct 31 13:58:42.549704 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:42.549314 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ptl2l" event={"ID":"1839bfc3-00a9-436e-bc8d-e4f20b0dea30","Type":"ContainerStarted","Data":"4a897263948cf5f9dd6e93fec800821ab0e141f3690bd23cf94248eb7f3966ca"} Oct 31 13:58:42.569735 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:42.569684 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ptl2l" podStartSLOduration=1.569672148 podStartE2EDuration="1.569672148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:58:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:58:42.56851632 +0000 UTC m=+5169.051639222" watchObservedRunningTime="2026-10-31 13:58:42.569672148 +0000 UTC m=+5169.052795049" Oct 31 13:58:43.555902 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:43.555864 6841 generic.go:356] "Generic (PLEG): container finished" podID="1839bfc3-00a9-436e-bc8d-e4f20b0dea30" containerID="6b2bf70da5f88104d6a4ef0b4560eba313afae195ba4a25666ff9738f6e43acc" exitCode=0 Oct 31 13:58:43.556318 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:43.555948 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ptl2l" event={"ID":"1839bfc3-00a9-436e-bc8d-e4f20b0dea30","Type":"ContainerDied","Data":"6b2bf70da5f88104d6a4ef0b4560eba313afae195ba4a25666ff9738f6e43acc"} Oct 31 13:58:44.695499 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:44.695472 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ptl2l" Oct 31 13:58:44.813677 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:44.813591 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1839bfc3-00a9-436e-bc8d-e4f20b0dea30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1839bfc3-00a9-436e-bc8d-e4f20b0dea30-maas-api-openshift-ca\") pod \"1839bfc3-00a9-436e-bc8d-e4f20b0dea30\" (UID: \"1839bfc3-00a9-436e-bc8d-e4f20b0dea30\") " Oct 31 13:58:44.813863 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:44.813678 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1839bfc3-00a9-436e-bc8d-e4f20b0dea30-kube-api-access-s4d65\" (UniqueName: \"kubernetes.io/projected/1839bfc3-00a9-436e-bc8d-e4f20b0dea30-kube-api-access-s4d65\") pod \"1839bfc3-00a9-436e-bc8d-e4f20b0dea30\" (UID: \"1839bfc3-00a9-436e-bc8d-e4f20b0dea30\") " Oct 31 13:58:44.813863 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:44.813712 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1839bfc3-00a9-436e-bc8d-e4f20b0dea30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1839bfc3-00a9-436e-bc8d-e4f20b0dea30-maas-api-xks-ca\") pod \"1839bfc3-00a9-436e-bc8d-e4f20b0dea30\" (UID: \"1839bfc3-00a9-436e-bc8d-e4f20b0dea30\") " Oct 31 13:58:44.813974 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:44.813924 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1839bfc3-00a9-436e-bc8d-e4f20b0dea30-maas-api-openshift-ca" pod "1839bfc3-00a9-436e-bc8d-e4f20b0dea30" (UID: "1839bfc3-00a9-436e-bc8d-e4f20b0dea30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:58:44.814172 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:44.814146 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1839bfc3-00a9-436e-bc8d-e4f20b0dea30-maas-api-openshift-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:58:44.815808 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:44.815786 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1839bfc3-00a9-436e-bc8d-e4f20b0dea30-kube-api-access-s4d65" pod "1839bfc3-00a9-436e-bc8d-e4f20b0dea30" (UID: "1839bfc3-00a9-436e-bc8d-e4f20b0dea30"). InnerVolumeSpecName "kube-api-access-s4d65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:58:44.815895 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:44.815867 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1839bfc3-00a9-436e-bc8d-e4f20b0dea30-maas-api-xks-ca" pod "1839bfc3-00a9-436e-bc8d-e4f20b0dea30" (UID: "1839bfc3-00a9-436e-bc8d-e4f20b0dea30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:58:44.914600 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:44.914571 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4d65\" (UniqueName: \"kubernetes.io/projected/1839bfc3-00a9-436e-bc8d-e4f20b0dea30-kube-api-access-s4d65\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:58:44.914600 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:44.914597 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1839bfc3-00a9-436e-bc8d-e4f20b0dea30-maas-api-xks-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:58:45.539736 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:45.539703 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:58:45.567467 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:45.567434 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ptl2l" Oct 31 13:58:45.567650 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:45.567464 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ptl2l" event={"ID":"1839bfc3-00a9-436e-bc8d-e4f20b0dea30","Type":"ContainerDied","Data":"4a897263948cf5f9dd6e93fec800821ab0e141f3690bd23cf94248eb7f3966ca"} Oct 31 13:58:45.567650 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:45.567512 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a897263948cf5f9dd6e93fec800821ab0e141f3690bd23cf94248eb7f3966ca" Oct 31 13:58:45.733943 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:45.733908 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:58:50.033603 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:58:50.033558 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:58:56.882652 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:56.882083 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-6lxfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:58:56.882652 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:56.882174 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" podUID="d0ad44f3-de70-4b31-8d39-ceeed24efd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:58:57.227722 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:57.227683 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nqfhk"] Oct 31 13:58:57.228262 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:57.228241 6841 cpu_manager.go:414] "containerMap: removing container" podUID="1839bfc3-00a9-436e-bc8d-e4f20b0dea30" containerName="revoke-keys" Oct 31 13:58:57.228262 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:57.228262 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1839bfc3-00a9-436e-bc8d-e4f20b0dea30" containerName="revoke-keys" Oct 31 13:58:57.228449 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:57.228385 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="1839bfc3-00a9-436e-bc8d-e4f20b0dea30" containerName="revoke-keys" Oct 31 13:58:57.231925 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:57.231905 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nqfhk" Oct 31 13:58:57.237219 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:57.237176 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:58:57.239114 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:57.239088 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:58:57.239226 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:57.239103 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:58:57.239319 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:57.239239 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 13:58:57.243617 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:57.243587 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nqfhk"] Oct 31 13:58:57.272411 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:57.272379 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8t5x\" (UniqueName: \"kubernetes.io/projected/4e8828f0-f12f-4915-9fdb-493d04e7906f-kube-api-access-m8t5x\") pod \"maas-api-revoke-sub-a4aef7d7-nqfhk\" (UID: \"4e8828f0-f12f-4915-9fdb-493d04e7906f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nqfhk" Oct 31 13:58:57.272567 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:57.272431 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e8828f0-f12f-4915-9fdb-493d04e7906f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-nqfhk\" (UID: \"4e8828f0-f12f-4915-9fdb-493d04e7906f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nqfhk" Oct 31 13:58:57.272614 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:57.272592 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e8828f0-f12f-4915-9fdb-493d04e7906f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-nqfhk\" (UID: \"4e8828f0-f12f-4915-9fdb-493d04e7906f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nqfhk" Oct 31 13:58:57.373416 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:57.373379 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e8828f0-f12f-4915-9fdb-493d04e7906f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-nqfhk\" (UID: \"4e8828f0-f12f-4915-9fdb-493d04e7906f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nqfhk" Oct 31 13:58:57.373622 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:57.373482 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8t5x\" (UniqueName: \"kubernetes.io/projected/4e8828f0-f12f-4915-9fdb-493d04e7906f-kube-api-access-m8t5x\") pod \"maas-api-revoke-sub-a4aef7d7-nqfhk\" (UID: \"4e8828f0-f12f-4915-9fdb-493d04e7906f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nqfhk" Oct 31 13:58:57.373622 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:57.373533 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e8828f0-f12f-4915-9fdb-493d04e7906f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-nqfhk\" (UID: \"4e8828f0-f12f-4915-9fdb-493d04e7906f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nqfhk" Oct 31 13:58:57.374194 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:57.374167 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e8828f0-f12f-4915-9fdb-493d04e7906f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-nqfhk\" (UID: \"4e8828f0-f12f-4915-9fdb-493d04e7906f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nqfhk" Oct 31 13:58:57.375845 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:57.375820 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e8828f0-f12f-4915-9fdb-493d04e7906f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-nqfhk\" (UID: \"4e8828f0-f12f-4915-9fdb-493d04e7906f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nqfhk" Oct 31 13:58:57.385924 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:57.385901 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8t5x\" (UniqueName: \"kubernetes.io/projected/4e8828f0-f12f-4915-9fdb-493d04e7906f-kube-api-access-m8t5x\") pod \"maas-api-revoke-sub-a4aef7d7-nqfhk\" (UID: \"4e8828f0-f12f-4915-9fdb-493d04e7906f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nqfhk" Oct 31 13:58:57.548337 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:57.548260 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nqfhk" Oct 31 13:58:57.825659 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:57.825634 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nqfhk"] Oct 31 13:58:57.827149 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:58:57.827122 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e8828f0_f12f_4915_9fdb_493d04e7906f.slice/crio-8b0d05a262882dc0d02228c46ef04e3ed5b017f2a07cc87a0d87bb0507c062e4 WatchSource:0}: Error finding container 8b0d05a262882dc0d02228c46ef04e3ed5b017f2a07cc87a0d87bb0507c062e4: Status 404 returned error can't find the container with id 8b0d05a262882dc0d02228c46ef04e3ed5b017f2a07cc87a0d87bb0507c062e4 Oct 31 13:58:58.673811 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:58.673782 6841 generic.go:356] "Generic (PLEG): container finished" podID="4e8828f0-f12f-4915-9fdb-493d04e7906f" containerID="92a194f0c8a847989dd2183b8e1bf03bdf2796292aa7cba9a3c6e8e6584f8e52" exitCode=0 Oct 31 13:58:58.674166 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:58.673834 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nqfhk" event={"ID":"4e8828f0-f12f-4915-9fdb-493d04e7906f","Type":"ContainerDied","Data":"92a194f0c8a847989dd2183b8e1bf03bdf2796292aa7cba9a3c6e8e6584f8e52"} Oct 31 13:58:58.674166 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:58.673876 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nqfhk" event={"ID":"4e8828f0-f12f-4915-9fdb-493d04e7906f","Type":"ContainerStarted","Data":"8b0d05a262882dc0d02228c46ef04e3ed5b017f2a07cc87a0d87bb0507c062e4"} Oct 31 13:58:59.818396 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:59.818369 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nqfhk" Oct 31 13:58:59.897912 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:59.897878 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e8828f0-f12f-4915-9fdb-493d04e7906f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e8828f0-f12f-4915-9fdb-493d04e7906f-maas-api-xks-ca\") pod \"4e8828f0-f12f-4915-9fdb-493d04e7906f\" (UID: \"4e8828f0-f12f-4915-9fdb-493d04e7906f\") " Oct 31 13:58:59.898086 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:59.898010 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e8828f0-f12f-4915-9fdb-493d04e7906f-kube-api-access-m8t5x\" (UniqueName: \"kubernetes.io/projected/4e8828f0-f12f-4915-9fdb-493d04e7906f-kube-api-access-m8t5x\") pod \"4e8828f0-f12f-4915-9fdb-493d04e7906f\" (UID: \"4e8828f0-f12f-4915-9fdb-493d04e7906f\") " Oct 31 13:58:59.898086 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:59.898061 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e8828f0-f12f-4915-9fdb-493d04e7906f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e8828f0-f12f-4915-9fdb-493d04e7906f-maas-api-openshift-ca\") pod \"4e8828f0-f12f-4915-9fdb-493d04e7906f\" (UID: \"4e8828f0-f12f-4915-9fdb-493d04e7906f\") " Oct 31 13:58:59.898475 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:59.898438 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e8828f0-f12f-4915-9fdb-493d04e7906f-maas-api-openshift-ca" pod "4e8828f0-f12f-4915-9fdb-493d04e7906f" (UID: "4e8828f0-f12f-4915-9fdb-493d04e7906f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:58:59.899963 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:59.899937 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e8828f0-f12f-4915-9fdb-493d04e7906f-maas-api-xks-ca" pod "4e8828f0-f12f-4915-9fdb-493d04e7906f" (UID: "4e8828f0-f12f-4915-9fdb-493d04e7906f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:58:59.900323 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:59.900303 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e8828f0-f12f-4915-9fdb-493d04e7906f-kube-api-access-m8t5x" pod "4e8828f0-f12f-4915-9fdb-493d04e7906f" (UID: "4e8828f0-f12f-4915-9fdb-493d04e7906f"). InnerVolumeSpecName "kube-api-access-m8t5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:58:59.999353 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:59.999326 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e8828f0-f12f-4915-9fdb-493d04e7906f-maas-api-xks-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:58:59.999353 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:59.999352 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8t5x\" (UniqueName: \"kubernetes.io/projected/4e8828f0-f12f-4915-9fdb-493d04e7906f-kube-api-access-m8t5x\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:58:59.999568 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:58:59.999362 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e8828f0-f12f-4915-9fdb-493d04e7906f-maas-api-openshift-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:00.687288 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:00.687251 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nqfhk" event={"ID":"4e8828f0-f12f-4915-9fdb-493d04e7906f","Type":"ContainerDied","Data":"8b0d05a262882dc0d02228c46ef04e3ed5b017f2a07cc87a0d87bb0507c062e4"} Oct 31 13:59:00.687288 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:00.687270 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nqfhk" Oct 31 13:59:00.687503 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:00.687305 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b0d05a262882dc0d02228c46ef04e3ed5b017f2a07cc87a0d87bb0507c062e4" Oct 31 13:59:03.023538 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:03.023497 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xfsql"] Oct 31 13:59:03.025661 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:03.025631 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xfsql"] Oct 31 13:59:03.642107 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:03.642070 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b759a18-e9b9-444a-84fa-d38a5201cdd7" path="/var/lib/kubelet/pods/0b759a18-e9b9-444a-84fa-d38a5201cdd7/volumes" Oct 31 13:59:12.915592 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:12.915552 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s7729"] Oct 31 13:59:12.917788 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:12.917761 6841 cpu_manager.go:414] "containerMap: removing container" podUID="4e8828f0-f12f-4915-9fdb-493d04e7906f" containerName="revoke-keys" Oct 31 13:59:12.917960 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:12.917943 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e8828f0-f12f-4915-9fdb-493d04e7906f" containerName="revoke-keys" Oct 31 13:59:12.918220 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:12.918202 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e8828f0-f12f-4915-9fdb-493d04e7906f" containerName="revoke-keys" Oct 31 13:59:12.922546 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:12.922524 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s7729" Oct 31 13:59:12.948005 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:12.944259 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s7729"] Oct 31 13:59:12.960025 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:12.959975 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 13:59:12.960316 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:12.960298 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:59:12.960545 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:12.960530 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:59:12.968003 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:12.964065 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:59:13.075802 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:13.075752 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d9pv\" (UniqueName: \"kubernetes.io/projected/6fe43ae2-c997-405d-b6a9-5b3d3d649c04-kube-api-access-6d9pv\") pod \"maas-api-revoke-sub-79805ca9-s7729\" (UID: \"6fe43ae2-c997-405d-b6a9-5b3d3d649c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s7729" Oct 31 13:59:13.076210 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:13.076172 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fe43ae2-c997-405d-b6a9-5b3d3d649c04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-s7729\" (UID: \"6fe43ae2-c997-405d-b6a9-5b3d3d649c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s7729" Oct 31 13:59:13.076539 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:13.076503 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fe43ae2-c997-405d-b6a9-5b3d3d649c04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-s7729\" (UID: \"6fe43ae2-c997-405d-b6a9-5b3d3d649c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s7729" Oct 31 13:59:13.227009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:13.217568 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d9pv\" (UniqueName: \"kubernetes.io/projected/6fe43ae2-c997-405d-b6a9-5b3d3d649c04-kube-api-access-6d9pv\") pod \"maas-api-revoke-sub-79805ca9-s7729\" (UID: \"6fe43ae2-c997-405d-b6a9-5b3d3d649c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s7729" Oct 31 13:59:13.227009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:13.222264 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fe43ae2-c997-405d-b6a9-5b3d3d649c04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-s7729\" (UID: \"6fe43ae2-c997-405d-b6a9-5b3d3d649c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s7729" Oct 31 13:59:13.227009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:13.222417 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fe43ae2-c997-405d-b6a9-5b3d3d649c04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-s7729\" (UID: \"6fe43ae2-c997-405d-b6a9-5b3d3d649c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s7729" Oct 31 13:59:13.227009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:13.225335 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fe43ae2-c997-405d-b6a9-5b3d3d649c04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-s7729\" (UID: \"6fe43ae2-c997-405d-b6a9-5b3d3d649c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s7729" Oct 31 13:59:13.227009 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:13.226832 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fe43ae2-c997-405d-b6a9-5b3d3d649c04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-s7729\" (UID: \"6fe43ae2-c997-405d-b6a9-5b3d3d649c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s7729" Oct 31 13:59:13.232184 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:13.232119 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d9pv\" (UniqueName: \"kubernetes.io/projected/6fe43ae2-c997-405d-b6a9-5b3d3d649c04-kube-api-access-6d9pv\") pod \"maas-api-revoke-sub-79805ca9-s7729\" (UID: \"6fe43ae2-c997-405d-b6a9-5b3d3d649c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s7729" Oct 31 13:59:13.411721 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:13.411685 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s7729" Oct 31 13:59:13.686370 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:13.686346 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s7729"] Oct 31 13:59:13.687397 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:59:13.687362 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe43ae2_c997_405d_b6a9_5b3d3d649c04.slice/crio-61fe22e8f34e14a2ab1b589034d4e05eb42726d99b2691d516c54ace55f56d2f WatchSource:0}: Error finding container 61fe22e8f34e14a2ab1b589034d4e05eb42726d99b2691d516c54ace55f56d2f: Status 404 returned error can't find the container with id 61fe22e8f34e14a2ab1b589034d4e05eb42726d99b2691d516c54ace55f56d2f Oct 31 13:59:13.866229 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:13.866199 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s7729" event={"ID":"6fe43ae2-c997-405d-b6a9-5b3d3d649c04","Type":"ContainerStarted","Data":"61fe22e8f34e14a2ab1b589034d4e05eb42726d99b2691d516c54ace55f56d2f"} Oct 31 13:59:14.873174 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:14.873089 6841 generic.go:356] "Generic (PLEG): container finished" podID="6fe43ae2-c997-405d-b6a9-5b3d3d649c04" containerID="0aba66c18ed3f368f02855cff5037c57cfd09a19b012040c788d545586562a59" exitCode=0 Oct 31 13:59:14.873557 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:14.873182 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s7729" event={"ID":"6fe43ae2-c997-405d-b6a9-5b3d3d649c04","Type":"ContainerDied","Data":"0aba66c18ed3f368f02855cff5037c57cfd09a19b012040c788d545586562a59"} Oct 31 13:59:16.015396 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:16.015372 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s7729" Oct 31 13:59:16.052704 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:16.052677 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fe43ae2-c997-405d-b6a9-5b3d3d649c04-kube-api-access-6d9pv\" (UniqueName: \"kubernetes.io/projected/6fe43ae2-c997-405d-b6a9-5b3d3d649c04-kube-api-access-6d9pv\") pod \"6fe43ae2-c997-405d-b6a9-5b3d3d649c04\" (UID: \"6fe43ae2-c997-405d-b6a9-5b3d3d649c04\") " Oct 31 13:59:16.052868 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:16.052772 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fe43ae2-c997-405d-b6a9-5b3d3d649c04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fe43ae2-c997-405d-b6a9-5b3d3d649c04-maas-api-xks-ca\") pod \"6fe43ae2-c997-405d-b6a9-5b3d3d649c04\" (UID: \"6fe43ae2-c997-405d-b6a9-5b3d3d649c04\") " Oct 31 13:59:16.052868 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:16.052802 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fe43ae2-c997-405d-b6a9-5b3d3d649c04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fe43ae2-c997-405d-b6a9-5b3d3d649c04-maas-api-openshift-ca\") pod \"6fe43ae2-c997-405d-b6a9-5b3d3d649c04\" (UID: \"6fe43ae2-c997-405d-b6a9-5b3d3d649c04\") " Oct 31 13:59:16.053319 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:16.053289 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fe43ae2-c997-405d-b6a9-5b3d3d649c04-maas-api-openshift-ca" pod "6fe43ae2-c997-405d-b6a9-5b3d3d649c04" (UID: "6fe43ae2-c997-405d-b6a9-5b3d3d649c04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:59:16.054868 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:16.054840 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fe43ae2-c997-405d-b6a9-5b3d3d649c04-maas-api-xks-ca" pod "6fe43ae2-c997-405d-b6a9-5b3d3d649c04" (UID: "6fe43ae2-c997-405d-b6a9-5b3d3d649c04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:59:16.054968 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:16.054947 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fe43ae2-c997-405d-b6a9-5b3d3d649c04-kube-api-access-6d9pv" pod "6fe43ae2-c997-405d-b6a9-5b3d3d649c04" (UID: "6fe43ae2-c997-405d-b6a9-5b3d3d649c04"). InnerVolumeSpecName "kube-api-access-6d9pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:59:16.153908 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:16.153828 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d9pv\" (UniqueName: \"kubernetes.io/projected/6fe43ae2-c997-405d-b6a9-5b3d3d649c04-kube-api-access-6d9pv\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:16.153908 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:16.153857 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fe43ae2-c997-405d-b6a9-5b3d3d649c04-maas-api-xks-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:16.153908 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:16.153868 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fe43ae2-c997-405d-b6a9-5b3d3d649c04-maas-api-openshift-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:16.886012 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:16.885960 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s7729" Oct 31 13:59:16.886188 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:16.886010 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s7729" event={"ID":"6fe43ae2-c997-405d-b6a9-5b3d3d649c04","Type":"ContainerDied","Data":"61fe22e8f34e14a2ab1b589034d4e05eb42726d99b2691d516c54ace55f56d2f"} Oct 31 13:59:16.886188 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:16.886057 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61fe22e8f34e14a2ab1b589034d4e05eb42726d99b2691d516c54ace55f56d2f" Oct 31 13:59:19.023762 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:19.023728 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tn6ng"] Oct 31 13:59:19.026074 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:19.026047 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tn6ng"] Oct 31 13:59:19.641701 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:19.641668 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="803b53b4-4ae4-493c-ae16-9cac9f6025c1" path="/var/lib/kubelet/pods/803b53b4-4ae4-493c-ae16-9cac9f6025c1/volumes" Oct 31 13:59:20.033722 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:59:20.033687 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:59:27.580345 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:27.580292 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zrfkx"] Oct 31 13:59:27.581620 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:27.581590 6841 cpu_manager.go:414] "containerMap: removing container" podUID="6fe43ae2-c997-405d-b6a9-5b3d3d649c04" containerName="revoke-keys" Oct 31 13:59:27.581800 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:27.581784 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fe43ae2-c997-405d-b6a9-5b3d3d649c04" containerName="revoke-keys" Oct 31 13:59:27.582092 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:27.582075 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fe43ae2-c997-405d-b6a9-5b3d3d649c04" containerName="revoke-keys" Oct 31 13:59:27.605230 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:27.605202 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zrfkx"] Oct 31 13:59:27.605596 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:27.605528 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zrfkx" Oct 31 13:59:27.611365 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:27.611341 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:59:27.616632 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:27.616611 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:59:27.616907 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:27.616882 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 13:59:27.617124 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:27.617108 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:59:27.727957 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:27.712195 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf72d6c7-7c81-4376-8d7a-1722174becc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-zrfkx\" (UID: \"bf72d6c7-7c81-4376-8d7a-1722174becc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zrfkx" Oct 31 13:59:27.727957 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:27.727046 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf72d6c7-7c81-4376-8d7a-1722174becc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-zrfkx\" (UID: \"bf72d6c7-7c81-4376-8d7a-1722174becc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zrfkx" Oct 31 13:59:27.727957 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:27.727150 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgnzc\" (UniqueName: \"kubernetes.io/projected/bf72d6c7-7c81-4376-8d7a-1722174becc6-kube-api-access-sgnzc\") pod \"maas-api-revoke-sub-0b7e8edb-zrfkx\" (UID: \"bf72d6c7-7c81-4376-8d7a-1722174becc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zrfkx" Oct 31 13:59:27.838065 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:27.834483 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf72d6c7-7c81-4376-8d7a-1722174becc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-zrfkx\" (UID: \"bf72d6c7-7c81-4376-8d7a-1722174becc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zrfkx" Oct 31 13:59:27.838065 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:27.834593 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf72d6c7-7c81-4376-8d7a-1722174becc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-zrfkx\" (UID: \"bf72d6c7-7c81-4376-8d7a-1722174becc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zrfkx" Oct 31 13:59:27.838065 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:27.834657 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgnzc\" (UniqueName: \"kubernetes.io/projected/bf72d6c7-7c81-4376-8d7a-1722174becc6-kube-api-access-sgnzc\") pod \"maas-api-revoke-sub-0b7e8edb-zrfkx\" (UID: \"bf72d6c7-7c81-4376-8d7a-1722174becc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zrfkx" Oct 31 13:59:27.838065 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:27.835918 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf72d6c7-7c81-4376-8d7a-1722174becc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-zrfkx\" (UID: \"bf72d6c7-7c81-4376-8d7a-1722174becc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zrfkx" Oct 31 13:59:27.840023 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:27.839707 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf72d6c7-7c81-4376-8d7a-1722174becc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-zrfkx\" (UID: \"bf72d6c7-7c81-4376-8d7a-1722174becc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zrfkx" Oct 31 13:59:27.874899 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:27.874854 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgnzc\" (UniqueName: \"kubernetes.io/projected/bf72d6c7-7c81-4376-8d7a-1722174becc6-kube-api-access-sgnzc\") pod \"maas-api-revoke-sub-0b7e8edb-zrfkx\" (UID: \"bf72d6c7-7c81-4376-8d7a-1722174becc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zrfkx" Oct 31 13:59:27.936840 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:27.936813 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zrfkx" Oct 31 13:59:28.713156 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:28.713095 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zrfkx"] Oct 31 13:59:28.758813 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:59:28.758756 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf72d6c7_7c81_4376_8d7a_1722174becc6.slice/crio-3b5e434874dc24cc8c28e50aa2f92d5749fbb80d477e34d51c8758c3dbf12ae0 WatchSource:0}: Error finding container 3b5e434874dc24cc8c28e50aa2f92d5749fbb80d477e34d51c8758c3dbf12ae0: Status 404 returned error can't find the container with id 3b5e434874dc24cc8c28e50aa2f92d5749fbb80d477e34d51c8758c3dbf12ae0 Oct 31 13:59:29.087274 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:29.087242 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zrfkx" event={"ID":"bf72d6c7-7c81-4376-8d7a-1722174becc6","Type":"ContainerStarted","Data":"6b26b1a253975f075cb931b0c662d40af509e0bcb5b46306ee91d0296901c7cb"} Oct 31 13:59:29.087384 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:29.087297 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zrfkx" event={"ID":"bf72d6c7-7c81-4376-8d7a-1722174becc6","Type":"ContainerStarted","Data":"3b5e434874dc24cc8c28e50aa2f92d5749fbb80d477e34d51c8758c3dbf12ae0"} Oct 31 13:59:29.107191 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:29.107148 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zrfkx" podStartSLOduration=2.107134551 podStartE2EDuration="2.107134551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:59:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:59:29.1047618 +0000 UTC m=+5215.587884704" watchObservedRunningTime="2026-10-31 13:59:29.107134551 +0000 UTC m=+5215.590257476" Oct 31 13:59:30.094298 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:30.094266 6841 generic.go:356] "Generic (PLEG): container finished" podID="bf72d6c7-7c81-4376-8d7a-1722174becc6" containerID="6b26b1a253975f075cb931b0c662d40af509e0bcb5b46306ee91d0296901c7cb" exitCode=0 Oct 31 13:59:30.094695 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:30.094356 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zrfkx" event={"ID":"bf72d6c7-7c81-4376-8d7a-1722174becc6","Type":"ContainerDied","Data":"6b26b1a253975f075cb931b0c662d40af509e0bcb5b46306ee91d0296901c7cb"} Oct 31 13:59:31.240507 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:31.240483 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zrfkx" Oct 31 13:59:31.275201 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:31.275168 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf72d6c7-7c81-4376-8d7a-1722174becc6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf72d6c7-7c81-4376-8d7a-1722174becc6-maas-api-xks-ca\") pod \"bf72d6c7-7c81-4376-8d7a-1722174becc6\" (UID: \"bf72d6c7-7c81-4376-8d7a-1722174becc6\") " Oct 31 13:59:31.275410 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:31.275220 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf72d6c7-7c81-4376-8d7a-1722174becc6-kube-api-access-sgnzc\" (UniqueName: \"kubernetes.io/projected/bf72d6c7-7c81-4376-8d7a-1722174becc6-kube-api-access-sgnzc\") pod \"bf72d6c7-7c81-4376-8d7a-1722174becc6\" (UID: \"bf72d6c7-7c81-4376-8d7a-1722174becc6\") " Oct 31 13:59:31.275410 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:31.275249 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf72d6c7-7c81-4376-8d7a-1722174becc6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf72d6c7-7c81-4376-8d7a-1722174becc6-maas-api-openshift-ca\") pod \"bf72d6c7-7c81-4376-8d7a-1722174becc6\" (UID: \"bf72d6c7-7c81-4376-8d7a-1722174becc6\") " Oct 31 13:59:31.276040 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:31.276010 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf72d6c7-7c81-4376-8d7a-1722174becc6-maas-api-openshift-ca" pod "bf72d6c7-7c81-4376-8d7a-1722174becc6" (UID: "bf72d6c7-7c81-4376-8d7a-1722174becc6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:59:31.277805 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:31.277765 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf72d6c7-7c81-4376-8d7a-1722174becc6-kube-api-access-sgnzc" pod "bf72d6c7-7c81-4376-8d7a-1722174becc6" (UID: "bf72d6c7-7c81-4376-8d7a-1722174becc6"). InnerVolumeSpecName "kube-api-access-sgnzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:59:31.281604 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:31.281544 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf72d6c7-7c81-4376-8d7a-1722174becc6-maas-api-xks-ca" pod "bf72d6c7-7c81-4376-8d7a-1722174becc6" (UID: "bf72d6c7-7c81-4376-8d7a-1722174becc6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:59:31.376916 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:31.376822 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf72d6c7-7c81-4376-8d7a-1722174becc6-maas-api-xks-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:31.376916 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:31.376862 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgnzc\" (UniqueName: \"kubernetes.io/projected/bf72d6c7-7c81-4376-8d7a-1722174becc6-kube-api-access-sgnzc\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:31.376916 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:31.376873 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf72d6c7-7c81-4376-8d7a-1722174becc6-maas-api-openshift-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:32.107688 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:32.107654 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zrfkx" Oct 31 13:59:32.107864 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:32.107689 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zrfkx" event={"ID":"bf72d6c7-7c81-4376-8d7a-1722174becc6","Type":"ContainerDied","Data":"3b5e434874dc24cc8c28e50aa2f92d5749fbb80d477e34d51c8758c3dbf12ae0"} Oct 31 13:59:32.107864 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:32.107745 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b5e434874dc24cc8c28e50aa2f92d5749fbb80d477e34d51c8758c3dbf12ae0" Oct 31 13:59:36.032506 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:36.032462 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-953a6bca-kmqwr"] Oct 31 13:59:36.035242 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:36.035215 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-953a6bca-kmqwr"] Oct 31 13:59:37.642753 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:37.642716 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14c1666f-e6d5-4f5e-bf05-43253e9239eb" path="/var/lib/kubelet/pods/14c1666f-e6d5-4f5e-bf05-43253e9239eb/volumes" Oct 31 13:59:38.234676 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:38.234642 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzvpt"] Oct 31 13:59:38.235107 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:38.235087 6841 cpu_manager.go:414] "containerMap: removing container" podUID="bf72d6c7-7c81-4376-8d7a-1722174becc6" containerName="revoke-keys" Oct 31 13:59:38.235173 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:38.235108 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf72d6c7-7c81-4376-8d7a-1722174becc6" containerName="revoke-keys" Oct 31 13:59:38.235231 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:38.235214 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf72d6c7-7c81-4376-8d7a-1722174becc6" containerName="revoke-keys" Oct 31 13:59:38.238548 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:38.238533 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzvpt" Oct 31 13:59:38.244997 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:38.244954 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:59:38.245203 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:38.244968 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:59:38.245203 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:38.245177 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:59:38.245891 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:38.245559 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 13:59:38.247077 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:38.247047 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzvpt"] Oct 31 13:59:38.343958 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:38.343922 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ea46daf-7c90-4ac4-aa65-29290f7d6fff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qzvpt\" (UID: \"9ea46daf-7c90-4ac4-aa65-29290f7d6fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzvpt" Oct 31 13:59:38.344176 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:38.344091 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ea46daf-7c90-4ac4-aa65-29290f7d6fff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qzvpt\" (UID: \"9ea46daf-7c90-4ac4-aa65-29290f7d6fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzvpt" Oct 31 13:59:38.344255 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:38.344169 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqnmj\" (UniqueName: \"kubernetes.io/projected/9ea46daf-7c90-4ac4-aa65-29290f7d6fff-kube-api-access-hqnmj\") pod \"maas-api-revoke-sub-3d4156f1-qzvpt\" (UID: \"9ea46daf-7c90-4ac4-aa65-29290f7d6fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzvpt" Oct 31 13:59:38.444872 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:38.444833 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ea46daf-7c90-4ac4-aa65-29290f7d6fff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qzvpt\" (UID: \"9ea46daf-7c90-4ac4-aa65-29290f7d6fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzvpt" Oct 31 13:59:38.445057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:38.444935 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ea46daf-7c90-4ac4-aa65-29290f7d6fff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qzvpt\" (UID: \"9ea46daf-7c90-4ac4-aa65-29290f7d6fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzvpt" Oct 31 13:59:38.445057 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:38.444970 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqnmj\" (UniqueName: \"kubernetes.io/projected/9ea46daf-7c90-4ac4-aa65-29290f7d6fff-kube-api-access-hqnmj\") pod \"maas-api-revoke-sub-3d4156f1-qzvpt\" (UID: \"9ea46daf-7c90-4ac4-aa65-29290f7d6fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzvpt" Oct 31 13:59:38.445551 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:38.445520 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ea46daf-7c90-4ac4-aa65-29290f7d6fff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qzvpt\" (UID: \"9ea46daf-7c90-4ac4-aa65-29290f7d6fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzvpt" Oct 31 13:59:38.447304 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:38.447281 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ea46daf-7c90-4ac4-aa65-29290f7d6fff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qzvpt\" (UID: \"9ea46daf-7c90-4ac4-aa65-29290f7d6fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzvpt" Oct 31 13:59:38.457692 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:38.457662 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqnmj\" (UniqueName: \"kubernetes.io/projected/9ea46daf-7c90-4ac4-aa65-29290f7d6fff-kube-api-access-hqnmj\") pod \"maas-api-revoke-sub-3d4156f1-qzvpt\" (UID: \"9ea46daf-7c90-4ac4-aa65-29290f7d6fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzvpt" Oct 31 13:59:38.552726 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:38.552641 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzvpt" Oct 31 13:59:38.829774 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:38.829745 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzvpt"] Oct 31 13:59:38.831530 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:59:38.831497 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ea46daf_7c90_4ac4_aa65_29290f7d6fff.slice/crio-f4b34a4bc48adfecd7ae890bd3bba6659f81b19e6ed66be4d46f88054e631bd3 WatchSource:0}: Error finding container f4b34a4bc48adfecd7ae890bd3bba6659f81b19e6ed66be4d46f88054e631bd3: Status 404 returned error can't find the container with id f4b34a4bc48adfecd7ae890bd3bba6659f81b19e6ed66be4d46f88054e631bd3 Oct 31 13:59:39.149514 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:39.149477 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzvpt" event={"ID":"9ea46daf-7c90-4ac4-aa65-29290f7d6fff","Type":"ContainerStarted","Data":"f4b34a4bc48adfecd7ae890bd3bba6659f81b19e6ed66be4d46f88054e631bd3"} Oct 31 13:59:40.156773 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:40.156740 6841 generic.go:356] "Generic (PLEG): container finished" podID="9ea46daf-7c90-4ac4-aa65-29290f7d6fff" containerID="be2ddfd4caaef1a1996c2e9345377312c0fbe3037237134429178ae005cf7c18" exitCode=0 Oct 31 13:59:40.157188 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:40.156831 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzvpt" event={"ID":"9ea46daf-7c90-4ac4-aa65-29290f7d6fff","Type":"ContainerDied","Data":"be2ddfd4caaef1a1996c2e9345377312c0fbe3037237134429178ae005cf7c18"} Oct 31 13:59:41.302443 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:41.302418 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzvpt" Oct 31 13:59:41.373252 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:41.373202 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ea46daf-7c90-4ac4-aa65-29290f7d6fff-kube-api-access-hqnmj\" (UniqueName: \"kubernetes.io/projected/9ea46daf-7c90-4ac4-aa65-29290f7d6fff-kube-api-access-hqnmj\") pod \"9ea46daf-7c90-4ac4-aa65-29290f7d6fff\" (UID: \"9ea46daf-7c90-4ac4-aa65-29290f7d6fff\") " Oct 31 13:59:41.373427 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:41.373281 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ea46daf-7c90-4ac4-aa65-29290f7d6fff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ea46daf-7c90-4ac4-aa65-29290f7d6fff-maas-api-openshift-ca\") pod \"9ea46daf-7c90-4ac4-aa65-29290f7d6fff\" (UID: \"9ea46daf-7c90-4ac4-aa65-29290f7d6fff\") " Oct 31 13:59:41.373427 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:41.373305 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ea46daf-7c90-4ac4-aa65-29290f7d6fff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ea46daf-7c90-4ac4-aa65-29290f7d6fff-maas-api-xks-ca\") pod \"9ea46daf-7c90-4ac4-aa65-29290f7d6fff\" (UID: \"9ea46daf-7c90-4ac4-aa65-29290f7d6fff\") " Oct 31 13:59:41.373663 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:41.373639 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ea46daf-7c90-4ac4-aa65-29290f7d6fff-maas-api-openshift-ca" pod "9ea46daf-7c90-4ac4-aa65-29290f7d6fff" (UID: "9ea46daf-7c90-4ac4-aa65-29290f7d6fff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:59:41.373851 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:41.373827 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ea46daf-7c90-4ac4-aa65-29290f7d6fff-maas-api-openshift-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:41.375423 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:41.375401 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ea46daf-7c90-4ac4-aa65-29290f7d6fff-maas-api-xks-ca" pod "9ea46daf-7c90-4ac4-aa65-29290f7d6fff" (UID: "9ea46daf-7c90-4ac4-aa65-29290f7d6fff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:59:41.375488 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:41.375467 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ea46daf-7c90-4ac4-aa65-29290f7d6fff-kube-api-access-hqnmj" pod "9ea46daf-7c90-4ac4-aa65-29290f7d6fff" (UID: "9ea46daf-7c90-4ac4-aa65-29290f7d6fff"). InnerVolumeSpecName "kube-api-access-hqnmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:59:41.475137 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:41.475091 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqnmj\" (UniqueName: \"kubernetes.io/projected/9ea46daf-7c90-4ac4-aa65-29290f7d6fff-kube-api-access-hqnmj\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:41.475137 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:41.475132 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ea46daf-7c90-4ac4-aa65-29290f7d6fff-maas-api-xks-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:42.018734 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:42.018697 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 13:59:42.169788 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:42.169757 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzvpt" Oct 31 13:59:42.169968 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:42.169789 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzvpt" event={"ID":"9ea46daf-7c90-4ac4-aa65-29290f7d6fff","Type":"ContainerDied","Data":"f4b34a4bc48adfecd7ae890bd3bba6659f81b19e6ed66be4d46f88054e631bd3"} Oct 31 13:59:42.169968 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:42.169840 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4b34a4bc48adfecd7ae890bd3bba6659f81b19e6ed66be4d46f88054e631bd3" Oct 31 13:59:50.046194 ip-10-0-138-70 kubenswrapper[6841]: E1031 13:59:50.046076 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 13:59:55.249114 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:55.249081 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzvpt"] Oct 31 13:59:55.254888 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:55.254857 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzvpt"] Oct 31 13:59:55.268779 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:55.268751 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lvb47"] Oct 31 13:59:55.269363 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:55.269342 6841 cpu_manager.go:414] "containerMap: removing container" podUID="9ea46daf-7c90-4ac4-aa65-29290f7d6fff" containerName="revoke-keys" Oct 31 13:59:55.269363 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:55.269361 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ea46daf-7c90-4ac4-aa65-29290f7d6fff" containerName="revoke-keys" Oct 31 13:59:55.269538 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:55.269447 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ea46daf-7c90-4ac4-aa65-29290f7d6fff" containerName="revoke-keys" Oct 31 13:59:55.272907 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:55.272884 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lvb47" Oct 31 13:59:55.276948 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:55.276917 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:59:55.278087 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:55.278062 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 13:59:55.278198 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:55.278123 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:59:55.278198 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:55.278123 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:59:55.286330 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:55.286304 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lvb47"] Oct 31 13:59:55.425884 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:55.425830 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjc5k\" (UniqueName: \"kubernetes.io/projected/fab2e407-04ed-4d98-be34-0ac71d1f2b55-kube-api-access-vjc5k\") pod \"maas-api-revoke-sub-3d4156f1-lvb47\" (UID: \"fab2e407-04ed-4d98-be34-0ac71d1f2b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lvb47" Oct 31 13:59:55.426116 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:55.425962 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fab2e407-04ed-4d98-be34-0ac71d1f2b55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lvb47\" (UID: \"fab2e407-04ed-4d98-be34-0ac71d1f2b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lvb47" Oct 31 13:59:55.426116 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:55.426041 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fab2e407-04ed-4d98-be34-0ac71d1f2b55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lvb47\" (UID: \"fab2e407-04ed-4d98-be34-0ac71d1f2b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lvb47" Oct 31 13:59:55.527488 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:55.527391 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fab2e407-04ed-4d98-be34-0ac71d1f2b55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lvb47\" (UID: \"fab2e407-04ed-4d98-be34-0ac71d1f2b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lvb47" Oct 31 13:59:55.527488 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:55.527449 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fab2e407-04ed-4d98-be34-0ac71d1f2b55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lvb47\" (UID: \"fab2e407-04ed-4d98-be34-0ac71d1f2b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lvb47" Oct 31 13:59:55.527702 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:55.527541 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjc5k\" (UniqueName: \"kubernetes.io/projected/fab2e407-04ed-4d98-be34-0ac71d1f2b55-kube-api-access-vjc5k\") pod \"maas-api-revoke-sub-3d4156f1-lvb47\" (UID: \"fab2e407-04ed-4d98-be34-0ac71d1f2b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lvb47" Oct 31 13:59:55.528104 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:55.528080 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fab2e407-04ed-4d98-be34-0ac71d1f2b55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lvb47\" (UID: \"fab2e407-04ed-4d98-be34-0ac71d1f2b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lvb47" Oct 31 13:59:55.529867 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:55.529831 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fab2e407-04ed-4d98-be34-0ac71d1f2b55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lvb47\" (UID: \"fab2e407-04ed-4d98-be34-0ac71d1f2b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lvb47" Oct 31 13:59:55.540046 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:55.540023 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjc5k\" (UniqueName: \"kubernetes.io/projected/fab2e407-04ed-4d98-be34-0ac71d1f2b55-kube-api-access-vjc5k\") pod \"maas-api-revoke-sub-3d4156f1-lvb47\" (UID: \"fab2e407-04ed-4d98-be34-0ac71d1f2b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lvb47" Oct 31 13:59:55.588662 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:55.588636 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lvb47" Oct 31 13:59:55.646954 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:55.646920 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ea46daf-7c90-4ac4-aa65-29290f7d6fff" path="/var/lib/kubelet/pods/9ea46daf-7c90-4ac4-aa65-29290f7d6fff/volumes" Oct 31 13:59:55.863707 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:55.863674 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lvb47"] Oct 31 13:59:55.865168 ip-10-0-138-70 kubenswrapper[6841]: W1031 13:59:55.865127 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab2e407_04ed_4d98_be34_0ac71d1f2b55.slice/crio-1e9b1ff2479fcd906e95a91949f0cf6fa4b9853f87af2199e9a9ddf7d4330b1b WatchSource:0}: Error finding container 1e9b1ff2479fcd906e95a91949f0cf6fa4b9853f87af2199e9a9ddf7d4330b1b: Status 404 returned error can't find the container with id 1e9b1ff2479fcd906e95a91949f0cf6fa4b9853f87af2199e9a9ddf7d4330b1b Oct 31 13:59:56.306717 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:56.306684 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lvb47" event={"ID":"fab2e407-04ed-4d98-be34-0ac71d1f2b55","Type":"ContainerStarted","Data":"03e32cbb51078ae50731fc1f55136335eaf839f6b69467d97f74380cb86b08b2"} Oct 31 13:59:56.307129 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:56.306739 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lvb47" event={"ID":"fab2e407-04ed-4d98-be34-0ac71d1f2b55","Type":"ContainerStarted","Data":"1e9b1ff2479fcd906e95a91949f0cf6fa4b9853f87af2199e9a9ddf7d4330b1b"} Oct 31 13:59:56.325571 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:56.325517 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lvb47" podStartSLOduration=1.325500318 podStartE2EDuration="1.325500318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:59:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:59:56.325016619 +0000 UTC m=+5242.808139517" watchObservedRunningTime="2026-10-31 13:59:56.325500318 +0000 UTC m=+5242.808623220" Oct 31 13:59:57.314427 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:57.314390 6841 generic.go:356] "Generic (PLEG): container finished" podID="fab2e407-04ed-4d98-be34-0ac71d1f2b55" containerID="03e32cbb51078ae50731fc1f55136335eaf839f6b69467d97f74380cb86b08b2" exitCode=0 Oct 31 13:59:57.314849 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:57.314472 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lvb47" event={"ID":"fab2e407-04ed-4d98-be34-0ac71d1f2b55","Type":"ContainerDied","Data":"03e32cbb51078ae50731fc1f55136335eaf839f6b69467d97f74380cb86b08b2"} Oct 31 13:59:58.460686 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:58.460662 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lvb47" Oct 31 13:59:58.559685 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:58.559653 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fab2e407-04ed-4d98-be34-0ac71d1f2b55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fab2e407-04ed-4d98-be34-0ac71d1f2b55-maas-api-xks-ca\") pod \"fab2e407-04ed-4d98-be34-0ac71d1f2b55\" (UID: \"fab2e407-04ed-4d98-be34-0ac71d1f2b55\") " Oct 31 13:59:58.559866 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:58.559715 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fab2e407-04ed-4d98-be34-0ac71d1f2b55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fab2e407-04ed-4d98-be34-0ac71d1f2b55-maas-api-openshift-ca\") pod \"fab2e407-04ed-4d98-be34-0ac71d1f2b55\" (UID: \"fab2e407-04ed-4d98-be34-0ac71d1f2b55\") " Oct 31 13:59:58.559866 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:58.559836 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fab2e407-04ed-4d98-be34-0ac71d1f2b55-kube-api-access-vjc5k\" (UniqueName: \"kubernetes.io/projected/fab2e407-04ed-4d98-be34-0ac71d1f2b55-kube-api-access-vjc5k\") pod \"fab2e407-04ed-4d98-be34-0ac71d1f2b55\" (UID: \"fab2e407-04ed-4d98-be34-0ac71d1f2b55\") " Oct 31 13:59:58.560142 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:58.560115 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fab2e407-04ed-4d98-be34-0ac71d1f2b55-maas-api-openshift-ca" pod "fab2e407-04ed-4d98-be34-0ac71d1f2b55" (UID: "fab2e407-04ed-4d98-be34-0ac71d1f2b55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:59:58.561951 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:58.561931 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fab2e407-04ed-4d98-be34-0ac71d1f2b55-maas-api-xks-ca" pod "fab2e407-04ed-4d98-be34-0ac71d1f2b55" (UID: "fab2e407-04ed-4d98-be34-0ac71d1f2b55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:59:58.562067 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:58.562045 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab2e407-04ed-4d98-be34-0ac71d1f2b55-kube-api-access-vjc5k" pod "fab2e407-04ed-4d98-be34-0ac71d1f2b55" (UID: "fab2e407-04ed-4d98-be34-0ac71d1f2b55"). InnerVolumeSpecName "kube-api-access-vjc5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:59:58.661063 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:58.660951 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjc5k\" (UniqueName: \"kubernetes.io/projected/fab2e407-04ed-4d98-be34-0ac71d1f2b55-kube-api-access-vjc5k\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:58.661063 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:58.661015 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fab2e407-04ed-4d98-be34-0ac71d1f2b55-maas-api-xks-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:58.661063 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:58.661028 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fab2e407-04ed-4d98-be34-0ac71d1f2b55-maas-api-openshift-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:59.326518 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:59.326488 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lvb47" Oct 31 13:59:59.326697 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:59.326516 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lvb47" event={"ID":"fab2e407-04ed-4d98-be34-0ac71d1f2b55","Type":"ContainerDied","Data":"1e9b1ff2479fcd906e95a91949f0cf6fa4b9853f87af2199e9a9ddf7d4330b1b"} Oct 31 13:59:59.326697 ip-10-0-138-70 kubenswrapper[6841]: I1031 13:59:59.326563 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e9b1ff2479fcd906e95a91949f0cf6fa4b9853f87af2199e9a9ddf7d4330b1b" Oct 31 14:00:00.153297 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:00.153263 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890920-nd56l"] Oct 31 14:00:00.153875 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:00.153849 6841 cpu_manager.go:414] "containerMap: removing container" podUID="fab2e407-04ed-4d98-be34-0ac71d1f2b55" containerName="revoke-keys" Oct 31 14:00:00.153875 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:00.153873 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fab2e407-04ed-4d98-be34-0ac71d1f2b55" containerName="revoke-keys" Oct 31 14:00:00.154038 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:00.153990 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="fab2e407-04ed-4d98-be34-0ac71d1f2b55" containerName="revoke-keys" Oct 31 14:00:00.157480 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:00.157463 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-nd56l" Oct 31 14:00:00.161741 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:00.161715 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 14:00:00.161741 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:00.161715 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 14:00:00.163054 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:00.163031 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 14:00:00.170329 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:00.170300 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890920-nd56l"] Oct 31 14:00:00.278137 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:00.278101 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16b2424b-0e5e-48a8-8614-70742dc5fa28-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890920-nd56l\" (UID: \"16b2424b-0e5e-48a8-8614-70742dc5fa28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-nd56l" Oct 31 14:00:00.278322 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:00.278155 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lbjs\" (UniqueName: \"kubernetes.io/projected/16b2424b-0e5e-48a8-8614-70742dc5fa28-kube-api-access-8lbjs\") pod \"maas-api-key-cleanup-29890920-nd56l\" (UID: \"16b2424b-0e5e-48a8-8614-70742dc5fa28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-nd56l" Oct 31 14:00:00.379272 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:00.379239 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16b2424b-0e5e-48a8-8614-70742dc5fa28-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890920-nd56l\" (UID: \"16b2424b-0e5e-48a8-8614-70742dc5fa28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-nd56l" Oct 31 14:00:00.379436 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:00.379293 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lbjs\" (UniqueName: \"kubernetes.io/projected/16b2424b-0e5e-48a8-8614-70742dc5fa28-kube-api-access-8lbjs\") pod \"maas-api-key-cleanup-29890920-nd56l\" (UID: \"16b2424b-0e5e-48a8-8614-70742dc5fa28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-nd56l" Oct 31 14:00:00.379936 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:00.379917 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16b2424b-0e5e-48a8-8614-70742dc5fa28-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890920-nd56l\" (UID: \"16b2424b-0e5e-48a8-8614-70742dc5fa28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-nd56l" Oct 31 14:00:00.392024 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:00.391999 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lbjs\" (UniqueName: \"kubernetes.io/projected/16b2424b-0e5e-48a8-8614-70742dc5fa28-kube-api-access-8lbjs\") pod \"maas-api-key-cleanup-29890920-nd56l\" (UID: \"16b2424b-0e5e-48a8-8614-70742dc5fa28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-nd56l" Oct 31 14:00:00.476631 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:00.476599 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-nd56l" Oct 31 14:00:00.755071 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:00.755040 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890920-nd56l"] Oct 31 14:00:00.755753 ip-10-0-138-70 kubenswrapper[6841]: W1031 14:00:00.755727 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b2424b_0e5e_48a8_8614_70742dc5fa28.slice/crio-c5572ae38c9e77bf1573766747335c006f8d1e199686eb2d31d543f46a6a5d6a WatchSource:0}: Error finding container c5572ae38c9e77bf1573766747335c006f8d1e199686eb2d31d543f46a6a5d6a: Status 404 returned error can't find the container with id c5572ae38c9e77bf1573766747335c006f8d1e199686eb2d31d543f46a6a5d6a Oct 31 14:00:01.340937 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:01.340848 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-nd56l" event={"ID":"16b2424b-0e5e-48a8-8614-70742dc5fa28","Type":"ContainerStarted","Data":"040bc01753adc0573d613ac28769faf04bd7f9ef7971ca2d03411e73c494c2b6"} Oct 31 14:00:01.341354 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:01.340939 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-nd56l" event={"ID":"16b2424b-0e5e-48a8-8614-70742dc5fa28","Type":"ContainerStarted","Data":"c5572ae38c9e77bf1573766747335c006f8d1e199686eb2d31d543f46a6a5d6a"} Oct 31 14:00:01.361076 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:01.361027 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-nd56l" podStartSLOduration=1.361013663 podStartE2EDuration="1.361013663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:00:01.358562052 +0000 UTC m=+5247.841684954" watchObservedRunningTime="2026-10-31 14:00:01.361013663 +0000 UTC m=+5247.844136564" Oct 31 14:00:02.349499 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:02.349463 6841 generic.go:356] "Generic (PLEG): container finished" podID="16b2424b-0e5e-48a8-8614-70742dc5fa28" containerID="040bc01753adc0573d613ac28769faf04bd7f9ef7971ca2d03411e73c494c2b6" exitCode=60 Oct 31 14:00:02.349939 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:02.349547 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-nd56l" event={"ID":"16b2424b-0e5e-48a8-8614-70742dc5fa28","Type":"ContainerDied","Data":"040bc01753adc0573d613ac28769faf04bd7f9ef7971ca2d03411e73c494c2b6"} Oct 31 14:00:02.349939 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:02.349755 6841 scope.go:122] "RemoveContainer" containerID="040bc01753adc0573d613ac28769faf04bd7f9ef7971ca2d03411e73c494c2b6" Oct 31 14:00:03.358382 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:03.358346 6841 generic.go:356] "Generic (PLEG): container finished" podID="16b2424b-0e5e-48a8-8614-70742dc5fa28" containerID="f39f3f05b1b884824cff7da33bf1d72a1184336f43ff5b05b05f8ecc309d198a" exitCode=60 Oct 31 14:00:03.358804 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:03.358430 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-nd56l" event={"ID":"16b2424b-0e5e-48a8-8614-70742dc5fa28","Type":"ContainerDied","Data":"f39f3f05b1b884824cff7da33bf1d72a1184336f43ff5b05b05f8ecc309d198a"} Oct 31 14:00:03.358804 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:03.358486 6841 scope.go:122] "RemoveContainer" containerID="040bc01753adc0573d613ac28769faf04bd7f9ef7971ca2d03411e73c494c2b6" Oct 31 14:00:03.358804 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:03.358745 6841 scope.go:122] "RemoveContainer" containerID="f39f3f05b1b884824cff7da33bf1d72a1184336f43ff5b05b05f8ecc309d198a" Oct 31 14:00:03.359049 ip-10-0-138-70 kubenswrapper[6841]: E1031 14:00:03.359028 6841 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890920-nd56l_odh-ai-gateway-infra(16b2424b-0e5e-48a8-8614-70742dc5fa28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-nd56l" podUID="16b2424b-0e5e-48a8-8614-70742dc5fa28" Oct 31 14:00:04.367145 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:04.367116 6841 scope.go:122] "RemoveContainer" containerID="f39f3f05b1b884824cff7da33bf1d72a1184336f43ff5b05b05f8ecc309d198a" Oct 31 14:00:04.367666 ip-10-0-138-70 kubenswrapper[6841]: E1031 14:00:04.367326 6841 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890920-nd56l_odh-ai-gateway-infra(16b2424b-0e5e-48a8-8614-70742dc5fa28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-nd56l" podUID="16b2424b-0e5e-48a8-8614-70742dc5fa28" Oct 31 14:00:15.637007 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:15.636251 6841 scope.go:122] "RemoveContainer" containerID="f39f3f05b1b884824cff7da33bf1d72a1184336f43ff5b05b05f8ecc309d198a" Oct 31 14:00:16.484006 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:16.482202 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-nd56l" event={"ID":"16b2424b-0e5e-48a8-8614-70742dc5fa28","Type":"ContainerStarted","Data":"a1aa9cd4612fb4e3e610989d921ced5247b61aeb8dec44248a4baa9f490bffc3"} Oct 31 14:00:17.490630 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:17.490595 6841 generic.go:356] "Generic (PLEG): container finished" podID="16b2424b-0e5e-48a8-8614-70742dc5fa28" containerID="a1aa9cd4612fb4e3e610989d921ced5247b61aeb8dec44248a4baa9f490bffc3" exitCode=60 Oct 31 14:00:17.491084 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:17.490662 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-nd56l" event={"ID":"16b2424b-0e5e-48a8-8614-70742dc5fa28","Type":"ContainerDied","Data":"a1aa9cd4612fb4e3e610989d921ced5247b61aeb8dec44248a4baa9f490bffc3"} Oct 31 14:00:17.491084 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:17.490727 6841 scope.go:122] "RemoveContainer" containerID="f39f3f05b1b884824cff7da33bf1d72a1184336f43ff5b05b05f8ecc309d198a" Oct 31 14:00:17.491214 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:17.491094 6841 scope.go:122] "RemoveContainer" containerID="a1aa9cd4612fb4e3e610989d921ced5247b61aeb8dec44248a4baa9f490bffc3" Oct 31 14:00:17.491409 ip-10-0-138-70 kubenswrapper[6841]: E1031 14:00:17.491386 6841 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890920-nd56l_odh-ai-gateway-infra(16b2424b-0e5e-48a8-8614-70742dc5fa28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-nd56l" podUID="16b2424b-0e5e-48a8-8614-70742dc5fa28" Oct 31 14:00:17.520324 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:17.520291 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890920-nd56l"] Oct 31 14:00:17.661092 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:17.661054 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kk87w"] Oct 31 14:00:17.666660 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:17.666636 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kk87w" Oct 31 14:00:17.670951 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:17.670925 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:00:17.673189 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:17.673159 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kk87w"] Oct 31 14:00:17.804850 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:17.804762 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4ab4f8e-4112-4065-839c-b4603efe3e74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-kk87w\" (UID: \"f4ab4f8e-4112-4065-839c-b4603efe3e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kk87w" Oct 31 14:00:17.805057 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:17.804872 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4ab4f8e-4112-4065-839c-b4603efe3e74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-kk87w\" (UID: \"f4ab4f8e-4112-4065-839c-b4603efe3e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kk87w" Oct 31 14:00:17.805057 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:17.805036 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b7r6\" (UniqueName: \"kubernetes.io/projected/f4ab4f8e-4112-4065-839c-b4603efe3e74-kube-api-access-2b7r6\") pod \"maas-api-revoke-sub-e2fc9c5b-kk87w\" (UID: \"f4ab4f8e-4112-4065-839c-b4603efe3e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kk87w" Oct 31 14:00:17.905796 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:17.905761 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b7r6\" (UniqueName: \"kubernetes.io/projected/f4ab4f8e-4112-4065-839c-b4603efe3e74-kube-api-access-2b7r6\") pod \"maas-api-revoke-sub-e2fc9c5b-kk87w\" (UID: \"f4ab4f8e-4112-4065-839c-b4603efe3e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kk87w" Oct 31 14:00:17.905796 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:17.905808 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4ab4f8e-4112-4065-839c-b4603efe3e74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-kk87w\" (UID: \"f4ab4f8e-4112-4065-839c-b4603efe3e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kk87w" Oct 31 14:00:17.906127 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:17.905857 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4ab4f8e-4112-4065-839c-b4603efe3e74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-kk87w\" (UID: \"f4ab4f8e-4112-4065-839c-b4603efe3e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kk87w" Oct 31 14:00:17.906552 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:17.906526 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4ab4f8e-4112-4065-839c-b4603efe3e74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-kk87w\" (UID: \"f4ab4f8e-4112-4065-839c-b4603efe3e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kk87w" Oct 31 14:00:17.908347 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:17.908318 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4ab4f8e-4112-4065-839c-b4603efe3e74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-kk87w\" (UID: \"f4ab4f8e-4112-4065-839c-b4603efe3e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kk87w" Oct 31 14:00:17.918139 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:17.918110 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b7r6\" (UniqueName: \"kubernetes.io/projected/f4ab4f8e-4112-4065-839c-b4603efe3e74-kube-api-access-2b7r6\") pod \"maas-api-revoke-sub-e2fc9c5b-kk87w\" (UID: \"f4ab4f8e-4112-4065-839c-b4603efe3e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kk87w" Oct 31 14:00:17.996207 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:17.996180 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kk87w" Oct 31 14:00:18.273631 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:18.273605 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kk87w"] Oct 31 14:00:18.275436 ip-10-0-138-70 kubenswrapper[6841]: W1031 14:00:18.275408 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4ab4f8e_4112_4065_839c_b4603efe3e74.slice/crio-201c40961b2f229f11e623d0b3b548ae669073a67d28c248ade9645f3d52ca77 WatchSource:0}: Error finding container 201c40961b2f229f11e623d0b3b548ae669073a67d28c248ade9645f3d52ca77: Status 404 returned error can't find the container with id 201c40961b2f229f11e623d0b3b548ae669073a67d28c248ade9645f3d52ca77 Oct 31 14:00:18.500056 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:18.500022 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kk87w" event={"ID":"f4ab4f8e-4112-4065-839c-b4603efe3e74","Type":"ContainerStarted","Data":"201c40961b2f229f11e623d0b3b548ae669073a67d28c248ade9645f3d52ca77"} Oct 31 14:00:18.660774 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:18.660745 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-nd56l" Oct 31 14:00:18.715142 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:18.715105 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16b2424b-0e5e-48a8-8614-70742dc5fa28-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16b2424b-0e5e-48a8-8614-70742dc5fa28-maas-api-service-ca\") pod \"16b2424b-0e5e-48a8-8614-70742dc5fa28\" (UID: \"16b2424b-0e5e-48a8-8614-70742dc5fa28\") " Oct 31 14:00:18.715621 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:18.715180 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16b2424b-0e5e-48a8-8614-70742dc5fa28-kube-api-access-8lbjs\" (UniqueName: \"kubernetes.io/projected/16b2424b-0e5e-48a8-8614-70742dc5fa28-kube-api-access-8lbjs\") pod \"16b2424b-0e5e-48a8-8614-70742dc5fa28\" (UID: \"16b2424b-0e5e-48a8-8614-70742dc5fa28\") " Oct 31 14:00:18.715621 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:18.715481 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16b2424b-0e5e-48a8-8614-70742dc5fa28-maas-api-service-ca" pod "16b2424b-0e5e-48a8-8614-70742dc5fa28" (UID: "16b2424b-0e5e-48a8-8614-70742dc5fa28"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:00:18.718038 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:18.718012 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b2424b-0e5e-48a8-8614-70742dc5fa28-kube-api-access-8lbjs" pod "16b2424b-0e5e-48a8-8614-70742dc5fa28" (UID: "16b2424b-0e5e-48a8-8614-70742dc5fa28"). InnerVolumeSpecName "kube-api-access-8lbjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:00:18.816794 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:18.816695 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16b2424b-0e5e-48a8-8614-70742dc5fa28-maas-api-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:18.816794 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:18.816738 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lbjs\" (UniqueName: \"kubernetes.io/projected/16b2424b-0e5e-48a8-8614-70742dc5fa28-kube-api-access-8lbjs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:19.508320 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:19.508291 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-nd56l" Oct 31 14:00:19.508793 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:19.508286 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-nd56l" event={"ID":"16b2424b-0e5e-48a8-8614-70742dc5fa28","Type":"ContainerDied","Data":"c5572ae38c9e77bf1573766747335c006f8d1e199686eb2d31d543f46a6a5d6a"} Oct 31 14:00:19.508793 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:19.508422 6841 scope.go:122] "RemoveContainer" containerID="a1aa9cd4612fb4e3e610989d921ced5247b61aeb8dec44248a4baa9f490bffc3" Oct 31 14:00:19.510115 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:19.510094 6841 generic.go:356] "Generic (PLEG): container finished" podID="f4ab4f8e-4112-4065-839c-b4603efe3e74" containerID="c663a524daed45855cd802ef4639852efe95ea7c26fac03081c8f26a57f291c1" exitCode=0 Oct 31 14:00:19.510197 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:19.510131 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kk87w" event={"ID":"f4ab4f8e-4112-4065-839c-b4603efe3e74","Type":"ContainerDied","Data":"c663a524daed45855cd802ef4639852efe95ea7c26fac03081c8f26a57f291c1"} Oct 31 14:00:19.545825 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:19.545795 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890920-nd56l"] Oct 31 14:00:19.550370 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:19.550345 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890920-nd56l"] Oct 31 14:00:19.640961 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:19.640933 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16b2424b-0e5e-48a8-8614-70742dc5fa28" path="/var/lib/kubelet/pods/16b2424b-0e5e-48a8-8614-70742dc5fa28/volumes" Oct 31 14:00:20.033626 ip-10-0-138-70 kubenswrapper[6841]: E1031 14:00:20.033592 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 14:00:20.654640 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:20.654612 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kk87w" Oct 31 14:00:20.736901 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:20.736861 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4ab4f8e-4112-4065-839c-b4603efe3e74-kube-api-access-2b7r6\" (UniqueName: \"kubernetes.io/projected/f4ab4f8e-4112-4065-839c-b4603efe3e74-kube-api-access-2b7r6\") pod \"f4ab4f8e-4112-4065-839c-b4603efe3e74\" (UID: \"f4ab4f8e-4112-4065-839c-b4603efe3e74\") " Oct 31 14:00:20.737097 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:20.737020 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4ab4f8e-4112-4065-839c-b4603efe3e74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4ab4f8e-4112-4065-839c-b4603efe3e74-maas-api-xks-ca\") pod \"f4ab4f8e-4112-4065-839c-b4603efe3e74\" (UID: \"f4ab4f8e-4112-4065-839c-b4603efe3e74\") " Oct 31 14:00:20.737097 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:20.737059 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4ab4f8e-4112-4065-839c-b4603efe3e74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4ab4f8e-4112-4065-839c-b4603efe3e74-maas-api-openshift-ca\") pod \"f4ab4f8e-4112-4065-839c-b4603efe3e74\" (UID: \"f4ab4f8e-4112-4065-839c-b4603efe3e74\") " Oct 31 14:00:20.737464 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:20.737437 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4ab4f8e-4112-4065-839c-b4603efe3e74-maas-api-openshift-ca" pod "f4ab4f8e-4112-4065-839c-b4603efe3e74" (UID: "f4ab4f8e-4112-4065-839c-b4603efe3e74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:00:20.739024 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:20.739002 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4ab4f8e-4112-4065-839c-b4603efe3e74-kube-api-access-2b7r6" pod "f4ab4f8e-4112-4065-839c-b4603efe3e74" (UID: "f4ab4f8e-4112-4065-839c-b4603efe3e74"). InnerVolumeSpecName "kube-api-access-2b7r6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:00:20.739153 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:20.739035 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4ab4f8e-4112-4065-839c-b4603efe3e74-maas-api-xks-ca" pod "f4ab4f8e-4112-4065-839c-b4603efe3e74" (UID: "f4ab4f8e-4112-4065-839c-b4603efe3e74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:00:20.838610 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:20.838532 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4ab4f8e-4112-4065-839c-b4603efe3e74-maas-api-xks-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:20.838610 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:20.838563 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4ab4f8e-4112-4065-839c-b4603efe3e74-maas-api-openshift-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:20.838610 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:20.838573 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b7r6\" (UniqueName: \"kubernetes.io/projected/f4ab4f8e-4112-4065-839c-b4603efe3e74-kube-api-access-2b7r6\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:21.524819 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:21.524783 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kk87w" event={"ID":"f4ab4f8e-4112-4065-839c-b4603efe3e74","Type":"ContainerDied","Data":"201c40961b2f229f11e623d0b3b548ae669073a67d28c248ade9645f3d52ca77"} Oct 31 14:00:21.524819 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:21.524807 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kk87w" Oct 31 14:00:21.525166 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:21.524837 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="201c40961b2f229f11e623d0b3b548ae669073a67d28c248ade9645f3d52ca77" Oct 31 14:00:27.882035 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:27.882000 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-n449f"] Oct 31 14:00:27.882456 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:27.882445 6841 cpu_manager.go:414] "containerMap: removing container" podUID="16b2424b-0e5e-48a8-8614-70742dc5fa28" containerName="cleanup" Oct 31 14:00:27.882518 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:27.882458 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16b2424b-0e5e-48a8-8614-70742dc5fa28" containerName="cleanup" Oct 31 14:00:27.882518 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:27.882466 6841 cpu_manager.go:414] "containerMap: removing container" podUID="f4ab4f8e-4112-4065-839c-b4603efe3e74" containerName="revoke-keys" Oct 31 14:00:27.882518 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:27.882471 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4ab4f8e-4112-4065-839c-b4603efe3e74" containerName="revoke-keys" Oct 31 14:00:27.882518 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:27.882492 6841 cpu_manager.go:414] "containerMap: removing container" podUID="16b2424b-0e5e-48a8-8614-70742dc5fa28" containerName="cleanup" Oct 31 14:00:27.882518 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:27.882497 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16b2424b-0e5e-48a8-8614-70742dc5fa28" containerName="cleanup" Oct 31 14:00:27.882702 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:27.882585 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="16b2424b-0e5e-48a8-8614-70742dc5fa28" containerName="cleanup" Oct 31 14:00:27.882702 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:27.882597 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="16b2424b-0e5e-48a8-8614-70742dc5fa28" containerName="cleanup" Oct 31 14:00:27.882702 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:27.882602 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4ab4f8e-4112-4065-839c-b4603efe3e74" containerName="revoke-keys" Oct 31 14:00:27.886035 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:27.886017 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-n449f" Oct 31 14:00:27.891793 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:27.891764 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 14:00:27.891938 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:27.891830 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 14:00:27.893050 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:27.893026 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 14:00:27.894101 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:27.894079 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:00:27.896958 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:27.896931 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-n449f"] Oct 31 14:00:28.008269 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:28.008236 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr6zx\" (UniqueName: \"kubernetes.io/projected/4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da-kube-api-access-xr6zx\") pod \"maas-api-revoke-sub-5a29efb3-n449f\" (UID: \"4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-n449f" Oct 31 14:00:28.008466 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:28.008300 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-n449f\" (UID: \"4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-n449f" Oct 31 14:00:28.008466 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:28.008423 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-n449f\" (UID: \"4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-n449f" Oct 31 14:00:28.109257 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:28.109222 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr6zx\" (UniqueName: \"kubernetes.io/projected/4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da-kube-api-access-xr6zx\") pod \"maas-api-revoke-sub-5a29efb3-n449f\" (UID: \"4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-n449f" Oct 31 14:00:28.109426 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:28.109287 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-n449f\" (UID: \"4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-n449f" Oct 31 14:00:28.109426 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:28.109338 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-n449f\" (UID: \"4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-n449f" Oct 31 14:00:28.110024 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:28.109965 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-n449f\" (UID: \"4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-n449f" Oct 31 14:00:28.111695 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:28.111674 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-n449f\" (UID: \"4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-n449f" Oct 31 14:00:28.121524 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:28.121501 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr6zx\" (UniqueName: \"kubernetes.io/projected/4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da-kube-api-access-xr6zx\") pod \"maas-api-revoke-sub-5a29efb3-n449f\" (UID: \"4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-n449f" Oct 31 14:00:28.201679 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:28.201654 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-n449f" Oct 31 14:00:28.479433 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:28.479402 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-n449f"] Oct 31 14:00:28.481256 ip-10-0-138-70 kubenswrapper[6841]: W1031 14:00:28.481227 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a8a2bfc_2a51_4bda_a1c2_06a5c49f06da.slice/crio-16aad3129474a6797683a1600a84d3c97ec50a4c844220bafafb3d001a4364a2 WatchSource:0}: Error finding container 16aad3129474a6797683a1600a84d3c97ec50a4c844220bafafb3d001a4364a2: Status 404 returned error can't find the container with id 16aad3129474a6797683a1600a84d3c97ec50a4c844220bafafb3d001a4364a2 Oct 31 14:00:28.569320 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:28.569282 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-n449f" event={"ID":"4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da","Type":"ContainerStarted","Data":"16aad3129474a6797683a1600a84d3c97ec50a4c844220bafafb3d001a4364a2"} Oct 31 14:00:29.581861 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:29.581821 6841 generic.go:356] "Generic (PLEG): container finished" podID="4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da" containerID="98da05c867a4d9291a37ed924092ec28e5381ade6cc8f0e7f9241ef42539a1ab" exitCode=0 Oct 31 14:00:29.582429 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:29.581927 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-n449f" event={"ID":"4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da","Type":"ContainerDied","Data":"98da05c867a4d9291a37ed924092ec28e5381ade6cc8f0e7f9241ef42539a1ab"} Oct 31 14:00:31.341530 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:31.341181 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-n449f" Oct 31 14:00:31.436387 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:31.435541 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da-maas-api-openshift-ca\") pod \"4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da\" (UID: \"4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da\") " Oct 31 14:00:31.436387 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:31.435610 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da-kube-api-access-xr6zx\" (UniqueName: \"kubernetes.io/projected/4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da-kube-api-access-xr6zx\") pod \"4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da\" (UID: \"4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da\") " Oct 31 14:00:31.436387 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:31.435640 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da-maas-api-xks-ca\") pod \"4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da\" (UID: \"4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da\") " Oct 31 14:00:31.437313 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:31.437288 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da-maas-api-openshift-ca" pod "4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da" (UID: "4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:00:31.440502 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:31.440462 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da-maas-api-xks-ca" pod "4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da" (UID: "4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:00:31.456566 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:31.456519 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da-kube-api-access-xr6zx" pod "4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da" (UID: "4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da"). InnerVolumeSpecName "kube-api-access-xr6zx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:00:31.537846 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:31.537692 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da-maas-api-openshift-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:31.537846 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:31.537739 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr6zx\" (UniqueName: \"kubernetes.io/projected/4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da-kube-api-access-xr6zx\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:31.537846 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:31.537755 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da-maas-api-xks-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:31.615754 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:31.615714 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-n449f" event={"ID":"4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da","Type":"ContainerDied","Data":"16aad3129474a6797683a1600a84d3c97ec50a4c844220bafafb3d001a4364a2"} Oct 31 14:00:31.616162 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:31.616083 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16aad3129474a6797683a1600a84d3c97ec50a4c844220bafafb3d001a4364a2" Oct 31 14:00:31.616391 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:31.616346 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-n449f" Oct 31 14:00:41.709820 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:41.709789 6841 scope.go:122] "RemoveContainer" containerID="437b7ef0f2a363f2bb98cb8365fe2bcca7a98a6e08060dbe4655587943d443a0" Oct 31 14:00:41.718904 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:41.718884 6841 scope.go:122] "RemoveContainer" containerID="b1c70bd947e2501e01686606a7eb777eeab61629176eeb9adf4fab3d78c1cf2a" Oct 31 14:00:41.726958 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:41.726939 6841 scope.go:122] "RemoveContainer" containerID="d14d35a1c9c4023a5cfd3f4ec5e2839c71b448f4efc723a2e887f756662baae9" Oct 31 14:00:41.734755 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:41.734735 6841 scope.go:122] "RemoveContainer" containerID="1e4a188ecd9d1c4a76c35134e1e00ae1699701d8bad8eb95ffb67d465ef0226c" Oct 31 14:00:41.987912 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:41.987830 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:00:43.670440 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:43.670400 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lvb47"] Oct 31 14:00:43.677102 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:43.677071 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lvb47"] Oct 31 14:00:43.695305 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:43.695272 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd8nh"] Oct 31 14:00:43.695730 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:43.695717 6841 cpu_manager.go:414] "containerMap: removing container" podUID="4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da" containerName="revoke-keys" Oct 31 14:00:43.695777 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:43.695731 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da" containerName="revoke-keys" Oct 31 14:00:43.695777 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:43.695745 6841 cpu_manager.go:414] "containerMap: removing container" podUID="16b2424b-0e5e-48a8-8614-70742dc5fa28" containerName="cleanup" Oct 31 14:00:43.695777 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:43.695751 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16b2424b-0e5e-48a8-8614-70742dc5fa28" containerName="cleanup" Oct 31 14:00:43.695881 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:43.695818 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da" containerName="revoke-keys" Oct 31 14:00:43.695881 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:43.695830 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="16b2424b-0e5e-48a8-8614-70742dc5fa28" containerName="cleanup" Oct 31 14:00:43.700473 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:43.700456 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd8nh" Oct 31 14:00:43.705512 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:43.705485 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 14:00:43.705687 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:43.705557 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:00:43.705687 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:43.705594 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 14:00:43.705798 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:43.705691 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 14:00:43.708092 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:43.708067 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd8nh"] Oct 31 14:00:43.756562 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:43.756529 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q9fp\" (UniqueName: \"kubernetes.io/projected/8cfa8aff-e9c5-464f-abe3-a5da3a967233-kube-api-access-8q9fp\") pod \"maas-api-revoke-sub-3d4156f1-fd8nh\" (UID: \"8cfa8aff-e9c5-464f-abe3-a5da3a967233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd8nh" Oct 31 14:00:43.756731 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:43.756576 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cfa8aff-e9c5-464f-abe3-a5da3a967233-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fd8nh\" (UID: \"8cfa8aff-e9c5-464f-abe3-a5da3a967233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd8nh" Oct 31 14:00:43.756794 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:43.756738 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cfa8aff-e9c5-464f-abe3-a5da3a967233-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fd8nh\" (UID: \"8cfa8aff-e9c5-464f-abe3-a5da3a967233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd8nh" Oct 31 14:00:43.858151 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:43.858112 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cfa8aff-e9c5-464f-abe3-a5da3a967233-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fd8nh\" (UID: \"8cfa8aff-e9c5-464f-abe3-a5da3a967233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd8nh" Oct 31 14:00:43.858354 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:43.858197 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q9fp\" (UniqueName: \"kubernetes.io/projected/8cfa8aff-e9c5-464f-abe3-a5da3a967233-kube-api-access-8q9fp\") pod \"maas-api-revoke-sub-3d4156f1-fd8nh\" (UID: \"8cfa8aff-e9c5-464f-abe3-a5da3a967233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd8nh" Oct 31 14:00:43.858354 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:43.858224 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cfa8aff-e9c5-464f-abe3-a5da3a967233-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fd8nh\" (UID: \"8cfa8aff-e9c5-464f-abe3-a5da3a967233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd8nh" Oct 31 14:00:43.858806 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:43.858783 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cfa8aff-e9c5-464f-abe3-a5da3a967233-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fd8nh\" (UID: \"8cfa8aff-e9c5-464f-abe3-a5da3a967233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd8nh" Oct 31 14:00:43.860600 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:43.860581 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cfa8aff-e9c5-464f-abe3-a5da3a967233-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fd8nh\" (UID: \"8cfa8aff-e9c5-464f-abe3-a5da3a967233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd8nh" Oct 31 14:00:43.870445 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:43.870413 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q9fp\" (UniqueName: \"kubernetes.io/projected/8cfa8aff-e9c5-464f-abe3-a5da3a967233-kube-api-access-8q9fp\") pod \"maas-api-revoke-sub-3d4156f1-fd8nh\" (UID: \"8cfa8aff-e9c5-464f-abe3-a5da3a967233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd8nh" Oct 31 14:00:44.020860 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:44.020828 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd8nh" Oct 31 14:00:44.296835 ip-10-0-138-70 kubenswrapper[6841]: W1031 14:00:44.296805 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cfa8aff_e9c5_464f_abe3_a5da3a967233.slice/crio-2ea17bcb079fd0ed22526a7050cc59016437846085912fe97d0a488294dba510 WatchSource:0}: Error finding container 2ea17bcb079fd0ed22526a7050cc59016437846085912fe97d0a488294dba510: Status 404 returned error can't find the container with id 2ea17bcb079fd0ed22526a7050cc59016437846085912fe97d0a488294dba510 Oct 31 14:00:44.297333 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:44.297297 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd8nh"] Oct 31 14:00:44.692210 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:44.692173 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd8nh" event={"ID":"8cfa8aff-e9c5-464f-abe3-a5da3a967233","Type":"ContainerStarted","Data":"96f8ca8dcd990a49bfeea1375467aa8a924938239e52b86729f2724f8c3bf1fb"} Oct 31 14:00:44.692618 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:44.692227 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd8nh" event={"ID":"8cfa8aff-e9c5-464f-abe3-a5da3a967233","Type":"ContainerStarted","Data":"2ea17bcb079fd0ed22526a7050cc59016437846085912fe97d0a488294dba510"} Oct 31 14:00:44.711745 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:44.711696 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd8nh" podStartSLOduration=1.711684263 podStartE2EDuration="1.711684263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:00:44.709839845 +0000 UTC m=+5291.192962748" watchObservedRunningTime="2026-10-31 14:00:44.711684263 +0000 UTC m=+5291.194807188" Oct 31 14:00:45.641824 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:45.641777 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fab2e407-04ed-4d98-be34-0ac71d1f2b55" path="/var/lib/kubelet/pods/fab2e407-04ed-4d98-be34-0ac71d1f2b55/volumes" Oct 31 14:00:45.698642 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:45.698604 6841 generic.go:356] "Generic (PLEG): container finished" podID="8cfa8aff-e9c5-464f-abe3-a5da3a967233" containerID="96f8ca8dcd990a49bfeea1375467aa8a924938239e52b86729f2724f8c3bf1fb" exitCode=0 Oct 31 14:00:45.699043 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:45.698652 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd8nh" event={"ID":"8cfa8aff-e9c5-464f-abe3-a5da3a967233","Type":"ContainerDied","Data":"96f8ca8dcd990a49bfeea1375467aa8a924938239e52b86729f2724f8c3bf1fb"} Oct 31 14:00:46.837230 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:46.837208 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd8nh" Oct 31 14:00:46.990394 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:46.990311 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cfa8aff-e9c5-464f-abe3-a5da3a967233-kube-api-access-8q9fp\" (UniqueName: \"kubernetes.io/projected/8cfa8aff-e9c5-464f-abe3-a5da3a967233-kube-api-access-8q9fp\") pod \"8cfa8aff-e9c5-464f-abe3-a5da3a967233\" (UID: \"8cfa8aff-e9c5-464f-abe3-a5da3a967233\") " Oct 31 14:00:46.990394 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:46.990348 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cfa8aff-e9c5-464f-abe3-a5da3a967233-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cfa8aff-e9c5-464f-abe3-a5da3a967233-maas-api-xks-ca\") pod \"8cfa8aff-e9c5-464f-abe3-a5da3a967233\" (UID: \"8cfa8aff-e9c5-464f-abe3-a5da3a967233\") " Oct 31 14:00:46.990601 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:46.990446 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cfa8aff-e9c5-464f-abe3-a5da3a967233-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cfa8aff-e9c5-464f-abe3-a5da3a967233-maas-api-openshift-ca\") pod \"8cfa8aff-e9c5-464f-abe3-a5da3a967233\" (UID: \"8cfa8aff-e9c5-464f-abe3-a5da3a967233\") " Oct 31 14:00:46.990796 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:46.990772 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cfa8aff-e9c5-464f-abe3-a5da3a967233-maas-api-openshift-ca" pod "8cfa8aff-e9c5-464f-abe3-a5da3a967233" (UID: "8cfa8aff-e9c5-464f-abe3-a5da3a967233"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:00:46.992540 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:46.992515 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cfa8aff-e9c5-464f-abe3-a5da3a967233-maas-api-xks-ca" pod "8cfa8aff-e9c5-464f-abe3-a5da3a967233" (UID: "8cfa8aff-e9c5-464f-abe3-a5da3a967233"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:00:46.992622 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:46.992536 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cfa8aff-e9c5-464f-abe3-a5da3a967233-kube-api-access-8q9fp" pod "8cfa8aff-e9c5-464f-abe3-a5da3a967233" (UID: "8cfa8aff-e9c5-464f-abe3-a5da3a967233"). InnerVolumeSpecName "kube-api-access-8q9fp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:00:47.091447 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:47.091409 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q9fp\" (UniqueName: \"kubernetes.io/projected/8cfa8aff-e9c5-464f-abe3-a5da3a967233-kube-api-access-8q9fp\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:47.091447 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:47.091439 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cfa8aff-e9c5-464f-abe3-a5da3a967233-maas-api-xks-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:47.091447 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:47.091451 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cfa8aff-e9c5-464f-abe3-a5da3a967233-maas-api-openshift-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:47.713086 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:47.713042 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd8nh" Oct 31 14:00:47.713303 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:47.713046 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd8nh" event={"ID":"8cfa8aff-e9c5-464f-abe3-a5da3a967233","Type":"ContainerDied","Data":"2ea17bcb079fd0ed22526a7050cc59016437846085912fe97d0a488294dba510"} Oct 31 14:00:47.713303 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:47.713231 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ea17bcb079fd0ed22526a7050cc59016437846085912fe97d0a488294dba510" Oct 31 14:00:50.033391 ip-10-0-138-70 kubenswrapper[6841]: E1031 14:00:50.033356 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 14:00:58.970429 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:58.970358 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:00:59.252661 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.252567 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw"] Oct 31 14:00:59.253114 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.253096 6841 cpu_manager.go:414] "containerMap: removing container" podUID="8cfa8aff-e9c5-464f-abe3-a5da3a967233" containerName="revoke-keys" Oct 31 14:00:59.253114 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.253114 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cfa8aff-e9c5-464f-abe3-a5da3a967233" containerName="revoke-keys" Oct 31 14:00:59.253265 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.253190 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cfa8aff-e9c5-464f-abe3-a5da3a967233" containerName="revoke-keys" Oct 31 14:00:59.256605 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.256586 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:00:59.261696 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.261675 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-5d3f86-openshift-default-dockercfg-zsbjv\"" Oct 31 14:00:59.272187 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.272163 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw"] Oct 31 14:00:59.335356 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.335325 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab81877e-88af-4508-82b3-7ee4e9a7e021-istio-podinfo\") pod \"e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw\" (UID: \"ab81877e-88af-4508-82b3-7ee4e9a7e021\") " pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:00:59.335529 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.335365 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab81877e-88af-4508-82b3-7ee4e9a7e021-credential-socket\") pod \"e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw\" (UID: \"ab81877e-88af-4508-82b3-7ee4e9a7e021\") " pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:00:59.335529 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.335386 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab81877e-88af-4508-82b3-7ee4e9a7e021-istio-data\") pod \"e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw\" (UID: \"ab81877e-88af-4508-82b3-7ee4e9a7e021\") " pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:00:59.335529 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.335408 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab81877e-88af-4508-82b3-7ee4e9a7e021-istio-token\") pod \"e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw\" (UID: \"ab81877e-88af-4508-82b3-7ee4e9a7e021\") " pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:00:59.335529 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.335502 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab81877e-88af-4508-82b3-7ee4e9a7e021-workload-socket\") pod \"e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw\" (UID: \"ab81877e-88af-4508-82b3-7ee4e9a7e021\") " pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:00:59.335712 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.335562 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab81877e-88af-4508-82b3-7ee4e9a7e021-istio-envoy\") pod \"e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw\" (UID: \"ab81877e-88af-4508-82b3-7ee4e9a7e021\") " pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:00:59.335712 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.335589 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttgpn\" (UniqueName: \"kubernetes.io/projected/ab81877e-88af-4508-82b3-7ee4e9a7e021-kube-api-access-ttgpn\") pod \"e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw\" (UID: \"ab81877e-88af-4508-82b3-7ee4e9a7e021\") " pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:00:59.335712 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.335694 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab81877e-88af-4508-82b3-7ee4e9a7e021-istiod-ca-cert\") pod \"e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw\" (UID: \"ab81877e-88af-4508-82b3-7ee4e9a7e021\") " pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:00:59.335841 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.335764 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab81877e-88af-4508-82b3-7ee4e9a7e021-workload-certs\") pod \"e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw\" (UID: \"ab81877e-88af-4508-82b3-7ee4e9a7e021\") " pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:00:59.382311 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.382277 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:00:59.436546 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.436506 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab81877e-88af-4508-82b3-7ee4e9a7e021-istiod-ca-cert\") pod \"e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw\" (UID: \"ab81877e-88af-4508-82b3-7ee4e9a7e021\") " pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:00:59.436733 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.436602 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab81877e-88af-4508-82b3-7ee4e9a7e021-workload-certs\") pod \"e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw\" (UID: \"ab81877e-88af-4508-82b3-7ee4e9a7e021\") " pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:00:59.436733 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.436635 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab81877e-88af-4508-82b3-7ee4e9a7e021-istio-podinfo\") pod \"e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw\" (UID: \"ab81877e-88af-4508-82b3-7ee4e9a7e021\") " pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:00:59.436733 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.436660 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab81877e-88af-4508-82b3-7ee4e9a7e021-credential-socket\") pod \"e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw\" (UID: \"ab81877e-88af-4508-82b3-7ee4e9a7e021\") " pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:00:59.436733 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.436676 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab81877e-88af-4508-82b3-7ee4e9a7e021-istio-data\") pod \"e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw\" (UID: \"ab81877e-88af-4508-82b3-7ee4e9a7e021\") " pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:00:59.436733 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.436693 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab81877e-88af-4508-82b3-7ee4e9a7e021-istio-token\") pod \"e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw\" (UID: \"ab81877e-88af-4508-82b3-7ee4e9a7e021\") " pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:00:59.436733 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.436718 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab81877e-88af-4508-82b3-7ee4e9a7e021-workload-socket\") pod \"e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw\" (UID: \"ab81877e-88af-4508-82b3-7ee4e9a7e021\") " pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:00:59.437133 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.436742 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab81877e-88af-4508-82b3-7ee4e9a7e021-istio-envoy\") pod \"e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw\" (UID: \"ab81877e-88af-4508-82b3-7ee4e9a7e021\") " pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:00:59.437133 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.436825 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttgpn\" (UniqueName: \"kubernetes.io/projected/ab81877e-88af-4508-82b3-7ee4e9a7e021-kube-api-access-ttgpn\") pod \"e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw\" (UID: \"ab81877e-88af-4508-82b3-7ee4e9a7e021\") " pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:00:59.437133 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.437052 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab81877e-88af-4508-82b3-7ee4e9a7e021-workload-certs\") pod \"e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw\" (UID: \"ab81877e-88af-4508-82b3-7ee4e9a7e021\") " pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:00:59.437317 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.437172 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab81877e-88af-4508-82b3-7ee4e9a7e021-workload-socket\") pod \"e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw\" (UID: \"ab81877e-88af-4508-82b3-7ee4e9a7e021\") " pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:00:59.437525 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.437500 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab81877e-88af-4508-82b3-7ee4e9a7e021-istiod-ca-cert\") pod \"e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw\" (UID: \"ab81877e-88af-4508-82b3-7ee4e9a7e021\") " pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:00:59.438136 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.438115 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab81877e-88af-4508-82b3-7ee4e9a7e021-credential-socket\") pod \"e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw\" (UID: \"ab81877e-88af-4508-82b3-7ee4e9a7e021\") " pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:00:59.438228 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.438145 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab81877e-88af-4508-82b3-7ee4e9a7e021-istio-data\") pod \"e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw\" (UID: \"ab81877e-88af-4508-82b3-7ee4e9a7e021\") " pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:00:59.439557 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.439529 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab81877e-88af-4508-82b3-7ee4e9a7e021-istio-envoy\") pod \"e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw\" (UID: \"ab81877e-88af-4508-82b3-7ee4e9a7e021\") " pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:00:59.440269 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.440250 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab81877e-88af-4508-82b3-7ee4e9a7e021-istio-podinfo\") pod \"e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw\" (UID: \"ab81877e-88af-4508-82b3-7ee4e9a7e021\") " pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:00:59.449957 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.449936 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttgpn\" (UniqueName: \"kubernetes.io/projected/ab81877e-88af-4508-82b3-7ee4e9a7e021-kube-api-access-ttgpn\") pod \"e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw\" (UID: \"ab81877e-88af-4508-82b3-7ee4e9a7e021\") " pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:00:59.450095 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.450077 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab81877e-88af-4508-82b3-7ee4e9a7e021-istio-token\") pod \"e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw\" (UID: \"ab81877e-88af-4508-82b3-7ee4e9a7e021\") " pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:00:59.569394 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.569310 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:00:59.583096 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.583062 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:00:59.855354 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.855320 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw"] Oct 31 14:00:59.856430 ip-10-0-138-70 kubenswrapper[6841]: W1031 14:00:59.856402 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab81877e_88af_4508_82b3_7ee4e9a7e021.slice/crio-b5598709f612aa534b72bc028ab299becd6391e4e57e7b5b0356697b8181adf2 WatchSource:0}: Error finding container b5598709f612aa534b72bc028ab299becd6391e4e57e7b5b0356697b8181adf2: Status 404 returned error can't find the container with id b5598709f612aa534b72bc028ab299becd6391e4e57e7b5b0356697b8181adf2 Oct 31 14:00:59.858628 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.858598 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 14:00:59.858701 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.858660 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 14:00:59.858701 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:00:59.858687 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 14:01:00.815308 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:00.815273 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" event={"ID":"ab81877e-88af-4508-82b3-7ee4e9a7e021","Type":"ContainerStarted","Data":"6a91ecac25f68866b7ad7d1a72fd595d0e5134239f4099e193df0b2ccc6d0dc1"} Oct 31 14:01:00.815775 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:00.815325 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" event={"ID":"ab81877e-88af-4508-82b3-7ee4e9a7e021","Type":"ContainerStarted","Data":"b5598709f612aa534b72bc028ab299becd6391e4e57e7b5b0356697b8181adf2"} Oct 31 14:01:00.842093 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:00.842048 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" podStartSLOduration=1.84203559 podStartE2EDuration="1.84203559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:01:00.839413159 +0000 UTC m=+5307.322536075" watchObservedRunningTime="2026-10-31 14:01:00.84203559 +0000 UTC m=+5307.325158490" Oct 31 14:01:01.570419 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:01.570379 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:01:01.575405 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:01.575376 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:01:01.822250 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:01.822162 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:01:01.823162 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:01.823142 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:01:03.191188 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:03.191151 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:03.388222 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:03.388180 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:04.232055 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.232011 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx"] Oct 31 14:01:04.237922 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.237892 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx" Oct 31 14:01:04.245023 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.244932 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx"] Oct 31 14:01:04.245376 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.245253 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 14:01:04.245509 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.245416 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 14:01:04.281506 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.281461 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 14:01:04.281749 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.281509 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-5d3f86-serving-cert\"" Oct 31 14:01:04.281869 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.281760 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rhftr\"" Oct 31 14:01:04.363043 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.362997 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c44bmjrj"] Oct 31 14:01:04.366886 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.366869 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c44bmjrj" Oct 31 14:01:04.372076 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.372052 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-5d3f86\"" Oct 31 14:01:04.372211 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.372118 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-fc5607b7-dockercfg-68z22\"" Oct 31 14:01:04.382825 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.382804 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c44bmjrj"] Oct 31 14:01:04.396807 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.396738 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47e20e00-07b7-47d6-bbdc-aa1595401671-metrics-certs\") pod \"maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx\" (UID: \"47e20e00-07b7-47d6-bbdc-aa1595401671\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx" Oct 31 14:01:04.396807 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.396801 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47e20e00-07b7-47d6-bbdc-aa1595401671-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx\" (UID: \"47e20e00-07b7-47d6-bbdc-aa1595401671\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx" Oct 31 14:01:04.397067 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.396876 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47e20e00-07b7-47d6-bbdc-aa1595401671-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx\" (UID: \"47e20e00-07b7-47d6-bbdc-aa1595401671\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx" Oct 31 14:01:04.397067 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.396956 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stnhh\" (UniqueName: \"kubernetes.io/projected/47e20e00-07b7-47d6-bbdc-aa1595401671-kube-api-access-stnhh\") pod \"maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx\" (UID: \"47e20e00-07b7-47d6-bbdc-aa1595401671\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx" Oct 31 14:01:04.405388 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.405359 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6bj97d9"] Oct 31 14:01:04.409079 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.409062 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6bj97d9" Oct 31 14:01:04.419485 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.419460 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:04.421239 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.421203 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6bj97d9"] Oct 31 14:01:04.497669 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.497570 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47e20e00-07b7-47d6-bbdc-aa1595401671-metrics-certs\") pod \"maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx\" (UID: \"47e20e00-07b7-47d6-bbdc-aa1595401671\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx" Oct 31 14:01:04.497669 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.497637 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47e20e00-07b7-47d6-bbdc-aa1595401671-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx\" (UID: \"47e20e00-07b7-47d6-bbdc-aa1595401671\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx" Oct 31 14:01:04.498290 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.497706 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47e20e00-07b7-47d6-bbdc-aa1595401671-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx\" (UID: \"47e20e00-07b7-47d6-bbdc-aa1595401671\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx" Oct 31 14:01:04.498290 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.497745 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzrgj\" (UniqueName: \"kubernetes.io/projected/cf735100-a839-46f3-9f12-4070a7ed7571-kube-api-access-vzrgj\") pod \"payload-pre-processing-e2e-worker-single-main-5d3f86-59c44bmjrj\" (UID: \"cf735100-a839-46f3-9f12-4070a7ed7571\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c44bmjrj" Oct 31 14:01:04.498290 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.497812 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stnhh\" (UniqueName: \"kubernetes.io/projected/47e20e00-07b7-47d6-bbdc-aa1595401671-kube-api-access-stnhh\") pod \"maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx\" (UID: \"47e20e00-07b7-47d6-bbdc-aa1595401671\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx" Oct 31 14:01:04.498290 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.497885 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf735100-a839-46f3-9f12-4070a7ed7571-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-5d3f86-59c44bmjrj\" (UID: \"cf735100-a839-46f3-9f12-4070a7ed7571\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c44bmjrj" Oct 31 14:01:04.499684 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.498729 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47e20e00-07b7-47d6-bbdc-aa1595401671-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx\" (UID: \"47e20e00-07b7-47d6-bbdc-aa1595401671\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx" Oct 31 14:01:04.502952 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.502902 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47e20e00-07b7-47d6-bbdc-aa1595401671-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx\" (UID: \"47e20e00-07b7-47d6-bbdc-aa1595401671\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx" Oct 31 14:01:04.504177 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.504159 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47e20e00-07b7-47d6-bbdc-aa1595401671-metrics-certs\") pod \"maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx\" (UID: \"47e20e00-07b7-47d6-bbdc-aa1595401671\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx" Oct 31 14:01:04.512610 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.512580 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stnhh\" (UniqueName: \"kubernetes.io/projected/47e20e00-07b7-47d6-bbdc-aa1595401671-kube-api-access-stnhh\") pod \"maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx\" (UID: \"47e20e00-07b7-47d6-bbdc-aa1595401671\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx" Oct 31 14:01:04.559565 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.559533 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx" Oct 31 14:01:04.599477 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.599401 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp8rx\" (UniqueName: \"kubernetes.io/projected/b83e3502-59c9-4edf-a4ba-b4ce136f2df4-kube-api-access-pp8rx\") pod \"payload-processing-e2e-worker-single-main-5d3f86-695d6cc6bj97d9\" (UID: \"b83e3502-59c9-4edf-a4ba-b4ce136f2df4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6bj97d9" Oct 31 14:01:04.599477 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.599478 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzrgj\" (UniqueName: \"kubernetes.io/projected/cf735100-a839-46f3-9f12-4070a7ed7571-kube-api-access-vzrgj\") pod \"payload-pre-processing-e2e-worker-single-main-5d3f86-59c44bmjrj\" (UID: \"cf735100-a839-46f3-9f12-4070a7ed7571\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c44bmjrj" Oct 31 14:01:04.599726 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.599576 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf735100-a839-46f3-9f12-4070a7ed7571-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-5d3f86-59c44bmjrj\" (UID: \"cf735100-a839-46f3-9f12-4070a7ed7571\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c44bmjrj" Oct 31 14:01:04.599726 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.599690 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b83e3502-59c9-4edf-a4ba-b4ce136f2df4-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-5d3f86-695d6cc6bj97d9\" (UID: \"b83e3502-59c9-4edf-a4ba-b4ce136f2df4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6bj97d9" Oct 31 14:01:04.600609 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.600581 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf735100-a839-46f3-9f12-4070a7ed7571-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-5d3f86-59c44bmjrj\" (UID: \"cf735100-a839-46f3-9f12-4070a7ed7571\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c44bmjrj" Oct 31 14:01:04.613775 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.613751 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzrgj\" (UniqueName: \"kubernetes.io/projected/cf735100-a839-46f3-9f12-4070a7ed7571-kube-api-access-vzrgj\") pod \"payload-pre-processing-e2e-worker-single-main-5d3f86-59c44bmjrj\" (UID: \"cf735100-a839-46f3-9f12-4070a7ed7571\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c44bmjrj" Oct 31 14:01:04.629677 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.629645 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:04.680814 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.680763 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c44bmjrj" Oct 31 14:01:04.700481 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.700454 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b83e3502-59c9-4edf-a4ba-b4ce136f2df4-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-5d3f86-695d6cc6bj97d9\" (UID: \"b83e3502-59c9-4edf-a4ba-b4ce136f2df4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6bj97d9" Oct 31 14:01:04.700605 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.700503 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp8rx\" (UniqueName: \"kubernetes.io/projected/b83e3502-59c9-4edf-a4ba-b4ce136f2df4-kube-api-access-pp8rx\") pod \"payload-processing-e2e-worker-single-main-5d3f86-695d6cc6bj97d9\" (UID: \"b83e3502-59c9-4edf-a4ba-b4ce136f2df4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6bj97d9" Oct 31 14:01:04.701176 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.701153 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b83e3502-59c9-4edf-a4ba-b4ce136f2df4-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-5d3f86-695d6cc6bj97d9\" (UID: \"b83e3502-59c9-4edf-a4ba-b4ce136f2df4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6bj97d9" Oct 31 14:01:04.712696 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.712671 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp8rx\" (UniqueName: \"kubernetes.io/projected/b83e3502-59c9-4edf-a4ba-b4ce136f2df4-kube-api-access-pp8rx\") pod \"payload-processing-e2e-worker-single-main-5d3f86-695d6cc6bj97d9\" (UID: \"b83e3502-59c9-4edf-a4ba-b4ce136f2df4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6bj97d9" Oct 31 14:01:04.725220 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.725196 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6bj97d9" Oct 31 14:01:04.838595 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.838569 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx"] Oct 31 14:01:04.841119 ip-10-0-138-70 kubenswrapper[6841]: W1031 14:01:04.840447 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e20e00_07b7_47d6_bbdc_aa1595401671.slice/crio-1f32f73e3484cbb1c20e141de0e58adf8d3941d1e8b1228bf04206fbc99f9743 WatchSource:0}: Error finding container 1f32f73e3484cbb1c20e141de0e58adf8d3941d1e8b1228bf04206fbc99f9743: Status 404 returned error can't find the container with id 1f32f73e3484cbb1c20e141de0e58adf8d3941d1e8b1228bf04206fbc99f9743 Oct 31 14:01:04.842025 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.841978 6841 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 14:01:04.956363 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:04.956340 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c44bmjrj"] Oct 31 14:01:04.957973 ip-10-0-138-70 kubenswrapper[6841]: W1031 14:01:04.957940 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf735100_a839_46f3_9f12_4070a7ed7571.slice/crio-9bdb2be182a8954e0b8814b9006abaabdde75a48e2477026e5c98d52093c8435 WatchSource:0}: Error finding container 9bdb2be182a8954e0b8814b9006abaabdde75a48e2477026e5c98d52093c8435: Status 404 returned error can't find the container with id 9bdb2be182a8954e0b8814b9006abaabdde75a48e2477026e5c98d52093c8435 Oct 31 14:01:05.014143 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:05.014100 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6bj97d9"] Oct 31 14:01:05.015088 ip-10-0-138-70 kubenswrapper[6841]: W1031 14:01:05.015060 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb83e3502_59c9_4edf_a4ba_b4ce136f2df4.slice/crio-65beba5d666e53f7b9eecd6be2a5e39d6e8a79b06361c4597831e0ae2dc553a5 WatchSource:0}: Error finding container 65beba5d666e53f7b9eecd6be2a5e39d6e8a79b06361c4597831e0ae2dc553a5: Status 404 returned error can't find the container with id 65beba5d666e53f7b9eecd6be2a5e39d6e8a79b06361c4597831e0ae2dc553a5 Oct 31 14:01:05.389155 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:05.389123 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:05.850952 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:05.850914 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx" event={"ID":"47e20e00-07b7-47d6-bbdc-aa1595401671","Type":"ContainerStarted","Data":"fff945822c50d3305d0f3de07527cb8456000c0261e8567f40293ca05987a5e0"} Oct 31 14:01:05.851189 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:05.850971 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx" event={"ID":"47e20e00-07b7-47d6-bbdc-aa1595401671","Type":"ContainerStarted","Data":"1f32f73e3484cbb1c20e141de0e58adf8d3941d1e8b1228bf04206fbc99f9743"} Oct 31 14:01:05.851189 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:05.851029 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx" Oct 31 14:01:05.852536 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:05.852510 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c44bmjrj" event={"ID":"cf735100-a839-46f3-9f12-4070a7ed7571","Type":"ContainerStarted","Data":"e74961b6668324f17e056d3e8810aa0d954b8e7b9195348caaf560762f64f637"} Oct 31 14:01:05.852666 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:05.852560 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c44bmjrj" event={"ID":"cf735100-a839-46f3-9f12-4070a7ed7571","Type":"ContainerStarted","Data":"9bdb2be182a8954e0b8814b9006abaabdde75a48e2477026e5c98d52093c8435"} Oct 31 14:01:05.852666 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:05.852601 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c44bmjrj" Oct 31 14:01:05.854216 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:05.854193 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6bj97d9" event={"ID":"b83e3502-59c9-4edf-a4ba-b4ce136f2df4","Type":"ContainerStarted","Data":"6e18eaf7907453be47da81b341a38ab6fe5a2ea573cea2adefc3163e40fa384f"} Oct 31 14:01:05.854308 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:05.854234 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6bj97d9" event={"ID":"b83e3502-59c9-4edf-a4ba-b4ce136f2df4","Type":"ContainerStarted","Data":"65beba5d666e53f7b9eecd6be2a5e39d6e8a79b06361c4597831e0ae2dc553a5"} Oct 31 14:01:05.854308 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:05.854283 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6bj97d9" Oct 31 14:01:05.870797 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:05.870750 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx" podStartSLOduration=1.36849894 podStartE2EDuration="1.870734161s" totalImagesPullingTime="502.235221ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:01:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:01:05.870699173 +0000 UTC m=+5312.353822078" watchObservedRunningTime="2026-10-31 14:01:05.870734161 +0000 UTC m=+5312.353857063" Oct 31 14:01:05.891908 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:05.891860 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c44bmjrj" podStartSLOduration=1.8918247670000001 podStartE2EDuration="1.891824767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:01:05.888706787 +0000 UTC m=+5312.371829691" watchObservedRunningTime="2026-10-31 14:01:05.891824767 +0000 UTC m=+5312.374947667" Oct 31 14:01:05.907894 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:05.907855 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6bj97d9" podStartSLOduration=1.9078425810000001 podStartE2EDuration="1.907842581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:01:05.906380716 +0000 UTC m=+5312.389503620" watchObservedRunningTime="2026-10-31 14:01:05.907842581 +0000 UTC m=+5312.390965482" Oct 31 14:01:06.102413 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:06.102324 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:06.793151 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:06.793111 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:07.571934 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:07.571898 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:08.373863 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:08.373819 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:09.195767 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:09.195735 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:10.083569 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:10.083538 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:10.887218 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:10.887177 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:11.599593 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:11.599554 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:11.873289 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:11.873211 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx" Oct 31 14:01:12.393867 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:12.393831 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:13.267885 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:13.267849 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:14.053115 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:14.053074 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:14.776376 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:14.776340 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:15.467581 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:15.467547 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:16.178123 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:16.178087 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:16.824544 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:16.824172 6841 patch_prober.go:29] interesting pod/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" start-of-body= Oct 31 14:01:16.824544 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:16.824239 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" podUID="ab81877e-88af-4508-82b3-7ee4e9a7e021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" Oct 31 14:01:16.869702 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:16.869677 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c44bmjrj" Oct 31 14:01:16.870260 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:16.870228 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6bj97d9" Oct 31 14:01:16.874205 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:16.874183 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:16.934416 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:16.934329 6841 generic.go:356] "Generic (PLEG): container finished" podID="ab81877e-88af-4508-82b3-7ee4e9a7e021" containerID="6a91ecac25f68866b7ad7d1a72fd595d0e5134239f4099e193df0b2ccc6d0dc1" exitCode=0 Oct 31 14:01:16.934416 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:16.934392 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" event={"ID":"ab81877e-88af-4508-82b3-7ee4e9a7e021","Type":"ContainerDied","Data":"6a91ecac25f68866b7ad7d1a72fd595d0e5134239f4099e193df0b2ccc6d0dc1"} Oct 31 14:01:16.934815 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:16.934797 6841 scope.go:122] "RemoveContainer" containerID="6a91ecac25f68866b7ad7d1a72fd595d0e5134239f4099e193df0b2ccc6d0dc1" Oct 31 14:01:16.936125 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:16.936098 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 14:01:16.936243 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:16.936174 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 14:01:16.936243 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:16.936223 6841 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 14:01:17.694043 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:17.694004 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:17.959070 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:17.958628 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" event={"ID":"ab81877e-88af-4508-82b3-7ee4e9a7e021","Type":"ContainerStarted","Data":"94c8f4fdf9489ba4b00f9daa0dbbd307ada07f75c5cb0a0d5de329eb7460b756"} Oct 31 14:01:18.216426 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:18.216334 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:18.570481 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:18.570386 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:01:18.571763 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:18.570763 6841 patch_prober.go:29] interesting pod/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" start-of-body= Oct 31 14:01:18.571973 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:18.571948 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" podUID="ab81877e-88af-4508-82b3-7ee4e9a7e021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" Oct 31 14:01:18.594782 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:18.594755 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:18.867106 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:18.864456 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:19.068746 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:19.068708 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:19.496783 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:19.496742 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:19.570669 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:19.570629 6841 patch_prober.go:29] interesting pod/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" start-of-body= Oct 31 14:01:19.570853 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:19.570690 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" podUID="ab81877e-88af-4508-82b3-7ee4e9a7e021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" Oct 31 14:01:20.034678 ip-10-0-138-70 kubenswrapper[6841]: E1031 14:01:20.034645 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 14:01:20.504088 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:20.504051 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:20.570608 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:20.570564 6841 patch_prober.go:29] interesting pod/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" start-of-body= Oct 31 14:01:20.570843 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:20.570636 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" podUID="ab81877e-88af-4508-82b3-7ee4e9a7e021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" Oct 31 14:01:20.816323 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:20.816238 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:21.290783 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:21.290746 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:21.570605 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:21.570521 6841 patch_prober.go:29] interesting pod/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" start-of-body= Oct 31 14:01:21.570751 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:21.570595 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" podUID="ab81877e-88af-4508-82b3-7ee4e9a7e021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" Oct 31 14:01:22.090181 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:22.090149 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:22.570584 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:22.570539 6841 patch_prober.go:29] interesting pod/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" start-of-body= Oct 31 14:01:22.571132 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:22.570607 6841 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" podUID="ab81877e-88af-4508-82b3-7ee4e9a7e021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" Oct 31 14:01:22.992508 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:22.992475 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:23.574399 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:23.574363 6841 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:01:23.574922 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:23.574654 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:01:23.575356 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:23.575337 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:01:23.875122 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:23.875015 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:24.583314 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:24.583276 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:25.376564 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:25.376525 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:26.238816 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:26.238784 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:26.974720 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:26.974685 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:27.698319 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:27.698279 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:28.489458 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:28.489427 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:29.185330 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:29.185291 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:29.885962 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:29.885919 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:30.782311 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:30.782258 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:31.578461 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:31.578418 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:32.287328 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:32.287287 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:32.980375 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:32.980339 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:33.677542 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:33.677505 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:34.380257 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:34.380217 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:35.079304 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:35.079266 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:35.781150 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:35.781111 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:36.661946 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:36.661911 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:37.383038 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:37.383001 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:38.092462 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:38.092427 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:38.880997 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:38.880944 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:39.811100 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:39.811052 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:40.550952 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:40.550914 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:41.364261 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:41.364227 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:42.037966 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:42.037932 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:42.237958 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:42.237893 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:42.766897 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:42.766864 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:43.469176 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:43.469148 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:44.163572 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:44.163540 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:44.880007 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:44.879945 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:45.587384 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:45.587349 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:46.278354 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:46.278316 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:46.975352 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:46.975318 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:47.680230 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:47.680199 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:48.385774 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:48.385739 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:49.081289 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:49.081252 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:49.886596 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:49.886562 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:50.034795 ip-10-0-138-70 kubenswrapper[6841]: E1031 14:01:50.034764 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 14:01:50.586856 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:50.586827 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:51.273302 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:51.273266 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:51.986017 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:51.985964 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:52.679749 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:52.679710 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:53.381537 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:53.381504 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:54.065998 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:54.065955 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:54.864221 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:54.864185 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:55.672669 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:55.672634 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:56.423583 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:56.423546 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:57.166655 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:57.166617 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:57.866487 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:57.866443 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:58.577064 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:58.577030 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:59.268124 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:59.268087 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:01:59.986478 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:01:59.986438 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:00.033580 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:00.033542 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-w724m"] Oct 31 14:02:00.036116 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:00.036085 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-w724m"] Oct 31 14:02:00.786180 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:00.786148 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:01.212826 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:01.212793 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:01.589457 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:01.589372 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:01.641565 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:01.641529 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7756303-b1fa-4e6b-9ecb-dea12e211565" path="/var/lib/kubelet/pods/e7756303-b1fa-4e6b-9ecb-dea12e211565/volumes" Oct 31 14:02:01.788713 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:01.788672 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:02.365035 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:02.364997 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:03.085339 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:03.085304 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:03.984894 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:03.984857 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:04.690266 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:04.690233 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:05.389452 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:05.389416 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:06.089559 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:06.089522 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:06.286476 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:06.286434 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:06.778585 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:06.778550 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:07.067517 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:07.067432 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6bj97d9"] Oct 31 14:02:07.089789 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:07.089759 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c44bmjrj"] Oct 31 14:02:07.311465 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:07.311427 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:07.544251 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:07.544213 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:07.742565 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:07.742534 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:08.096624 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:08.096582 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx"] Oct 31 14:02:08.097205 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:08.096828 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx" podUID="47e20e00-07b7-47d6-bbdc-aa1595401671" containerName="maas-api" containerID="cri-o://fff945822c50d3305d0f3de07527cb8456000c0261e8567f40293ca05987a5e0" gracePeriod=30 Oct 31 14:02:08.243656 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:08.243620 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:08.442312 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:08.442274 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:08.547824 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:08.547798 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx" Oct 31 14:02:08.707519 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:08.707490 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e20e00-07b7-47d6-bbdc-aa1595401671-kube-api-access-stnhh\" (UniqueName: \"kubernetes.io/projected/47e20e00-07b7-47d6-bbdc-aa1595401671-kube-api-access-stnhh\") pod \"47e20e00-07b7-47d6-bbdc-aa1595401671\" (UID: \"47e20e00-07b7-47d6-bbdc-aa1595401671\") " Oct 31 14:02:08.707720 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:08.707541 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47e20e00-07b7-47d6-bbdc-aa1595401671-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47e20e00-07b7-47d6-bbdc-aa1595401671-openshift-service-ca\") pod \"47e20e00-07b7-47d6-bbdc-aa1595401671\" (UID: \"47e20e00-07b7-47d6-bbdc-aa1595401671\") " Oct 31 14:02:08.707720 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:08.707571 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47e20e00-07b7-47d6-bbdc-aa1595401671-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47e20e00-07b7-47d6-bbdc-aa1595401671-maas-api-tls\") pod \"47e20e00-07b7-47d6-bbdc-aa1595401671\" (UID: \"47e20e00-07b7-47d6-bbdc-aa1595401671\") " Oct 31 14:02:08.707720 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:08.707622 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47e20e00-07b7-47d6-bbdc-aa1595401671-metrics-certs\" (UniqueName: \"kubernetes.io/secret/47e20e00-07b7-47d6-bbdc-aa1595401671-metrics-certs\") pod \"47e20e00-07b7-47d6-bbdc-aa1595401671\" (UID: \"47e20e00-07b7-47d6-bbdc-aa1595401671\") " Oct 31 14:02:08.708063 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:08.708038 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e20e00-07b7-47d6-bbdc-aa1595401671-openshift-service-ca" pod "47e20e00-07b7-47d6-bbdc-aa1595401671" (UID: "47e20e00-07b7-47d6-bbdc-aa1595401671"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:02:08.709843 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:08.709819 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47e20e00-07b7-47d6-bbdc-aa1595401671-metrics-certs" pod "47e20e00-07b7-47d6-bbdc-aa1595401671" (UID: "47e20e00-07b7-47d6-bbdc-aa1595401671"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:02:08.709943 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:08.709895 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e20e00-07b7-47d6-bbdc-aa1595401671-kube-api-access-stnhh" pod "47e20e00-07b7-47d6-bbdc-aa1595401671" (UID: "47e20e00-07b7-47d6-bbdc-aa1595401671"). InnerVolumeSpecName "kube-api-access-stnhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:02:08.709943 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:08.709918 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47e20e00-07b7-47d6-bbdc-aa1595401671-maas-api-tls" pod "47e20e00-07b7-47d6-bbdc-aa1595401671" (UID: "47e20e00-07b7-47d6-bbdc-aa1595401671"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:02:08.809234 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:08.809203 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47e20e00-07b7-47d6-bbdc-aa1595401671-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:08.809234 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:08.809229 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47e20e00-07b7-47d6-bbdc-aa1595401671-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:08.809234 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:08.809240 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47e20e00-07b7-47d6-bbdc-aa1595401671-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:08.809458 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:08.809250 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stnhh\" (UniqueName: \"kubernetes.io/projected/47e20e00-07b7-47d6-bbdc-aa1595401671-kube-api-access-stnhh\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:09.298248 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:09.298217 6841 generic.go:356] "Generic (PLEG): container finished" podID="47e20e00-07b7-47d6-bbdc-aa1595401671" containerID="fff945822c50d3305d0f3de07527cb8456000c0261e8567f40293ca05987a5e0" exitCode=0 Oct 31 14:02:09.298665 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:09.298265 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx" Oct 31 14:02:09.298665 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:09.298310 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx" event={"ID":"47e20e00-07b7-47d6-bbdc-aa1595401671","Type":"ContainerDied","Data":"fff945822c50d3305d0f3de07527cb8456000c0261e8567f40293ca05987a5e0"} Oct 31 14:02:09.298665 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:09.298365 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx" event={"ID":"47e20e00-07b7-47d6-bbdc-aa1595401671","Type":"ContainerDied","Data":"1f32f73e3484cbb1c20e141de0e58adf8d3941d1e8b1228bf04206fbc99f9743"} Oct 31 14:02:09.298665 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:09.298388 6841 scope.go:122] "RemoveContainer" containerID="fff945822c50d3305d0f3de07527cb8456000c0261e8567f40293ca05987a5e0" Oct 31 14:02:09.309707 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:09.309687 6841 scope.go:122] "RemoveContainer" containerID="fff945822c50d3305d0f3de07527cb8456000c0261e8567f40293ca05987a5e0" Oct 31 14:02:09.309963 ip-10-0-138-70 kubenswrapper[6841]: E1031 14:02:09.309942 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fff945822c50d3305d0f3de07527cb8456000c0261e8567f40293ca05987a5e0\": container with ID starting with fff945822c50d3305d0f3de07527cb8456000c0261e8567f40293ca05987a5e0 not found: ID does not exist" containerID="fff945822c50d3305d0f3de07527cb8456000c0261e8567f40293ca05987a5e0" Oct 31 14:02:09.310049 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:09.309971 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fff945822c50d3305d0f3de07527cb8456000c0261e8567f40293ca05987a5e0"} err="failed to get container status \"fff945822c50d3305d0f3de07527cb8456000c0261e8567f40293ca05987a5e0\": rpc error: code = NotFound desc = could not find container \"fff945822c50d3305d0f3de07527cb8456000c0261e8567f40293ca05987a5e0\": container with ID starting with fff945822c50d3305d0f3de07527cb8456000c0261e8567f40293ca05987a5e0 not found: ID does not exist" Oct 31 14:02:09.327081 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:09.327052 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx"] Oct 31 14:02:09.330189 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:09.330165 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-72pvx"] Oct 31 14:02:09.641747 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:09.641667 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47e20e00-07b7-47d6-bbdc-aa1595401671" path="/var/lib/kubelet/pods/47e20e00-07b7-47d6-bbdc-aa1595401671/volumes" Oct 31 14:02:12.068193 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:12.068148 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6bj97d9" podUID="b83e3502-59c9-4edf-a4ba-b4ce136f2df4" containerName="payload-processing" containerID="cri-o://6e18eaf7907453be47da81b341a38ab6fe5a2ea573cea2adefc3163e40fa384f" gracePeriod=25 Oct 31 14:02:12.090279 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:12.090242 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c44bmjrj" podUID="cf735100-a839-46f3-9f12-4070a7ed7571" containerName="payload-pre-processing" containerID="cri-o://e74961b6668324f17e056d3e8810aa0d954b8e7b9195348caaf560762f64f637" gracePeriod=25 Oct 31 14:02:14.081683 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:14.081645 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:15.660056 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:15.660019 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:16.868965 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:16.868919 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c44bmjrj" podUID="cf735100-a839-46f3-9f12-4070a7ed7571" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 14:02:16.869391 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:16.868947 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6bj97d9" podUID="b83e3502-59c9-4edf-a4ba-b4ce136f2df4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 14:02:20.034343 ip-10-0-138-70 kubenswrapper[6841]: E1031 14:02:20.034304 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 14:02:21.025163 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:21.025129 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cxsxg"] Oct 31 14:02:21.027694 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:21.027663 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cxsxg"] Oct 31 14:02:21.641415 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:21.641382 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cb24607-1513-4cd1-b3d1-ec45ec1f26b5" path="/var/lib/kubelet/pods/2cb24607-1513-4cd1-b3d1-ec45ec1f26b5/volumes" Oct 31 14:02:26.868688 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:26.868646 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c44bmjrj" podUID="cf735100-a839-46f3-9f12-4070a7ed7571" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 14:02:26.869159 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:26.868656 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6bj97d9" podUID="b83e3502-59c9-4edf-a4ba-b4ce136f2df4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 14:02:32.338366 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:32.338303 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6bj97d9" Oct 31 14:02:32.364223 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:32.364180 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c44bmjrj" Oct 31 14:02:32.435997 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:32.435964 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b83e3502-59c9-4edf-a4ba-b4ce136f2df4-kube-api-access-pp8rx\" (UniqueName: \"kubernetes.io/projected/b83e3502-59c9-4edf-a4ba-b4ce136f2df4-kube-api-access-pp8rx\") pod \"b83e3502-59c9-4edf-a4ba-b4ce136f2df4\" (UID: \"b83e3502-59c9-4edf-a4ba-b4ce136f2df4\") " Oct 31 14:02:32.436205 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:32.436048 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf735100-a839-46f3-9f12-4070a7ed7571-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf735100-a839-46f3-9f12-4070a7ed7571-plugins-config-volume\") pod \"cf735100-a839-46f3-9f12-4070a7ed7571\" (UID: \"cf735100-a839-46f3-9f12-4070a7ed7571\") " Oct 31 14:02:32.436205 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:32.436091 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b83e3502-59c9-4edf-a4ba-b4ce136f2df4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b83e3502-59c9-4edf-a4ba-b4ce136f2df4-plugins-config-volume\") pod \"b83e3502-59c9-4edf-a4ba-b4ce136f2df4\" (UID: \"b83e3502-59c9-4edf-a4ba-b4ce136f2df4\") " Oct 31 14:02:32.436205 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:32.436157 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf735100-a839-46f3-9f12-4070a7ed7571-kube-api-access-vzrgj\" (UniqueName: \"kubernetes.io/projected/cf735100-a839-46f3-9f12-4070a7ed7571-kube-api-access-vzrgj\") pod \"cf735100-a839-46f3-9f12-4070a7ed7571\" (UID: \"cf735100-a839-46f3-9f12-4070a7ed7571\") " Oct 31 14:02:32.436488 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:32.436454 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b83e3502-59c9-4edf-a4ba-b4ce136f2df4-plugins-config-volume" pod "b83e3502-59c9-4edf-a4ba-b4ce136f2df4" (UID: "b83e3502-59c9-4edf-a4ba-b4ce136f2df4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:02:32.436552 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:32.436464 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf735100-a839-46f3-9f12-4070a7ed7571-plugins-config-volume" pod "cf735100-a839-46f3-9f12-4070a7ed7571" (UID: "cf735100-a839-46f3-9f12-4070a7ed7571"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:02:32.438340 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:32.438314 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf735100-a839-46f3-9f12-4070a7ed7571-kube-api-access-vzrgj" pod "cf735100-a839-46f3-9f12-4070a7ed7571" (UID: "cf735100-a839-46f3-9f12-4070a7ed7571"). InnerVolumeSpecName "kube-api-access-vzrgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:02:32.438479 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:32.438381 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b83e3502-59c9-4edf-a4ba-b4ce136f2df4-kube-api-access-pp8rx" pod "b83e3502-59c9-4edf-a4ba-b4ce136f2df4" (UID: "b83e3502-59c9-4edf-a4ba-b4ce136f2df4"). InnerVolumeSpecName "kube-api-access-pp8rx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:02:32.439046 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:32.439024 6841 generic.go:356] "Generic (PLEG): container finished" podID="cf735100-a839-46f3-9f12-4070a7ed7571" containerID="e74961b6668324f17e056d3e8810aa0d954b8e7b9195348caaf560762f64f637" exitCode=0 Oct 31 14:02:32.439163 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:32.439069 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c44bmjrj" Oct 31 14:02:32.439163 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:32.439109 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c44bmjrj" event={"ID":"cf735100-a839-46f3-9f12-4070a7ed7571","Type":"ContainerDied","Data":"e74961b6668324f17e056d3e8810aa0d954b8e7b9195348caaf560762f64f637"} Oct 31 14:02:32.439289 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:32.439161 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c44bmjrj" event={"ID":"cf735100-a839-46f3-9f12-4070a7ed7571","Type":"ContainerDied","Data":"9bdb2be182a8954e0b8814b9006abaabdde75a48e2477026e5c98d52093c8435"} Oct 31 14:02:32.439289 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:32.439192 6841 scope.go:122] "RemoveContainer" containerID="e74961b6668324f17e056d3e8810aa0d954b8e7b9195348caaf560762f64f637" Oct 31 14:02:32.440714 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:32.440690 6841 generic.go:356] "Generic (PLEG): container finished" podID="b83e3502-59c9-4edf-a4ba-b4ce136f2df4" containerID="6e18eaf7907453be47da81b341a38ab6fe5a2ea573cea2adefc3163e40fa384f" exitCode=0 Oct 31 14:02:32.440855 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:32.440721 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6bj97d9" event={"ID":"b83e3502-59c9-4edf-a4ba-b4ce136f2df4","Type":"ContainerDied","Data":"6e18eaf7907453be47da81b341a38ab6fe5a2ea573cea2adefc3163e40fa384f"} Oct 31 14:02:32.440855 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:32.440729 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6bj97d9" Oct 31 14:02:32.440855 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:32.440749 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6bj97d9" event={"ID":"b83e3502-59c9-4edf-a4ba-b4ce136f2df4","Type":"ContainerDied","Data":"65beba5d666e53f7b9eecd6be2a5e39d6e8a79b06361c4597831e0ae2dc553a5"} Oct 31 14:02:32.449929 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:32.449913 6841 scope.go:122] "RemoveContainer" containerID="e74961b6668324f17e056d3e8810aa0d954b8e7b9195348caaf560762f64f637" Oct 31 14:02:32.450236 ip-10-0-138-70 kubenswrapper[6841]: E1031 14:02:32.450218 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e74961b6668324f17e056d3e8810aa0d954b8e7b9195348caaf560762f64f637\": container with ID starting with e74961b6668324f17e056d3e8810aa0d954b8e7b9195348caaf560762f64f637 not found: ID does not exist" containerID="e74961b6668324f17e056d3e8810aa0d954b8e7b9195348caaf560762f64f637" Oct 31 14:02:32.450306 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:32.450246 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e74961b6668324f17e056d3e8810aa0d954b8e7b9195348caaf560762f64f637"} err="failed to get container status \"e74961b6668324f17e056d3e8810aa0d954b8e7b9195348caaf560762f64f637\": rpc error: code = NotFound desc = could not find container \"e74961b6668324f17e056d3e8810aa0d954b8e7b9195348caaf560762f64f637\": container with ID starting with e74961b6668324f17e056d3e8810aa0d954b8e7b9195348caaf560762f64f637 not found: ID does not exist" Oct 31 14:02:32.450306 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:32.450263 6841 scope.go:122] "RemoveContainer" containerID="6e18eaf7907453be47da81b341a38ab6fe5a2ea573cea2adefc3163e40fa384f" Oct 31 14:02:32.459678 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:32.459660 6841 scope.go:122] "RemoveContainer" containerID="6e18eaf7907453be47da81b341a38ab6fe5a2ea573cea2adefc3163e40fa384f" Oct 31 14:02:32.459922 ip-10-0-138-70 kubenswrapper[6841]: E1031 14:02:32.459902 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e18eaf7907453be47da81b341a38ab6fe5a2ea573cea2adefc3163e40fa384f\": container with ID starting with 6e18eaf7907453be47da81b341a38ab6fe5a2ea573cea2adefc3163e40fa384f not found: ID does not exist" containerID="6e18eaf7907453be47da81b341a38ab6fe5a2ea573cea2adefc3163e40fa384f" Oct 31 14:02:32.460001 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:32.459929 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e18eaf7907453be47da81b341a38ab6fe5a2ea573cea2adefc3163e40fa384f"} err="failed to get container status \"6e18eaf7907453be47da81b341a38ab6fe5a2ea573cea2adefc3163e40fa384f\": rpc error: code = NotFound desc = could not find container \"6e18eaf7907453be47da81b341a38ab6fe5a2ea573cea2adefc3163e40fa384f\": container with ID starting with 6e18eaf7907453be47da81b341a38ab6fe5a2ea573cea2adefc3163e40fa384f not found: ID does not exist" Oct 31 14:02:32.465422 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:32.465399 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c44bmjrj"] Oct 31 14:02:32.469793 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:32.469772 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c44bmjrj"] Oct 31 14:02:32.480528 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:32.480509 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6bj97d9"] Oct 31 14:02:32.486673 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:32.486653 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6bj97d9"] Oct 31 14:02:32.536977 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:32.536951 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp8rx\" (UniqueName: \"kubernetes.io/projected/b83e3502-59c9-4edf-a4ba-b4ce136f2df4-kube-api-access-pp8rx\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:32.536977 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:32.536975 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf735100-a839-46f3-9f12-4070a7ed7571-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:32.537143 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:32.537008 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b83e3502-59c9-4edf-a4ba-b4ce136f2df4-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:32.537143 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:32.537018 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzrgj\" (UniqueName: \"kubernetes.io/projected/cf735100-a839-46f3-9f12-4070a7ed7571-kube-api-access-vzrgj\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:33.649493 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:33.649458 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b83e3502-59c9-4edf-a4ba-b4ce136f2df4" path="/var/lib/kubelet/pods/b83e3502-59c9-4edf-a4ba-b4ce136f2df4/volumes" Oct 31 14:02:33.650040 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:33.650015 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf735100-a839-46f3-9f12-4070a7ed7571" path="/var/lib/kubelet/pods/cf735100-a839-46f3-9f12-4070a7ed7571/volumes" Oct 31 14:02:36.215863 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:36.215826 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:38.034419 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:38.034382 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ctnd7"] Oct 31 14:02:38.037366 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:38.037337 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ctnd7"] Oct 31 14:02:38.446743 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:38.446709 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:38.647567 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:38.647531 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:39.582531 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.582492 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl"] Oct 31 14:02:39.583021 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.582998 6841 cpu_manager.go:414] "containerMap: removing container" podUID="cf735100-a839-46f3-9f12-4070a7ed7571" containerName="payload-pre-processing" Oct 31 14:02:39.583021 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.583014 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf735100-a839-46f3-9f12-4070a7ed7571" containerName="payload-pre-processing" Oct 31 14:02:39.583137 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.583026 6841 cpu_manager.go:414] "containerMap: removing container" podUID="b83e3502-59c9-4edf-a4ba-b4ce136f2df4" containerName="payload-processing" Oct 31 14:02:39.583137 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.583032 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b83e3502-59c9-4edf-a4ba-b4ce136f2df4" containerName="payload-processing" Oct 31 14:02:39.583137 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.583040 6841 cpu_manager.go:414] "containerMap: removing container" podUID="47e20e00-07b7-47d6-bbdc-aa1595401671" containerName="maas-api" Oct 31 14:02:39.583137 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.583045 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47e20e00-07b7-47d6-bbdc-aa1595401671" containerName="maas-api" Oct 31 14:02:39.583137 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.583119 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="b83e3502-59c9-4edf-a4ba-b4ce136f2df4" containerName="payload-processing" Oct 31 14:02:39.583137 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.583127 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="47e20e00-07b7-47d6-bbdc-aa1595401671" containerName="maas-api" Oct 31 14:02:39.583365 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.583142 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf735100-a839-46f3-9f12-4070a7ed7571" containerName="payload-pre-processing" Oct 31 14:02:39.587869 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.587846 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl" Oct 31 14:02:39.598283 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.598251 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-5d3f86-serving-cert\"" Oct 31 14:02:39.598393 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.598303 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rhftr\"" Oct 31 14:02:39.598393 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.598331 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 14:02:39.598393 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.598340 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 14:02:39.598393 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.598386 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 14:02:39.601294 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.601271 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl"] Oct 31 14:02:39.610109 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.610083 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c446qtzs"] Oct 31 14:02:39.613816 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.613797 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c446qtzs" Oct 31 14:02:39.622478 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.622429 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c446qtzs"] Oct 31 14:02:39.643468 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.643425 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4867ee48-7c82-47ab-9e23-93f4e2f87ae3" path="/var/lib/kubelet/pods/4867ee48-7c82-47ab-9e23-93f4e2f87ae3/volumes" Oct 31 14:02:39.649384 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.649358 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6b2fh2d"] Oct 31 14:02:39.651162 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.651136 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:39.653571 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.653553 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6b2fh2d" Oct 31 14:02:39.666932 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.666906 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6b2fh2d"] Oct 31 14:02:39.694221 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.694186 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-fc5607b7-dockercfg-pfbzq\"" Oct 31 14:02:39.694808 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.694776 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-5d3f86\"" Oct 31 14:02:39.707297 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.707265 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2163451f-5e2d-42cb-9961-c4556f83a9b8-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl\" (UID: \"2163451f-5e2d-42cb-9961-c4556f83a9b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl" Oct 31 14:02:39.707428 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.707414 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2163451f-5e2d-42cb-9961-c4556f83a9b8-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl\" (UID: \"2163451f-5e2d-42cb-9961-c4556f83a9b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl" Oct 31 14:02:39.707531 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.707514 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2163451f-5e2d-42cb-9961-c4556f83a9b8-metrics-certs\") pod \"maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl\" (UID: \"2163451f-5e2d-42cb-9961-c4556f83a9b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl" Oct 31 14:02:39.707588 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.707570 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdn46\" (UniqueName: \"kubernetes.io/projected/80df07ad-8a17-4e04-b3cb-ec95cd327a62-kube-api-access-zdn46\") pod \"payload-processing-e2e-worker-single-main-5d3f86-695d6cc6b2fh2d\" (UID: \"80df07ad-8a17-4e04-b3cb-ec95cd327a62\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6b2fh2d" Oct 31 14:02:39.707635 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.707604 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxfqz\" (UniqueName: \"kubernetes.io/projected/6df3dbb6-a790-4bfb-ad50-b311812fbe85-kube-api-access-sxfqz\") pod \"payload-pre-processing-e2e-worker-single-main-5d3f86-59c446qtzs\" (UID: \"6df3dbb6-a790-4bfb-ad50-b311812fbe85\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c446qtzs" Oct 31 14:02:39.707717 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.707699 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80df07ad-8a17-4e04-b3cb-ec95cd327a62-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-5d3f86-695d6cc6b2fh2d\" (UID: \"80df07ad-8a17-4e04-b3cb-ec95cd327a62\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6b2fh2d" Oct 31 14:02:39.707757 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.707730 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df3dbb6-a790-4bfb-ad50-b311812fbe85-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-5d3f86-59c446qtzs\" (UID: \"6df3dbb6-a790-4bfb-ad50-b311812fbe85\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c446qtzs" Oct 31 14:02:39.707757 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.707749 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsmw5\" (UniqueName: \"kubernetes.io/projected/2163451f-5e2d-42cb-9961-c4556f83a9b8-kube-api-access-wsmw5\") pod \"maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl\" (UID: \"2163451f-5e2d-42cb-9961-c4556f83a9b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl" Oct 31 14:02:39.808975 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.808936 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2163451f-5e2d-42cb-9961-c4556f83a9b8-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl\" (UID: \"2163451f-5e2d-42cb-9961-c4556f83a9b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl" Oct 31 14:02:39.809198 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.809033 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2163451f-5e2d-42cb-9961-c4556f83a9b8-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl\" (UID: \"2163451f-5e2d-42cb-9961-c4556f83a9b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl" Oct 31 14:02:39.809198 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.809108 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2163451f-5e2d-42cb-9961-c4556f83a9b8-metrics-certs\") pod \"maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl\" (UID: \"2163451f-5e2d-42cb-9961-c4556f83a9b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl" Oct 31 14:02:39.809198 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.809152 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdn46\" (UniqueName: \"kubernetes.io/projected/80df07ad-8a17-4e04-b3cb-ec95cd327a62-kube-api-access-zdn46\") pod \"payload-processing-e2e-worker-single-main-5d3f86-695d6cc6b2fh2d\" (UID: \"80df07ad-8a17-4e04-b3cb-ec95cd327a62\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6b2fh2d" Oct 31 14:02:39.809198 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.809182 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxfqz\" (UniqueName: \"kubernetes.io/projected/6df3dbb6-a790-4bfb-ad50-b311812fbe85-kube-api-access-sxfqz\") pod \"payload-pre-processing-e2e-worker-single-main-5d3f86-59c446qtzs\" (UID: \"6df3dbb6-a790-4bfb-ad50-b311812fbe85\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c446qtzs" Oct 31 14:02:39.809457 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.809243 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80df07ad-8a17-4e04-b3cb-ec95cd327a62-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-5d3f86-695d6cc6b2fh2d\" (UID: \"80df07ad-8a17-4e04-b3cb-ec95cd327a62\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6b2fh2d" Oct 31 14:02:39.809457 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.809273 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df3dbb6-a790-4bfb-ad50-b311812fbe85-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-5d3f86-59c446qtzs\" (UID: \"6df3dbb6-a790-4bfb-ad50-b311812fbe85\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c446qtzs" Oct 31 14:02:39.809457 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.809302 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsmw5\" (UniqueName: \"kubernetes.io/projected/2163451f-5e2d-42cb-9961-c4556f83a9b8-kube-api-access-wsmw5\") pod \"maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl\" (UID: \"2163451f-5e2d-42cb-9961-c4556f83a9b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl" Oct 31 14:02:39.809833 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.809808 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2163451f-5e2d-42cb-9961-c4556f83a9b8-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl\" (UID: \"2163451f-5e2d-42cb-9961-c4556f83a9b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl" Oct 31 14:02:39.810040 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.810018 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80df07ad-8a17-4e04-b3cb-ec95cd327a62-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-5d3f86-695d6cc6b2fh2d\" (UID: \"80df07ad-8a17-4e04-b3cb-ec95cd327a62\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6b2fh2d" Oct 31 14:02:39.810139 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.810079 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df3dbb6-a790-4bfb-ad50-b311812fbe85-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-5d3f86-59c446qtzs\" (UID: \"6df3dbb6-a790-4bfb-ad50-b311812fbe85\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c446qtzs" Oct 31 14:02:39.811696 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.811676 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2163451f-5e2d-42cb-9961-c4556f83a9b8-metrics-certs\") pod \"maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl\" (UID: \"2163451f-5e2d-42cb-9961-c4556f83a9b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl" Oct 31 14:02:39.811788 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.811717 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2163451f-5e2d-42cb-9961-c4556f83a9b8-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl\" (UID: \"2163451f-5e2d-42cb-9961-c4556f83a9b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl" Oct 31 14:02:39.828092 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.828063 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdn46\" (UniqueName: \"kubernetes.io/projected/80df07ad-8a17-4e04-b3cb-ec95cd327a62-kube-api-access-zdn46\") pod \"payload-processing-e2e-worker-single-main-5d3f86-695d6cc6b2fh2d\" (UID: \"80df07ad-8a17-4e04-b3cb-ec95cd327a62\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6b2fh2d" Oct 31 14:02:39.828468 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.828450 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxfqz\" (UniqueName: \"kubernetes.io/projected/6df3dbb6-a790-4bfb-ad50-b311812fbe85-kube-api-access-sxfqz\") pod \"payload-pre-processing-e2e-worker-single-main-5d3f86-59c446qtzs\" (UID: \"6df3dbb6-a790-4bfb-ad50-b311812fbe85\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c446qtzs" Oct 31 14:02:39.828516 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.828498 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsmw5\" (UniqueName: \"kubernetes.io/projected/2163451f-5e2d-42cb-9961-c4556f83a9b8-kube-api-access-wsmw5\") pod \"maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl\" (UID: \"2163451f-5e2d-42cb-9961-c4556f83a9b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl" Oct 31 14:02:39.860436 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.860369 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:39.906566 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.906540 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl" Oct 31 14:02:39.932185 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.932160 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c446qtzs" Oct 31 14:02:39.972615 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:39.971841 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6b2fh2d" Oct 31 14:02:40.058111 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:40.058035 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:40.229116 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:40.229085 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl"] Oct 31 14:02:40.230617 ip-10-0-138-70 kubenswrapper[6841]: W1031 14:02:40.230592 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2163451f_5e2d_42cb_9961_c4556f83a9b8.slice/crio-84db8d223eb577d7d9abd0291cfdc2266f21076cb738ddc5a0cc3a1234a0a50e WatchSource:0}: Error finding container 84db8d223eb577d7d9abd0291cfdc2266f21076cb738ddc5a0cc3a1234a0a50e: Status 404 returned error can't find the container with id 84db8d223eb577d7d9abd0291cfdc2266f21076cb738ddc5a0cc3a1234a0a50e Oct 31 14:02:40.246916 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:40.246889 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c446qtzs"] Oct 31 14:02:40.247348 ip-10-0-138-70 kubenswrapper[6841]: W1031 14:02:40.247320 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df3dbb6_a790_4bfb_ad50_b311812fbe85.slice/crio-efb21d1dfa02634d3f146868192c862fdc269e7be207f6c84b19ba2aed7787f2 WatchSource:0}: Error finding container efb21d1dfa02634d3f146868192c862fdc269e7be207f6c84b19ba2aed7787f2: Status 404 returned error can't find the container with id efb21d1dfa02634d3f146868192c862fdc269e7be207f6c84b19ba2aed7787f2 Oct 31 14:02:40.283176 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:40.282294 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6b2fh2d"] Oct 31 14:02:40.287309 ip-10-0-138-70 kubenswrapper[6841]: W1031 14:02:40.287280 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80df07ad_8a17_4e04_b3cb_ec95cd327a62.slice/crio-f5b73cc96b9051df1ac757b19b4ef7212b851799cbb0ff0e1755d2f7b7f66a69 WatchSource:0}: Error finding container f5b73cc96b9051df1ac757b19b4ef7212b851799cbb0ff0e1755d2f7b7f66a69: Status 404 returned error can't find the container with id f5b73cc96b9051df1ac757b19b4ef7212b851799cbb0ff0e1755d2f7b7f66a69 Oct 31 14:02:40.494966 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:40.494879 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6b2fh2d" event={"ID":"80df07ad-8a17-4e04-b3cb-ec95cd327a62","Type":"ContainerStarted","Data":"0838d792ec1e48ffbb7652d827ea994b77198460076d66bcd9c36c732afb8a6e"} Oct 31 14:02:40.494966 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:40.494939 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6b2fh2d" event={"ID":"80df07ad-8a17-4e04-b3cb-ec95cd327a62","Type":"ContainerStarted","Data":"f5b73cc96b9051df1ac757b19b4ef7212b851799cbb0ff0e1755d2f7b7f66a69"} Oct 31 14:02:40.495241 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:40.495012 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6b2fh2d" Oct 31 14:02:40.496728 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:40.496703 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c446qtzs" event={"ID":"6df3dbb6-a790-4bfb-ad50-b311812fbe85","Type":"ContainerStarted","Data":"c8cb8ccf0b50052d5816046df93c4e8663468eac5af06f68f1c6684e0707bd21"} Oct 31 14:02:40.496828 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:40.496749 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c446qtzs" Oct 31 14:02:40.496828 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:40.496773 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c446qtzs" event={"ID":"6df3dbb6-a790-4bfb-ad50-b311812fbe85","Type":"ContainerStarted","Data":"efb21d1dfa02634d3f146868192c862fdc269e7be207f6c84b19ba2aed7787f2"} Oct 31 14:02:40.498026 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:40.498002 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl" event={"ID":"2163451f-5e2d-42cb-9961-c4556f83a9b8","Type":"ContainerStarted","Data":"84db8d223eb577d7d9abd0291cfdc2266f21076cb738ddc5a0cc3a1234a0a50e"} Oct 31 14:02:40.516515 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:40.516474 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6b2fh2d" podStartSLOduration=1.5164623160000001 podStartE2EDuration="1.516462316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:02:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:02:40.512846517 +0000 UTC m=+5406.995969419" watchObservedRunningTime="2026-10-31 14:02:40.516462316 +0000 UTC m=+5406.999585216" Oct 31 14:02:40.579318 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:40.579280 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:41.369438 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:41.369405 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:41.506344 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:41.506304 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl" event={"ID":"2163451f-5e2d-42cb-9961-c4556f83a9b8","Type":"ContainerStarted","Data":"fddd9d3cd9245a2d6f5e8ba4442b0c8126522737194681787f76d1cd959ad04b"} Oct 31 14:02:41.506535 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:41.506375 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl" Oct 31 14:02:41.527509 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:41.527458 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl" podStartSLOduration=2.045697197 podStartE2EDuration="2.527443204s" totalImagesPullingTime="481.746007ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:02:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:02:41.52626964 +0000 UTC m=+5408.009392542" watchObservedRunningTime="2026-10-31 14:02:41.527443204 +0000 UTC m=+5408.010566101" Oct 31 14:02:41.528824 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:41.528791 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c446qtzs" podStartSLOduration=2.528780555 podStartE2EDuration="2.528780555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:02:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:02:40.538546097 +0000 UTC m=+5407.021668998" watchObservedRunningTime="2026-10-31 14:02:41.528780555 +0000 UTC m=+5408.011903503" Oct 31 14:02:42.076677 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:42.076586 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:42.273675 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:42.273630 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:42.966949 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:42.966909 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:43.438526 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:43.438437 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:43.768430 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:43.768392 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:44.343618 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:44.343574 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:44.572372 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:44.572336 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:45.472789 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:45.472757 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:46.361054 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:46.361012 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:47.075553 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:47.075516 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:47.524855 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:47.524825 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl" Oct 31 14:02:47.789356 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:47.789267 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:48.023626 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:48.023592 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4rjgt"] Oct 31 14:02:48.025991 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:48.025958 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4rjgt"] Oct 31 14:02:48.581385 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:48.581348 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:49.294115 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:49.294078 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:49.641329 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:49.641251 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7f0fe4e-47a4-43f3-a65b-e69b97144544" path="/var/lib/kubelet/pods/c7f0fe4e-47a4-43f3-a65b-e69b97144544/volumes" Oct 31 14:02:50.039047 ip-10-0-138-70 kubenswrapper[6841]: E1031 14:02:50.039011 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 14:02:50.077062 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:50.077029 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:50.277748 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:50.277712 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:50.478831 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:50.478793 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:50.876928 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:50.876835 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:51.509975 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:51.509939 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6b2fh2d" Oct 31 14:02:51.510211 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:51.510044 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c446qtzs" Oct 31 14:02:51.594244 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:51.594203 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:52.283444 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:52.283411 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:52.976728 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:52.976694 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:53.677163 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:53.677131 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:54.387041 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:54.386954 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:55.084173 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:55.084143 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:55.783353 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:55.783316 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:56.484704 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:56.484667 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:57.242062 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:57.242017 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:57.970050 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:57.970001 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:58.364041 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:58.363948 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6b2fh2d"] Oct 31 14:02:58.382512 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:58.382470 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c446qtzs"] Oct 31 14:02:58.538070 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:58.538031 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:58.735492 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:58.735458 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:59.102835 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:59.102755 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl"] Oct 31 14:02:59.103295 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:59.103035 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl" podUID="2163451f-5e2d-42cb-9961-c4556f83a9b8" containerName="maas-api" containerID="cri-o://fddd9d3cd9245a2d6f5e8ba4442b0c8126522737194681787f76d1cd959ad04b" gracePeriod=30 Oct 31 14:02:59.252778 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:59.252745 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:59.403133 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:59.403105 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl" Oct 31 14:02:59.455453 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:59.455413 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:02:59.523024 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:59.522950 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2163451f-5e2d-42cb-9961-c4556f83a9b8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2163451f-5e2d-42cb-9961-c4556f83a9b8-openshift-service-ca\") pod \"2163451f-5e2d-42cb-9961-c4556f83a9b8\" (UID: \"2163451f-5e2d-42cb-9961-c4556f83a9b8\") " Oct 31 14:02:59.523229 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:59.523078 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2163451f-5e2d-42cb-9961-c4556f83a9b8-kube-api-access-wsmw5\" (UniqueName: \"kubernetes.io/projected/2163451f-5e2d-42cb-9961-c4556f83a9b8-kube-api-access-wsmw5\") pod \"2163451f-5e2d-42cb-9961-c4556f83a9b8\" (UID: \"2163451f-5e2d-42cb-9961-c4556f83a9b8\") " Oct 31 14:02:59.523229 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:59.523112 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2163451f-5e2d-42cb-9961-c4556f83a9b8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2163451f-5e2d-42cb-9961-c4556f83a9b8-maas-api-tls\") pod \"2163451f-5e2d-42cb-9961-c4556f83a9b8\" (UID: \"2163451f-5e2d-42cb-9961-c4556f83a9b8\") " Oct 31 14:02:59.523229 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:59.523177 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2163451f-5e2d-42cb-9961-c4556f83a9b8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2163451f-5e2d-42cb-9961-c4556f83a9b8-metrics-certs\") pod \"2163451f-5e2d-42cb-9961-c4556f83a9b8\" (UID: \"2163451f-5e2d-42cb-9961-c4556f83a9b8\") " Oct 31 14:02:59.523425 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:59.523317 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2163451f-5e2d-42cb-9961-c4556f83a9b8-openshift-service-ca" pod "2163451f-5e2d-42cb-9961-c4556f83a9b8" (UID: "2163451f-5e2d-42cb-9961-c4556f83a9b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:02:59.523532 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:59.523506 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2163451f-5e2d-42cb-9961-c4556f83a9b8-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:59.525345 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:59.525316 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2163451f-5e2d-42cb-9961-c4556f83a9b8-kube-api-access-wsmw5" pod "2163451f-5e2d-42cb-9961-c4556f83a9b8" (UID: "2163451f-5e2d-42cb-9961-c4556f83a9b8"). InnerVolumeSpecName "kube-api-access-wsmw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:02:59.525345 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:59.525331 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2163451f-5e2d-42cb-9961-c4556f83a9b8-metrics-certs" pod "2163451f-5e2d-42cb-9961-c4556f83a9b8" (UID: "2163451f-5e2d-42cb-9961-c4556f83a9b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:02:59.525465 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:59.525321 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2163451f-5e2d-42cb-9961-c4556f83a9b8-maas-api-tls" pod "2163451f-5e2d-42cb-9961-c4556f83a9b8" (UID: "2163451f-5e2d-42cb-9961-c4556f83a9b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:02:59.624860 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:59.624767 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsmw5\" (UniqueName: \"kubernetes.io/projected/2163451f-5e2d-42cb-9961-c4556f83a9b8-kube-api-access-wsmw5\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:59.624860 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:59.624814 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2163451f-5e2d-42cb-9961-c4556f83a9b8-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:59.624860 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:59.624831 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2163451f-5e2d-42cb-9961-c4556f83a9b8-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:59.643378 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:59.643339 6841 generic.go:356] "Generic (PLEG): container finished" podID="2163451f-5e2d-42cb-9961-c4556f83a9b8" containerID="fddd9d3cd9245a2d6f5e8ba4442b0c8126522737194681787f76d1cd959ad04b" exitCode=0 Oct 31 14:02:59.643564 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:59.643387 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl" Oct 31 14:02:59.643564 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:59.643391 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl" event={"ID":"2163451f-5e2d-42cb-9961-c4556f83a9b8","Type":"ContainerDied","Data":"fddd9d3cd9245a2d6f5e8ba4442b0c8126522737194681787f76d1cd959ad04b"} Oct 31 14:02:59.643564 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:59.643455 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl" event={"ID":"2163451f-5e2d-42cb-9961-c4556f83a9b8","Type":"ContainerDied","Data":"84db8d223eb577d7d9abd0291cfdc2266f21076cb738ddc5a0cc3a1234a0a50e"} Oct 31 14:02:59.643564 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:59.643486 6841 scope.go:122] "RemoveContainer" containerID="fddd9d3cd9245a2d6f5e8ba4442b0c8126522737194681787f76d1cd959ad04b" Oct 31 14:02:59.657109 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:59.657085 6841 scope.go:122] "RemoveContainer" containerID="fddd9d3cd9245a2d6f5e8ba4442b0c8126522737194681787f76d1cd959ad04b" Oct 31 14:02:59.657425 ip-10-0-138-70 kubenswrapper[6841]: E1031 14:02:59.657405 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fddd9d3cd9245a2d6f5e8ba4442b0c8126522737194681787f76d1cd959ad04b\": container with ID starting with fddd9d3cd9245a2d6f5e8ba4442b0c8126522737194681787f76d1cd959ad04b not found: ID does not exist" containerID="fddd9d3cd9245a2d6f5e8ba4442b0c8126522737194681787f76d1cd959ad04b" Oct 31 14:02:59.657493 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:59.657435 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fddd9d3cd9245a2d6f5e8ba4442b0c8126522737194681787f76d1cd959ad04b"} err="failed to get container status \"fddd9d3cd9245a2d6f5e8ba4442b0c8126522737194681787f76d1cd959ad04b\": rpc error: code = NotFound desc = could not find container \"fddd9d3cd9245a2d6f5e8ba4442b0c8126522737194681787f76d1cd959ad04b\": container with ID starting with fddd9d3cd9245a2d6f5e8ba4442b0c8126522737194681787f76d1cd959ad04b not found: ID does not exist" Oct 31 14:02:59.669251 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:59.669230 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl"] Oct 31 14:02:59.675734 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:59.675710 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5d3f86-795654d49b-n85bl"] Oct 31 14:02:59.716550 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:59.716517 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw"] Oct 31 14:02:59.717023 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:59.716960 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" podUID="ab81877e-88af-4508-82b3-7ee4e9a7e021" containerName="istio-proxy" containerID="cri-o://94c8f4fdf9489ba4b00f9daa0dbbd307ada07f75c5cb0a0d5de329eb7460b756" gracePeriod=30 Oct 31 14:02:59.857081 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:02:59.857048 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:03:01.062329 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:01.062295 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd8nh"] Oct 31 14:03:01.067070 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:01.067040 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd8nh"] Oct 31 14:03:01.088922 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:01.088898 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nlztb"] Oct 31 14:03:01.089373 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:01.089360 6841 cpu_manager.go:414] "containerMap: removing container" podUID="2163451f-5e2d-42cb-9961-c4556f83a9b8" containerName="maas-api" Oct 31 14:03:01.089432 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:01.089375 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2163451f-5e2d-42cb-9961-c4556f83a9b8" containerName="maas-api" Oct 31 14:03:01.089475 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:01.089458 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="2163451f-5e2d-42cb-9961-c4556f83a9b8" containerName="maas-api" Oct 31 14:03:01.094033 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:01.094014 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nlztb" Oct 31 14:03:01.099198 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:01.099074 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:03:01.099198 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:01.099127 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 14:03:01.099198 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:01.099165 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 14:03:01.099436 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:01.099237 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 14:03:01.102730 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:01.102707 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nlztb"] Oct 31 14:03:01.148444 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:01.148411 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bb799f2-3c7f-47eb-9bad-1fbdeb70e452-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nlztb\" (UID: \"1bb799f2-3c7f-47eb-9bad-1fbdeb70e452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nlztb" Oct 31 14:03:01.148608 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:01.148486 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q784\" (UniqueName: \"kubernetes.io/projected/1bb799f2-3c7f-47eb-9bad-1fbdeb70e452-kube-api-access-4q784\") pod \"maas-api-revoke-sub-3d4156f1-nlztb\" (UID: \"1bb799f2-3c7f-47eb-9bad-1fbdeb70e452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nlztb" Oct 31 14:03:01.148674 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:01.148646 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bb799f2-3c7f-47eb-9bad-1fbdeb70e452-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nlztb\" (UID: \"1bb799f2-3c7f-47eb-9bad-1fbdeb70e452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nlztb" Oct 31 14:03:01.249591 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:01.249553 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bb799f2-3c7f-47eb-9bad-1fbdeb70e452-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nlztb\" (UID: \"1bb799f2-3c7f-47eb-9bad-1fbdeb70e452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nlztb" Oct 31 14:03:01.249765 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:01.249617 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q784\" (UniqueName: \"kubernetes.io/projected/1bb799f2-3c7f-47eb-9bad-1fbdeb70e452-kube-api-access-4q784\") pod \"maas-api-revoke-sub-3d4156f1-nlztb\" (UID: \"1bb799f2-3c7f-47eb-9bad-1fbdeb70e452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nlztb" Oct 31 14:03:01.249765 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:01.249690 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bb799f2-3c7f-47eb-9bad-1fbdeb70e452-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nlztb\" (UID: \"1bb799f2-3c7f-47eb-9bad-1fbdeb70e452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nlztb" Oct 31 14:03:01.250363 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:01.250325 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bb799f2-3c7f-47eb-9bad-1fbdeb70e452-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nlztb\" (UID: \"1bb799f2-3c7f-47eb-9bad-1fbdeb70e452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nlztb" Oct 31 14:03:01.252164 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:01.252143 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bb799f2-3c7f-47eb-9bad-1fbdeb70e452-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nlztb\" (UID: \"1bb799f2-3c7f-47eb-9bad-1fbdeb70e452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nlztb" Oct 31 14:03:01.262185 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:01.262165 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q784\" (UniqueName: \"kubernetes.io/projected/1bb799f2-3c7f-47eb-9bad-1fbdeb70e452-kube-api-access-4q784\") pod \"maas-api-revoke-sub-3d4156f1-nlztb\" (UID: \"1bb799f2-3c7f-47eb-9bad-1fbdeb70e452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nlztb" Oct 31 14:03:01.410206 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:01.410125 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nlztb" Oct 31 14:03:01.641703 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:01.641670 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2163451f-5e2d-42cb-9961-c4556f83a9b8" path="/var/lib/kubelet/pods/2163451f-5e2d-42cb-9961-c4556f83a9b8/volumes" Oct 31 14:03:01.642139 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:01.642125 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cfa8aff-e9c5-464f-abe3-a5da3a967233" path="/var/lib/kubelet/pods/8cfa8aff-e9c5-464f-abe3-a5da3a967233/volumes" Oct 31 14:03:01.685538 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:01.685512 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nlztb"] Oct 31 14:03:01.687904 ip-10-0-138-70 kubenswrapper[6841]: W1031 14:03:01.687868 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb799f2_3c7f_47eb_9bad_1fbdeb70e452.slice/crio-ff56b5d0055df2ac85a71c2f5d5dc8f705a67a1854c4969249f8a75c5597692a WatchSource:0}: Error finding container ff56b5d0055df2ac85a71c2f5d5dc8f705a67a1854c4969249f8a75c5597692a: Status 404 returned error can't find the container with id ff56b5d0055df2ac85a71c2f5d5dc8f705a67a1854c4969249f8a75c5597692a Oct 31 14:03:02.664783 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:02.664751 6841 generic.go:356] "Generic (PLEG): container finished" podID="1bb799f2-3c7f-47eb-9bad-1fbdeb70e452" containerID="b9ef20a4246f48bed2eb76247fa799a9f2df05c01b25ab1c931e5db6aad6cd83" exitCode=0 Oct 31 14:03:02.665221 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:02.664829 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nlztb" event={"ID":"1bb799f2-3c7f-47eb-9bad-1fbdeb70e452","Type":"ContainerDied","Data":"b9ef20a4246f48bed2eb76247fa799a9f2df05c01b25ab1c931e5db6aad6cd83"} Oct 31 14:03:02.665221 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:02.664887 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nlztb" event={"ID":"1bb799f2-3c7f-47eb-9bad-1fbdeb70e452","Type":"ContainerStarted","Data":"ff56b5d0055df2ac85a71c2f5d5dc8f705a67a1854c4969249f8a75c5597692a"} Oct 31 14:03:03.364613 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.364572 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6b2fh2d" podUID="80df07ad-8a17-4e04-b3cb-ec95cd327a62" containerName="payload-processing" containerID="cri-o://0838d792ec1e48ffbb7652d827ea994b77198460076d66bcd9c36c732afb8a6e" gracePeriod=25 Oct 31 14:03:03.383251 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.383219 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c446qtzs" podUID="6df3dbb6-a790-4bfb-ad50-b311812fbe85" containerName="payload-pre-processing" containerID="cri-o://c8cb8ccf0b50052d5816046df93c4e8663468eac5af06f68f1c6684e0707bd21" gracePeriod=25 Oct 31 14:03:03.639059 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.639030 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6b2fh2d" Oct 31 14:03:03.665402 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.665373 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c446qtzs" Oct 31 14:03:03.669322 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.669288 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80df07ad-8a17-4e04-b3cb-ec95cd327a62-kube-api-access-zdn46\" (UniqueName: \"kubernetes.io/projected/80df07ad-8a17-4e04-b3cb-ec95cd327a62-kube-api-access-zdn46\") pod \"80df07ad-8a17-4e04-b3cb-ec95cd327a62\" (UID: \"80df07ad-8a17-4e04-b3cb-ec95cd327a62\") " Oct 31 14:03:03.669450 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.669369 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6df3dbb6-a790-4bfb-ad50-b311812fbe85-kube-api-access-sxfqz\" (UniqueName: \"kubernetes.io/projected/6df3dbb6-a790-4bfb-ad50-b311812fbe85-kube-api-access-sxfqz\") pod \"6df3dbb6-a790-4bfb-ad50-b311812fbe85\" (UID: \"6df3dbb6-a790-4bfb-ad50-b311812fbe85\") " Oct 31 14:03:03.669450 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.669436 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80df07ad-8a17-4e04-b3cb-ec95cd327a62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80df07ad-8a17-4e04-b3cb-ec95cd327a62-plugins-config-volume\") pod \"80df07ad-8a17-4e04-b3cb-ec95cd327a62\" (UID: \"80df07ad-8a17-4e04-b3cb-ec95cd327a62\") " Oct 31 14:03:03.669580 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.669501 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6df3dbb6-a790-4bfb-ad50-b311812fbe85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df3dbb6-a790-4bfb-ad50-b311812fbe85-plugins-config-volume\") pod \"6df3dbb6-a790-4bfb-ad50-b311812fbe85\" (UID: \"6df3dbb6-a790-4bfb-ad50-b311812fbe85\") " Oct 31 14:03:03.670109 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.670080 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df3dbb6-a790-4bfb-ad50-b311812fbe85-plugins-config-volume" pod "6df3dbb6-a790-4bfb-ad50-b311812fbe85" (UID: "6df3dbb6-a790-4bfb-ad50-b311812fbe85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:03:03.670308 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.670282 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80df07ad-8a17-4e04-b3cb-ec95cd327a62-plugins-config-volume" pod "80df07ad-8a17-4e04-b3cb-ec95cd327a62" (UID: "80df07ad-8a17-4e04-b3cb-ec95cd327a62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:03:03.671856 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.671824 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df3dbb6-a790-4bfb-ad50-b311812fbe85-kube-api-access-sxfqz" pod "6df3dbb6-a790-4bfb-ad50-b311812fbe85" (UID: "6df3dbb6-a790-4bfb-ad50-b311812fbe85"). InnerVolumeSpecName "kube-api-access-sxfqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:03:03.671945 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.671916 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80df07ad-8a17-4e04-b3cb-ec95cd327a62-kube-api-access-zdn46" pod "80df07ad-8a17-4e04-b3cb-ec95cd327a62" (UID: "80df07ad-8a17-4e04-b3cb-ec95cd327a62"). InnerVolumeSpecName "kube-api-access-zdn46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:03:03.674571 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.674544 6841 generic.go:356] "Generic (PLEG): container finished" podID="80df07ad-8a17-4e04-b3cb-ec95cd327a62" containerID="0838d792ec1e48ffbb7652d827ea994b77198460076d66bcd9c36c732afb8a6e" exitCode=0 Oct 31 14:03:03.674676 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.674582 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6b2fh2d" Oct 31 14:03:03.674676 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.674590 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6b2fh2d" event={"ID":"80df07ad-8a17-4e04-b3cb-ec95cd327a62","Type":"ContainerDied","Data":"0838d792ec1e48ffbb7652d827ea994b77198460076d66bcd9c36c732afb8a6e"} Oct 31 14:03:03.674676 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.674651 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6b2fh2d" event={"ID":"80df07ad-8a17-4e04-b3cb-ec95cd327a62","Type":"ContainerDied","Data":"f5b73cc96b9051df1ac757b19b4ef7212b851799cbb0ff0e1755d2f7b7f66a69"} Oct 31 14:03:03.674857 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.674680 6841 scope.go:122] "RemoveContainer" containerID="0838d792ec1e48ffbb7652d827ea994b77198460076d66bcd9c36c732afb8a6e" Oct 31 14:03:03.676383 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.676358 6841 generic.go:356] "Generic (PLEG): container finished" podID="6df3dbb6-a790-4bfb-ad50-b311812fbe85" containerID="c8cb8ccf0b50052d5816046df93c4e8663468eac5af06f68f1c6684e0707bd21" exitCode=0 Oct 31 14:03:03.676512 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.676410 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c446qtzs" Oct 31 14:03:03.676512 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.676445 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c446qtzs" event={"ID":"6df3dbb6-a790-4bfb-ad50-b311812fbe85","Type":"ContainerDied","Data":"c8cb8ccf0b50052d5816046df93c4e8663468eac5af06f68f1c6684e0707bd21"} Oct 31 14:03:03.676654 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.676517 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c446qtzs" event={"ID":"6df3dbb6-a790-4bfb-ad50-b311812fbe85","Type":"ContainerDied","Data":"efb21d1dfa02634d3f146868192c862fdc269e7be207f6c84b19ba2aed7787f2"} Oct 31 14:03:03.691186 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.691165 6841 scope.go:122] "RemoveContainer" containerID="0838d792ec1e48ffbb7652d827ea994b77198460076d66bcd9c36c732afb8a6e" Oct 31 14:03:03.692686 ip-10-0-138-70 kubenswrapper[6841]: E1031 14:03:03.692622 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0838d792ec1e48ffbb7652d827ea994b77198460076d66bcd9c36c732afb8a6e\": container with ID starting with 0838d792ec1e48ffbb7652d827ea994b77198460076d66bcd9c36c732afb8a6e not found: ID does not exist" containerID="0838d792ec1e48ffbb7652d827ea994b77198460076d66bcd9c36c732afb8a6e" Oct 31 14:03:03.692802 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.692696 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0838d792ec1e48ffbb7652d827ea994b77198460076d66bcd9c36c732afb8a6e"} err="failed to get container status \"0838d792ec1e48ffbb7652d827ea994b77198460076d66bcd9c36c732afb8a6e\": rpc error: code = NotFound desc = could not find container \"0838d792ec1e48ffbb7652d827ea994b77198460076d66bcd9c36c732afb8a6e\": container with ID starting with 0838d792ec1e48ffbb7652d827ea994b77198460076d66bcd9c36c732afb8a6e not found: ID does not exist" Oct 31 14:03:03.692802 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.692792 6841 scope.go:122] "RemoveContainer" containerID="c8cb8ccf0b50052d5816046df93c4e8663468eac5af06f68f1c6684e0707bd21" Oct 31 14:03:03.706013 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.705975 6841 scope.go:122] "RemoveContainer" containerID="c8cb8ccf0b50052d5816046df93c4e8663468eac5af06f68f1c6684e0707bd21" Oct 31 14:03:03.706463 ip-10-0-138-70 kubenswrapper[6841]: E1031 14:03:03.706429 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8cb8ccf0b50052d5816046df93c4e8663468eac5af06f68f1c6684e0707bd21\": container with ID starting with c8cb8ccf0b50052d5816046df93c4e8663468eac5af06f68f1c6684e0707bd21 not found: ID does not exist" containerID="c8cb8ccf0b50052d5816046df93c4e8663468eac5af06f68f1c6684e0707bd21" Oct 31 14:03:03.706571 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.706475 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8cb8ccf0b50052d5816046df93c4e8663468eac5af06f68f1c6684e0707bd21"} err="failed to get container status \"c8cb8ccf0b50052d5816046df93c4e8663468eac5af06f68f1c6684e0707bd21\": rpc error: code = NotFound desc = could not find container \"c8cb8ccf0b50052d5816046df93c4e8663468eac5af06f68f1c6684e0707bd21\": container with ID starting with c8cb8ccf0b50052d5816046df93c4e8663468eac5af06f68f1c6684e0707bd21 not found: ID does not exist" Oct 31 14:03:03.713865 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.713838 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6b2fh2d"] Oct 31 14:03:03.717649 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.717626 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5d3f86-695d6cc6b2fh2d"] Oct 31 14:03:03.729144 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.729121 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c446qtzs"] Oct 31 14:03:03.735365 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.735342 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5d3f86-59c446qtzs"] Oct 31 14:03:03.770667 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.770639 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdn46\" (UniqueName: \"kubernetes.io/projected/80df07ad-8a17-4e04-b3cb-ec95cd327a62-kube-api-access-zdn46\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:03.770667 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.770665 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxfqz\" (UniqueName: \"kubernetes.io/projected/6df3dbb6-a790-4bfb-ad50-b311812fbe85-kube-api-access-sxfqz\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:03.770822 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.770681 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80df07ad-8a17-4e04-b3cb-ec95cd327a62-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:03.770822 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.770692 6841 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df3dbb6-a790-4bfb-ad50-b311812fbe85-plugins-config-volume\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:03.820150 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.820127 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nlztb" Oct 31 14:03:03.871362 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.871290 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bb799f2-3c7f-47eb-9bad-1fbdeb70e452-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bb799f2-3c7f-47eb-9bad-1fbdeb70e452-maas-api-openshift-ca\") pod \"1bb799f2-3c7f-47eb-9bad-1fbdeb70e452\" (UID: \"1bb799f2-3c7f-47eb-9bad-1fbdeb70e452\") " Oct 31 14:03:03.871553 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.871363 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bb799f2-3c7f-47eb-9bad-1fbdeb70e452-kube-api-access-4q784\" (UniqueName: \"kubernetes.io/projected/1bb799f2-3c7f-47eb-9bad-1fbdeb70e452-kube-api-access-4q784\") pod \"1bb799f2-3c7f-47eb-9bad-1fbdeb70e452\" (UID: \"1bb799f2-3c7f-47eb-9bad-1fbdeb70e452\") " Oct 31 14:03:03.871553 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.871409 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bb799f2-3c7f-47eb-9bad-1fbdeb70e452-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bb799f2-3c7f-47eb-9bad-1fbdeb70e452-maas-api-xks-ca\") pod \"1bb799f2-3c7f-47eb-9bad-1fbdeb70e452\" (UID: \"1bb799f2-3c7f-47eb-9bad-1fbdeb70e452\") " Oct 31 14:03:03.871704 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.871643 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bb799f2-3c7f-47eb-9bad-1fbdeb70e452-maas-api-openshift-ca" pod "1bb799f2-3c7f-47eb-9bad-1fbdeb70e452" (UID: "1bb799f2-3c7f-47eb-9bad-1fbdeb70e452"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:03:03.871821 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.871805 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bb799f2-3c7f-47eb-9bad-1fbdeb70e452-maas-api-openshift-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:03.873406 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.873377 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bb799f2-3c7f-47eb-9bad-1fbdeb70e452-maas-api-xks-ca" pod "1bb799f2-3c7f-47eb-9bad-1fbdeb70e452" (UID: "1bb799f2-3c7f-47eb-9bad-1fbdeb70e452"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:03:03.873518 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.873448 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bb799f2-3c7f-47eb-9bad-1fbdeb70e452-kube-api-access-4q784" pod "1bb799f2-3c7f-47eb-9bad-1fbdeb70e452" (UID: "1bb799f2-3c7f-47eb-9bad-1fbdeb70e452"). InnerVolumeSpecName "kube-api-access-4q784". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:03:03.973129 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.973095 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q784\" (UniqueName: \"kubernetes.io/projected/1bb799f2-3c7f-47eb-9bad-1fbdeb70e452-kube-api-access-4q784\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:03.973129 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:03.973131 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bb799f2-3c7f-47eb-9bad-1fbdeb70e452-maas-api-xks-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:04.683744 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:04.683660 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nlztb" Oct 31 14:03:04.683744 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:04.683689 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nlztb" event={"ID":"1bb799f2-3c7f-47eb-9bad-1fbdeb70e452","Type":"ContainerDied","Data":"ff56b5d0055df2ac85a71c2f5d5dc8f705a67a1854c4969249f8a75c5597692a"} Oct 31 14:03:04.683744 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:04.683743 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff56b5d0055df2ac85a71c2f5d5dc8f705a67a1854c4969249f8a75c5597692a" Oct 31 14:03:04.988591 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:04.988568 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:03:05.083595 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.083563 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab81877e-88af-4508-82b3-7ee4e9a7e021-kube-api-access-ttgpn\" (UniqueName: \"kubernetes.io/projected/ab81877e-88af-4508-82b3-7ee4e9a7e021-kube-api-access-ttgpn\") pod \"ab81877e-88af-4508-82b3-7ee4e9a7e021\" (UID: \"ab81877e-88af-4508-82b3-7ee4e9a7e021\") " Oct 31 14:03:05.083799 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.083612 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab81877e-88af-4508-82b3-7ee4e9a7e021-istio-token\" (UniqueName: \"kubernetes.io/projected/ab81877e-88af-4508-82b3-7ee4e9a7e021-istio-token\") pod \"ab81877e-88af-4508-82b3-7ee4e9a7e021\" (UID: \"ab81877e-88af-4508-82b3-7ee4e9a7e021\") " Oct 31 14:03:05.083799 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.083649 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab81877e-88af-4508-82b3-7ee4e9a7e021-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab81877e-88af-4508-82b3-7ee4e9a7e021-workload-socket\") pod \"ab81877e-88af-4508-82b3-7ee4e9a7e021\" (UID: \"ab81877e-88af-4508-82b3-7ee4e9a7e021\") " Oct 31 14:03:05.083799 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.083681 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab81877e-88af-4508-82b3-7ee4e9a7e021-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab81877e-88af-4508-82b3-7ee4e9a7e021-istiod-ca-cert\") pod \"ab81877e-88af-4508-82b3-7ee4e9a7e021\" (UID: \"ab81877e-88af-4508-82b3-7ee4e9a7e021\") " Oct 31 14:03:05.083799 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.083705 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab81877e-88af-4508-82b3-7ee4e9a7e021-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab81877e-88af-4508-82b3-7ee4e9a7e021-istio-data\") pod \"ab81877e-88af-4508-82b3-7ee4e9a7e021\" (UID: \"ab81877e-88af-4508-82b3-7ee4e9a7e021\") " Oct 31 14:03:05.083799 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.083725 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab81877e-88af-4508-82b3-7ee4e9a7e021-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab81877e-88af-4508-82b3-7ee4e9a7e021-workload-certs\") pod \"ab81877e-88af-4508-82b3-7ee4e9a7e021\" (UID: \"ab81877e-88af-4508-82b3-7ee4e9a7e021\") " Oct 31 14:03:05.083799 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.083778 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab81877e-88af-4508-82b3-7ee4e9a7e021-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab81877e-88af-4508-82b3-7ee4e9a7e021-credential-socket\") pod \"ab81877e-88af-4508-82b3-7ee4e9a7e021\" (UID: \"ab81877e-88af-4508-82b3-7ee4e9a7e021\") " Oct 31 14:03:05.084192 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.083867 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab81877e-88af-4508-82b3-7ee4e9a7e021-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab81877e-88af-4508-82b3-7ee4e9a7e021-istio-envoy\") pod \"ab81877e-88af-4508-82b3-7ee4e9a7e021\" (UID: \"ab81877e-88af-4508-82b3-7ee4e9a7e021\") " Oct 31 14:03:05.084192 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.083899 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab81877e-88af-4508-82b3-7ee4e9a7e021-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab81877e-88af-4508-82b3-7ee4e9a7e021-istio-podinfo\") pod \"ab81877e-88af-4508-82b3-7ee4e9a7e021\" (UID: \"ab81877e-88af-4508-82b3-7ee4e9a7e021\") " Oct 31 14:03:05.084192 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.084065 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab81877e-88af-4508-82b3-7ee4e9a7e021-workload-socket" pod "ab81877e-88af-4508-82b3-7ee4e9a7e021" (UID: "ab81877e-88af-4508-82b3-7ee4e9a7e021"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:03:05.084192 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.084082 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab81877e-88af-4508-82b3-7ee4e9a7e021-istio-data" pod "ab81877e-88af-4508-82b3-7ee4e9a7e021" (UID: "ab81877e-88af-4508-82b3-7ee4e9a7e021"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:03:05.084192 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.084162 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab81877e-88af-4508-82b3-7ee4e9a7e021-istiod-ca-cert" pod "ab81877e-88af-4508-82b3-7ee4e9a7e021" (UID: "ab81877e-88af-4508-82b3-7ee4e9a7e021"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:03:05.084476 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.084276 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab81877e-88af-4508-82b3-7ee4e9a7e021-credential-socket" pod "ab81877e-88af-4508-82b3-7ee4e9a7e021" (UID: "ab81877e-88af-4508-82b3-7ee4e9a7e021"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:03:05.084476 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.084294 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab81877e-88af-4508-82b3-7ee4e9a7e021-workload-certs" pod "ab81877e-88af-4508-82b3-7ee4e9a7e021" (UID: "ab81877e-88af-4508-82b3-7ee4e9a7e021"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:03:05.084476 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.084418 6841 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab81877e-88af-4508-82b3-7ee4e9a7e021-workload-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:05.084476 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.084440 6841 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab81877e-88af-4508-82b3-7ee4e9a7e021-istiod-ca-cert\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:05.084476 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.084467 6841 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab81877e-88af-4508-82b3-7ee4e9a7e021-istio-data\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:05.084476 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.084477 6841 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab81877e-88af-4508-82b3-7ee4e9a7e021-workload-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:05.084857 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.084488 6841 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab81877e-88af-4508-82b3-7ee4e9a7e021-credential-socket\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:05.086015 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.085973 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab81877e-88af-4508-82b3-7ee4e9a7e021-istio-token" pod "ab81877e-88af-4508-82b3-7ee4e9a7e021" (UID: "ab81877e-88af-4508-82b3-7ee4e9a7e021"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:03:05.086112 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.086029 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab81877e-88af-4508-82b3-7ee4e9a7e021-kube-api-access-ttgpn" pod "ab81877e-88af-4508-82b3-7ee4e9a7e021" (UID: "ab81877e-88af-4508-82b3-7ee4e9a7e021"). InnerVolumeSpecName "kube-api-access-ttgpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:03:05.086505 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.086478 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab81877e-88af-4508-82b3-7ee4e9a7e021-istio-podinfo" pod "ab81877e-88af-4508-82b3-7ee4e9a7e021" (UID: "ab81877e-88af-4508-82b3-7ee4e9a7e021"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 14:03:05.086611 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.086555 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab81877e-88af-4508-82b3-7ee4e9a7e021-istio-envoy" pod "ab81877e-88af-4508-82b3-7ee4e9a7e021" (UID: "ab81877e-88af-4508-82b3-7ee4e9a7e021"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:03:05.185477 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.185448 6841 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab81877e-88af-4508-82b3-7ee4e9a7e021-istio-envoy\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:05.185477 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.185474 6841 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab81877e-88af-4508-82b3-7ee4e9a7e021-istio-podinfo\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:05.185679 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.185486 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttgpn\" (UniqueName: \"kubernetes.io/projected/ab81877e-88af-4508-82b3-7ee4e9a7e021-kube-api-access-ttgpn\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:05.185679 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.185496 6841 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab81877e-88af-4508-82b3-7ee4e9a7e021-istio-token\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:05.640796 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.640760 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6df3dbb6-a790-4bfb-ad50-b311812fbe85" path="/var/lib/kubelet/pods/6df3dbb6-a790-4bfb-ad50-b311812fbe85/volumes" Oct 31 14:03:05.641213 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.641198 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80df07ad-8a17-4e04-b3cb-ec95cd327a62" path="/var/lib/kubelet/pods/80df07ad-8a17-4e04-b3cb-ec95cd327a62/volumes" Oct 31 14:03:05.693099 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.693058 6841 generic.go:356] "Generic (PLEG): container finished" podID="ab81877e-88af-4508-82b3-7ee4e9a7e021" containerID="94c8f4fdf9489ba4b00f9daa0dbbd307ada07f75c5cb0a0d5de329eb7460b756" exitCode=0 Oct 31 14:03:05.693474 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.693115 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" event={"ID":"ab81877e-88af-4508-82b3-7ee4e9a7e021","Type":"ContainerDied","Data":"94c8f4fdf9489ba4b00f9daa0dbbd307ada07f75c5cb0a0d5de329eb7460b756"} Oct 31 14:03:05.693474 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.693128 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" Oct 31 14:03:05.693474 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.693184 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw" event={"ID":"ab81877e-88af-4508-82b3-7ee4e9a7e021","Type":"ContainerDied","Data":"b5598709f612aa534b72bc028ab299becd6391e4e57e7b5b0356697b8181adf2"} Oct 31 14:03:05.693474 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.693208 6841 scope.go:122] "RemoveContainer" containerID="94c8f4fdf9489ba4b00f9daa0dbbd307ada07f75c5cb0a0d5de329eb7460b756" Oct 31 14:03:05.702908 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.702883 6841 scope.go:122] "RemoveContainer" containerID="6a91ecac25f68866b7ad7d1a72fd595d0e5134239f4099e193df0b2ccc6d0dc1" Oct 31 14:03:05.711913 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.711889 6841 scope.go:122] "RemoveContainer" containerID="94c8f4fdf9489ba4b00f9daa0dbbd307ada07f75c5cb0a0d5de329eb7460b756" Oct 31 14:03:05.712250 ip-10-0-138-70 kubenswrapper[6841]: E1031 14:03:05.712228 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94c8f4fdf9489ba4b00f9daa0dbbd307ada07f75c5cb0a0d5de329eb7460b756\": container with ID starting with 94c8f4fdf9489ba4b00f9daa0dbbd307ada07f75c5cb0a0d5de329eb7460b756 not found: ID does not exist" containerID="94c8f4fdf9489ba4b00f9daa0dbbd307ada07f75c5cb0a0d5de329eb7460b756" Oct 31 14:03:05.712335 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.712259 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94c8f4fdf9489ba4b00f9daa0dbbd307ada07f75c5cb0a0d5de329eb7460b756"} err="failed to get container status \"94c8f4fdf9489ba4b00f9daa0dbbd307ada07f75c5cb0a0d5de329eb7460b756\": rpc error: code = NotFound desc = could not find container \"94c8f4fdf9489ba4b00f9daa0dbbd307ada07f75c5cb0a0d5de329eb7460b756\": container with ID starting with 94c8f4fdf9489ba4b00f9daa0dbbd307ada07f75c5cb0a0d5de329eb7460b756 not found: ID does not exist" Oct 31 14:03:05.712335 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.712277 6841 scope.go:122] "RemoveContainer" containerID="6a91ecac25f68866b7ad7d1a72fd595d0e5134239f4099e193df0b2ccc6d0dc1" Oct 31 14:03:05.712573 ip-10-0-138-70 kubenswrapper[6841]: E1031 14:03:05.712556 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a91ecac25f68866b7ad7d1a72fd595d0e5134239f4099e193df0b2ccc6d0dc1\": container with ID starting with 6a91ecac25f68866b7ad7d1a72fd595d0e5134239f4099e193df0b2ccc6d0dc1 not found: ID does not exist" containerID="6a91ecac25f68866b7ad7d1a72fd595d0e5134239f4099e193df0b2ccc6d0dc1" Oct 31 14:03:05.712630 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.712577 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a91ecac25f68866b7ad7d1a72fd595d0e5134239f4099e193df0b2ccc6d0dc1"} err="failed to get container status \"6a91ecac25f68866b7ad7d1a72fd595d0e5134239f4099e193df0b2ccc6d0dc1\": rpc error: code = NotFound desc = could not find container \"6a91ecac25f68866b7ad7d1a72fd595d0e5134239f4099e193df0b2ccc6d0dc1\": container with ID starting with 6a91ecac25f68866b7ad7d1a72fd595d0e5134239f4099e193df0b2ccc6d0dc1 not found: ID does not exist" Oct 31 14:03:05.713901 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.713880 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw"] Oct 31 14:03:05.717783 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:05.717763 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-5d3f86-openshift-default-78dff9cf97rrbnw"] Oct 31 14:03:07.641145 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:07.641114 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab81877e-88af-4508-82b3-7ee4e9a7e021" path="/var/lib/kubelet/pods/ab81877e-88af-4508-82b3-7ee4e9a7e021/volumes" Oct 31 14:03:15.025834 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:15.025800 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sftzl"] Oct 31 14:03:15.029698 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:15.029668 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sftzl"] Oct 31 14:03:15.641478 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:15.641448 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa" path="/var/lib/kubelet/pods/6bc1d5e6-8eb5-42e2-8349-06e66eddd4fa/volumes" Oct 31 14:03:20.034624 ip-10-0-138-70 kubenswrapper[6841]: E1031 14:03:20.034584 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 14:03:25.022414 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:25.022381 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8gpjs"] Oct 31 14:03:25.025677 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:25.025652 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8gpjs"] Oct 31 14:03:25.640651 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:25.640616 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4be917f1-8a0f-4e31-bb92-be147421d3dd" path="/var/lib/kubelet/pods/4be917f1-8a0f-4e31-bb92-be147421d3dd/volumes" Oct 31 14:03:41.812841 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:41.812809 6841 scope.go:122] "RemoveContainer" containerID="5cc6033e95ea8a00ac5f2a9b59fb74697b87895efb53008605ce8d374b0cc0f1" Oct 31 14:03:41.821976 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:41.821952 6841 scope.go:122] "RemoveContainer" containerID="d67e100292d3b5594bcc3c085c2ffa01abe6fc32b4be362f8903a14928a51567" Oct 31 14:03:41.830458 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:41.830433 6841 scope.go:122] "RemoveContainer" containerID="46616dfced959209a6a572c0daf3fcec76f39e82bdaebe7dae224fad85d19845" Oct 31 14:03:42.157736 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:42.157657 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:03:42.359110 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:42.359071 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:03:45.023373 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:45.023336 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ptl2l"] Oct 31 14:03:45.026337 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:45.026309 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ptl2l"] Oct 31 14:03:45.640732 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:03:45.640701 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1839bfc3-00a9-436e-bc8d-e4f20b0dea30" path="/var/lib/kubelet/pods/1839bfc3-00a9-436e-bc8d-e4f20b0dea30/volumes" Oct 31 14:03:50.033902 ip-10-0-138-70 kubenswrapper[6841]: E1031 14:03:50.033868 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 14:04:00.024512 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:00.024468 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nqfhk"] Oct 31 14:04:00.027022 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:00.026997 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nqfhk"] Oct 31 14:04:01.641832 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:01.641797 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e8828f0-f12f-4915-9fdb-493d04e7906f" path="/var/lib/kubelet/pods/4e8828f0-f12f-4915-9fdb-493d04e7906f/volumes" Oct 31 14:04:17.023866 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:17.023806 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s7729"] Oct 31 14:04:17.026288 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:17.026262 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s7729"] Oct 31 14:04:17.642740 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:17.642702 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fe43ae2-c997-405d-b6a9-5b3d3d649c04" path="/var/lib/kubelet/pods/6fe43ae2-c997-405d-b6a9-5b3d3d649c04/volumes" Oct 31 14:04:20.015574 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:20.015534 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:04:20.033805 ip-10-0-138-70 kubenswrapper[6841]: E1031 14:04:20.033770 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 14:04:20.227490 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:20.227455 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:04:31.037049 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:31.037013 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zrfkx"] Oct 31 14:04:31.040282 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:31.040254 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zrfkx"] Oct 31 14:04:31.641386 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:31.641351 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf72d6c7-7c81-4376-8d7a-1722174becc6" path="/var/lib/kubelet/pods/bf72d6c7-7c81-4376-8d7a-1722174becc6/volumes" Oct 31 14:04:41.898007 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:41.887869 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-6lxfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 14:04:41.898007 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:41.887949 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" podUID="d0ad44f3-de70-4b31-8d39-ceeed24efd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 14:04:41.898007 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:41.888124 6841 scope.go:122] "RemoveContainer" containerID="b431d3c8fa1e9ce3eb76b49b451af8436c9100dec9aad0157cb157b157c90109" Oct 31 14:04:41.931806 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:41.931648 6841 scope.go:122] "RemoveContainer" containerID="233e9dd8de307e70f9831a6126631a95b5892afc20f9cadf08b24adcec5542bb" Oct 31 14:04:41.981003 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:41.976642 6841 scope.go:122] "RemoveContainer" containerID="6b2bf70da5f88104d6a4ef0b4560eba313afae195ba4a25666ff9738f6e43acc" Oct 31 14:04:42.014475 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:42.014446 6841 scope.go:122] "RemoveContainer" containerID="958e5ce033cad86a832a7f2efabc8fc8d2eec87a2399448617c72f86933f2c9d" Oct 31 14:04:42.238997 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:42.238945 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:04:43.264093 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:43.264025 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xwpxp"] Oct 31 14:04:43.264578 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:43.264562 6841 cpu_manager.go:414] "containerMap: removing container" podUID="ab81877e-88af-4508-82b3-7ee4e9a7e021" containerName="istio-proxy" Oct 31 14:04:43.264632 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:43.264580 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab81877e-88af-4508-82b3-7ee4e9a7e021" containerName="istio-proxy" Oct 31 14:04:43.264632 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:43.264593 6841 cpu_manager.go:414] "containerMap: removing container" podUID="ab81877e-88af-4508-82b3-7ee4e9a7e021" containerName="istio-proxy" Oct 31 14:04:43.264632 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:43.264600 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab81877e-88af-4508-82b3-7ee4e9a7e021" containerName="istio-proxy" Oct 31 14:04:43.264632 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:43.264615 6841 cpu_manager.go:414] "containerMap: removing container" podUID="1bb799f2-3c7f-47eb-9bad-1fbdeb70e452" containerName="revoke-keys" Oct 31 14:04:43.264632 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:43.264621 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bb799f2-3c7f-47eb-9bad-1fbdeb70e452" containerName="revoke-keys" Oct 31 14:04:43.264833 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:43.264634 6841 cpu_manager.go:414] "containerMap: removing container" podUID="6df3dbb6-a790-4bfb-ad50-b311812fbe85" containerName="payload-pre-processing" Oct 31 14:04:43.264833 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:43.264639 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6df3dbb6-a790-4bfb-ad50-b311812fbe85" containerName="payload-pre-processing" Oct 31 14:04:43.264833 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:43.264655 6841 cpu_manager.go:414] "containerMap: removing container" podUID="80df07ad-8a17-4e04-b3cb-ec95cd327a62" containerName="payload-processing" Oct 31 14:04:43.264833 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:43.264660 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80df07ad-8a17-4e04-b3cb-ec95cd327a62" containerName="payload-processing" Oct 31 14:04:43.264833 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:43.264721 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bb799f2-3c7f-47eb-9bad-1fbdeb70e452" containerName="revoke-keys" Oct 31 14:04:43.264833 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:43.264732 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="6df3dbb6-a790-4bfb-ad50-b311812fbe85" containerName="payload-pre-processing" Oct 31 14:04:43.264833 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:43.264740 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="80df07ad-8a17-4e04-b3cb-ec95cd327a62" containerName="payload-processing" Oct 31 14:04:43.264833 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:43.264748 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab81877e-88af-4508-82b3-7ee4e9a7e021" containerName="istio-proxy" Oct 31 14:04:43.264833 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:43.264754 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab81877e-88af-4508-82b3-7ee4e9a7e021" containerName="istio-proxy" Oct 31 14:04:43.269624 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:43.269604 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xwpxp" Oct 31 14:04:43.277456 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:43.277431 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:04:43.277573 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:43.277553 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 14:04:43.279131 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:43.279103 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xwpxp"] Oct 31 14:04:43.344029 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:43.344002 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 14:04:43.344206 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:43.344174 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 14:04:43.409226 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:43.409191 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fcdba1a-06c8-4081-b522-6bb813e249d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-xwpxp\" (UID: \"5fcdba1a-06c8-4081-b522-6bb813e249d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xwpxp" Oct 31 14:04:43.409400 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:43.409230 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fcdba1a-06c8-4081-b522-6bb813e249d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-xwpxp\" (UID: \"5fcdba1a-06c8-4081-b522-6bb813e249d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xwpxp" Oct 31 14:04:43.409400 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:43.409348 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmvlv\" (UniqueName: \"kubernetes.io/projected/5fcdba1a-06c8-4081-b522-6bb813e249d4-kube-api-access-vmvlv\") pod \"maas-api-revoke-sub-d576e68b-xwpxp\" (UID: \"5fcdba1a-06c8-4081-b522-6bb813e249d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xwpxp" Oct 31 14:04:43.510762 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:43.510724 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fcdba1a-06c8-4081-b522-6bb813e249d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-xwpxp\" (UID: \"5fcdba1a-06c8-4081-b522-6bb813e249d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xwpxp" Oct 31 14:04:43.510942 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:43.510770 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fcdba1a-06c8-4081-b522-6bb813e249d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-xwpxp\" (UID: \"5fcdba1a-06c8-4081-b522-6bb813e249d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xwpxp" Oct 31 14:04:43.510942 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:43.510820 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmvlv\" (UniqueName: \"kubernetes.io/projected/5fcdba1a-06c8-4081-b522-6bb813e249d4-kube-api-access-vmvlv\") pod \"maas-api-revoke-sub-d576e68b-xwpxp\" (UID: \"5fcdba1a-06c8-4081-b522-6bb813e249d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xwpxp" Oct 31 14:04:43.511652 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:43.511627 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fcdba1a-06c8-4081-b522-6bb813e249d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-xwpxp\" (UID: \"5fcdba1a-06c8-4081-b522-6bb813e249d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xwpxp" Oct 31 14:04:43.513220 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:43.513202 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fcdba1a-06c8-4081-b522-6bb813e249d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-xwpxp\" (UID: \"5fcdba1a-06c8-4081-b522-6bb813e249d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xwpxp" Oct 31 14:04:43.523447 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:43.523389 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmvlv\" (UniqueName: \"kubernetes.io/projected/5fcdba1a-06c8-4081-b522-6bb813e249d4-kube-api-access-vmvlv\") pod \"maas-api-revoke-sub-d576e68b-xwpxp\" (UID: \"5fcdba1a-06c8-4081-b522-6bb813e249d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xwpxp" Oct 31 14:04:43.584590 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:43.584561 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xwpxp" Oct 31 14:04:43.858833 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:43.858802 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xwpxp"] Oct 31 14:04:43.860951 ip-10-0-138-70 kubenswrapper[6841]: W1031 14:04:43.860924 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fcdba1a_06c8_4081_b522_6bb813e249d4.slice/crio-7c8d2078e594b7c486a818359d650052abe79fdc0afaee16b2499d940e523f91 WatchSource:0}: Error finding container 7c8d2078e594b7c486a818359d650052abe79fdc0afaee16b2499d940e523f91: Status 404 returned error can't find the container with id 7c8d2078e594b7c486a818359d650052abe79fdc0afaee16b2499d940e523f91 Oct 31 14:04:44.349998 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:44.349947 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xwpxp" event={"ID":"5fcdba1a-06c8-4081-b522-6bb813e249d4","Type":"ContainerStarted","Data":"eda854e273748b21f7de481ae08e1ee1e8f0882f51c824e0c3aa4793abb5210c"} Oct 31 14:04:44.350369 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:44.350062 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xwpxp" event={"ID":"5fcdba1a-06c8-4081-b522-6bb813e249d4","Type":"ContainerStarted","Data":"7c8d2078e594b7c486a818359d650052abe79fdc0afaee16b2499d940e523f91"} Oct 31 14:04:44.369797 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:44.369753 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xwpxp" podStartSLOduration=1.3697387220000001 podStartE2EDuration="1.369738722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:04:44.368413554 +0000 UTC m=+5530.851536456" watchObservedRunningTime="2026-10-31 14:04:44.369738722 +0000 UTC m=+5530.852861647" Oct 31 14:04:45.357197 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:45.357164 6841 generic.go:356] "Generic (PLEG): container finished" podID="5fcdba1a-06c8-4081-b522-6bb813e249d4" containerID="eda854e273748b21f7de481ae08e1ee1e8f0882f51c824e0c3aa4793abb5210c" exitCode=0 Oct 31 14:04:45.357591 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:45.357242 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xwpxp" event={"ID":"5fcdba1a-06c8-4081-b522-6bb813e249d4","Type":"ContainerDied","Data":"eda854e273748b21f7de481ae08e1ee1e8f0882f51c824e0c3aa4793abb5210c"} Oct 31 14:04:46.494954 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:46.494929 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xwpxp" Oct 31 14:04:46.644022 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:46.643916 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fcdba1a-06c8-4081-b522-6bb813e249d4-kube-api-access-vmvlv\" (UniqueName: \"kubernetes.io/projected/5fcdba1a-06c8-4081-b522-6bb813e249d4-kube-api-access-vmvlv\") pod \"5fcdba1a-06c8-4081-b522-6bb813e249d4\" (UID: \"5fcdba1a-06c8-4081-b522-6bb813e249d4\") " Oct 31 14:04:46.644022 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:46.643966 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fcdba1a-06c8-4081-b522-6bb813e249d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fcdba1a-06c8-4081-b522-6bb813e249d4-maas-api-xks-ca\") pod \"5fcdba1a-06c8-4081-b522-6bb813e249d4\" (UID: \"5fcdba1a-06c8-4081-b522-6bb813e249d4\") " Oct 31 14:04:46.644276 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:46.644111 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fcdba1a-06c8-4081-b522-6bb813e249d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fcdba1a-06c8-4081-b522-6bb813e249d4-maas-api-openshift-ca\") pod \"5fcdba1a-06c8-4081-b522-6bb813e249d4\" (UID: \"5fcdba1a-06c8-4081-b522-6bb813e249d4\") " Oct 31 14:04:46.644496 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:46.644466 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fcdba1a-06c8-4081-b522-6bb813e249d4-maas-api-openshift-ca" pod "5fcdba1a-06c8-4081-b522-6bb813e249d4" (UID: "5fcdba1a-06c8-4081-b522-6bb813e249d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:04:46.646055 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:46.646021 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fcdba1a-06c8-4081-b522-6bb813e249d4-maas-api-xks-ca" pod "5fcdba1a-06c8-4081-b522-6bb813e249d4" (UID: "5fcdba1a-06c8-4081-b522-6bb813e249d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:04:46.646168 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:46.646145 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fcdba1a-06c8-4081-b522-6bb813e249d4-kube-api-access-vmvlv" pod "5fcdba1a-06c8-4081-b522-6bb813e249d4" (UID: "5fcdba1a-06c8-4081-b522-6bb813e249d4"). InnerVolumeSpecName "kube-api-access-vmvlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:04:46.745794 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:46.745756 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fcdba1a-06c8-4081-b522-6bb813e249d4-maas-api-openshift-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:04:46.745794 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:46.745788 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmvlv\" (UniqueName: \"kubernetes.io/projected/5fcdba1a-06c8-4081-b522-6bb813e249d4-kube-api-access-vmvlv\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:04:46.746102 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:46.745803 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fcdba1a-06c8-4081-b522-6bb813e249d4-maas-api-xks-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:04:47.370133 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:47.370100 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xwpxp" Oct 31 14:04:47.370352 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:47.370129 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xwpxp" event={"ID":"5fcdba1a-06c8-4081-b522-6bb813e249d4","Type":"ContainerDied","Data":"7c8d2078e594b7c486a818359d650052abe79fdc0afaee16b2499d940e523f91"} Oct 31 14:04:47.370352 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:47.370178 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c8d2078e594b7c486a818359d650052abe79fdc0afaee16b2499d940e523f91" Oct 31 14:04:50.033202 ip-10-0-138-70 kubenswrapper[6841]: E1031 14:04:50.033165 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 14:04:59.558947 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:59.558911 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kz2mk"] Oct 31 14:04:59.559442 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:59.559426 6841 cpu_manager.go:414] "containerMap: removing container" podUID="5fcdba1a-06c8-4081-b522-6bb813e249d4" containerName="revoke-keys" Oct 31 14:04:59.559495 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:59.559443 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fcdba1a-06c8-4081-b522-6bb813e249d4" containerName="revoke-keys" Oct 31 14:04:59.559558 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:59.559549 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fcdba1a-06c8-4081-b522-6bb813e249d4" containerName="revoke-keys" Oct 31 14:04:59.563207 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:59.563180 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kz2mk" Oct 31 14:04:59.567368 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:59.567335 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 14:04:59.567588 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:59.567447 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 14:04:59.568734 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:59.568714 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:04:59.568846 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:59.568777 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 14:04:59.573893 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:59.573869 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kz2mk"] Oct 31 14:04:59.639913 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:59.639858 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4c27\" (UniqueName: \"kubernetes.io/projected/aed7592e-126e-4dda-b079-5e0e6c15efd8-kube-api-access-x4c27\") pod \"maas-api-revoke-sub-d2bc93bc-kz2mk\" (UID: \"aed7592e-126e-4dda-b079-5e0e6c15efd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kz2mk" Oct 31 14:04:59.639913 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:59.639903 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aed7592e-126e-4dda-b079-5e0e6c15efd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-kz2mk\" (UID: \"aed7592e-126e-4dda-b079-5e0e6c15efd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kz2mk" Oct 31 14:04:59.640224 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:59.640060 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aed7592e-126e-4dda-b079-5e0e6c15efd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-kz2mk\" (UID: \"aed7592e-126e-4dda-b079-5e0e6c15efd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kz2mk" Oct 31 14:04:59.740821 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:59.740765 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4c27\" (UniqueName: \"kubernetes.io/projected/aed7592e-126e-4dda-b079-5e0e6c15efd8-kube-api-access-x4c27\") pod \"maas-api-revoke-sub-d2bc93bc-kz2mk\" (UID: \"aed7592e-126e-4dda-b079-5e0e6c15efd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kz2mk" Oct 31 14:04:59.740821 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:59.740834 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aed7592e-126e-4dda-b079-5e0e6c15efd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-kz2mk\" (UID: \"aed7592e-126e-4dda-b079-5e0e6c15efd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kz2mk" Oct 31 14:04:59.741117 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:59.740893 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aed7592e-126e-4dda-b079-5e0e6c15efd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-kz2mk\" (UID: \"aed7592e-126e-4dda-b079-5e0e6c15efd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kz2mk" Oct 31 14:04:59.741595 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:59.741574 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aed7592e-126e-4dda-b079-5e0e6c15efd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-kz2mk\" (UID: \"aed7592e-126e-4dda-b079-5e0e6c15efd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kz2mk" Oct 31 14:04:59.743292 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:59.743268 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aed7592e-126e-4dda-b079-5e0e6c15efd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-kz2mk\" (UID: \"aed7592e-126e-4dda-b079-5e0e6c15efd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kz2mk" Oct 31 14:04:59.753550 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:59.753514 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4c27\" (UniqueName: \"kubernetes.io/projected/aed7592e-126e-4dda-b079-5e0e6c15efd8-kube-api-access-x4c27\") pod \"maas-api-revoke-sub-d2bc93bc-kz2mk\" (UID: \"aed7592e-126e-4dda-b079-5e0e6c15efd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kz2mk" Oct 31 14:04:59.879244 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:04:59.879173 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kz2mk" Oct 31 14:05:00.154534 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:00.154503 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kz2mk"] Oct 31 14:05:00.156046 ip-10-0-138-70 kubenswrapper[6841]: W1031 14:05:00.156021 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaed7592e_126e_4dda_b079_5e0e6c15efd8.slice/crio-e655fcffcf00ff1161762398b516e9243446a4d4047850aa77fccfeb85b0d224 WatchSource:0}: Error finding container e655fcffcf00ff1161762398b516e9243446a4d4047850aa77fccfeb85b0d224: Status 404 returned error can't find the container with id e655fcffcf00ff1161762398b516e9243446a4d4047850aa77fccfeb85b0d224 Oct 31 14:05:00.414487 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:00.414410 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:05:00.463135 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:00.463103 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kz2mk" event={"ID":"aed7592e-126e-4dda-b079-5e0e6c15efd8","Type":"ContainerStarted","Data":"65e272779aa24e624d82cb71e3a8123500338589a8e602f148d9058a1ce137a1"} Oct 31 14:05:00.463251 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:00.463157 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kz2mk" event={"ID":"aed7592e-126e-4dda-b079-5e0e6c15efd8","Type":"ContainerStarted","Data":"e655fcffcf00ff1161762398b516e9243446a4d4047850aa77fccfeb85b0d224"} Oct 31 14:05:00.485655 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:00.485609 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kz2mk" podStartSLOduration=1.485596423 podStartE2EDuration="1.485596423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:05:00.483773977 +0000 UTC m=+5546.966896873" watchObservedRunningTime="2026-10-31 14:05:00.485596423 +0000 UTC m=+5546.968719337" Oct 31 14:05:01.470202 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:01.470169 6841 generic.go:356] "Generic (PLEG): container finished" podID="aed7592e-126e-4dda-b079-5e0e6c15efd8" containerID="65e272779aa24e624d82cb71e3a8123500338589a8e602f148d9058a1ce137a1" exitCode=0 Oct 31 14:05:01.470669 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:01.470254 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kz2mk" event={"ID":"aed7592e-126e-4dda-b079-5e0e6c15efd8","Type":"ContainerDied","Data":"65e272779aa24e624d82cb71e3a8123500338589a8e602f148d9058a1ce137a1"} Oct 31 14:05:02.607932 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:02.607905 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kz2mk" Oct 31 14:05:02.668589 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:02.668559 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aed7592e-126e-4dda-b079-5e0e6c15efd8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aed7592e-126e-4dda-b079-5e0e6c15efd8-maas-api-xks-ca\") pod \"aed7592e-126e-4dda-b079-5e0e6c15efd8\" (UID: \"aed7592e-126e-4dda-b079-5e0e6c15efd8\") " Oct 31 14:05:02.668768 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:02.668599 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aed7592e-126e-4dda-b079-5e0e6c15efd8-kube-api-access-x4c27\" (UniqueName: \"kubernetes.io/projected/aed7592e-126e-4dda-b079-5e0e6c15efd8-kube-api-access-x4c27\") pod \"aed7592e-126e-4dda-b079-5e0e6c15efd8\" (UID: \"aed7592e-126e-4dda-b079-5e0e6c15efd8\") " Oct 31 14:05:02.668768 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:02.668653 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aed7592e-126e-4dda-b079-5e0e6c15efd8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aed7592e-126e-4dda-b079-5e0e6c15efd8-maas-api-openshift-ca\") pod \"aed7592e-126e-4dda-b079-5e0e6c15efd8\" (UID: \"aed7592e-126e-4dda-b079-5e0e6c15efd8\") " Oct 31 14:05:02.669073 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:02.669048 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aed7592e-126e-4dda-b079-5e0e6c15efd8-maas-api-openshift-ca" pod "aed7592e-126e-4dda-b079-5e0e6c15efd8" (UID: "aed7592e-126e-4dda-b079-5e0e6c15efd8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:05:02.670726 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:02.670696 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aed7592e-126e-4dda-b079-5e0e6c15efd8-maas-api-xks-ca" pod "aed7592e-126e-4dda-b079-5e0e6c15efd8" (UID: "aed7592e-126e-4dda-b079-5e0e6c15efd8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:05:02.670831 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:02.670792 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aed7592e-126e-4dda-b079-5e0e6c15efd8-kube-api-access-x4c27" pod "aed7592e-126e-4dda-b079-5e0e6c15efd8" (UID: "aed7592e-126e-4dda-b079-5e0e6c15efd8"). InnerVolumeSpecName "kube-api-access-x4c27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:05:02.770233 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:02.770154 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aed7592e-126e-4dda-b079-5e0e6c15efd8-maas-api-xks-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:05:02.770233 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:02.770186 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4c27\" (UniqueName: \"kubernetes.io/projected/aed7592e-126e-4dda-b079-5e0e6c15efd8-kube-api-access-x4c27\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:05:02.770233 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:02.770196 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aed7592e-126e-4dda-b079-5e0e6c15efd8-maas-api-openshift-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:05:03.482046 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:03.482012 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kz2mk" event={"ID":"aed7592e-126e-4dda-b079-5e0e6c15efd8","Type":"ContainerDied","Data":"e655fcffcf00ff1161762398b516e9243446a4d4047850aa77fccfeb85b0d224"} Oct 31 14:05:03.482046 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:03.482040 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kz2mk" Oct 31 14:05:03.482263 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:03.482067 6841 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e655fcffcf00ff1161762398b516e9243446a4d4047850aa77fccfeb85b0d224" Oct 31 14:05:11.620612 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.620580 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-547b4fbdb9-kz25g"] Oct 31 14:05:11.621078 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.621063 6841 cpu_manager.go:414] "containerMap: removing container" podUID="aed7592e-126e-4dda-b079-5e0e6c15efd8" containerName="revoke-keys" Oct 31 14:05:11.621139 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.621080 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aed7592e-126e-4dda-b079-5e0e6c15efd8" containerName="revoke-keys" Oct 31 14:05:11.621189 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.621180 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="aed7592e-126e-4dda-b079-5e0e6c15efd8" containerName="revoke-keys" Oct 31 14:05:11.628879 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.628856 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-kz25g" Oct 31 14:05:11.636088 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.636064 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 14:05:11.636238 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.636118 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 14:05:11.636755 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.636716 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 14:05:11.636951 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.636929 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 14:05:11.636951 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.636853 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rhftr\"" Oct 31 14:05:11.644125 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.644098 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-547b4fbdb9-kz25g"] Oct 31 14:05:11.645027 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.645007 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-f787c88f-9dt8r"] Oct 31 14:05:11.648950 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.648924 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f787c88f-9dt8r" Oct 31 14:05:11.660592 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.660333 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f787c88f-9dt8r"] Oct 31 14:05:11.751566 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.751531 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39db85ed-30bf-4988-8f37-2a7bd97d79f5-openshift-service-ca\") pod \"maas-api-f787c88f-9dt8r\" (UID: \"39db85ed-30bf-4988-8f37-2a7bd97d79f5\") " pod="odh-ai-gateway-infra/maas-api-f787c88f-9dt8r" Oct 31 14:05:11.751740 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.751583 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbe3f678-5b35-4552-9c0a-7a10cd9386fb-maas-api-tls\") pod \"maas-api-547b4fbdb9-kz25g\" (UID: \"fbe3f678-5b35-4552-9c0a-7a10cd9386fb\") " pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-kz25g" Oct 31 14:05:11.751740 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.751676 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcff4\" (UniqueName: \"kubernetes.io/projected/39db85ed-30bf-4988-8f37-2a7bd97d79f5-kube-api-access-lcff4\") pod \"maas-api-f787c88f-9dt8r\" (UID: \"39db85ed-30bf-4988-8f37-2a7bd97d79f5\") " pod="odh-ai-gateway-infra/maas-api-f787c88f-9dt8r" Oct 31 14:05:11.751740 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.751706 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39db85ed-30bf-4988-8f37-2a7bd97d79f5-maas-api-tls\") pod \"maas-api-f787c88f-9dt8r\" (UID: \"39db85ed-30bf-4988-8f37-2a7bd97d79f5\") " pod="odh-ai-gateway-infra/maas-api-f787c88f-9dt8r" Oct 31 14:05:11.751868 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.751737 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbe3f678-5b35-4552-9c0a-7a10cd9386fb-metrics-certs\") pod \"maas-api-547b4fbdb9-kz25g\" (UID: \"fbe3f678-5b35-4552-9c0a-7a10cd9386fb\") " pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-kz25g" Oct 31 14:05:11.751868 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.751831 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbe3f678-5b35-4552-9c0a-7a10cd9386fb-openshift-service-ca\") pod \"maas-api-547b4fbdb9-kz25g\" (UID: \"fbe3f678-5b35-4552-9c0a-7a10cd9386fb\") " pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-kz25g" Oct 31 14:05:11.751954 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.751867 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39db85ed-30bf-4988-8f37-2a7bd97d79f5-metrics-certs\") pod \"maas-api-f787c88f-9dt8r\" (UID: \"39db85ed-30bf-4988-8f37-2a7bd97d79f5\") " pod="odh-ai-gateway-infra/maas-api-f787c88f-9dt8r" Oct 31 14:05:11.751954 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.751888 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tprr\" (UniqueName: \"kubernetes.io/projected/fbe3f678-5b35-4552-9c0a-7a10cd9386fb-kube-api-access-4tprr\") pod \"maas-api-547b4fbdb9-kz25g\" (UID: \"fbe3f678-5b35-4552-9c0a-7a10cd9386fb\") " pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-kz25g" Oct 31 14:05:11.853446 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.853410 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbe3f678-5b35-4552-9c0a-7a10cd9386fb-metrics-certs\") pod \"maas-api-547b4fbdb9-kz25g\" (UID: \"fbe3f678-5b35-4552-9c0a-7a10cd9386fb\") " pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-kz25g" Oct 31 14:05:11.853446 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.853449 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbe3f678-5b35-4552-9c0a-7a10cd9386fb-openshift-service-ca\") pod \"maas-api-547b4fbdb9-kz25g\" (UID: \"fbe3f678-5b35-4552-9c0a-7a10cd9386fb\") " pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-kz25g" Oct 31 14:05:11.853680 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.853500 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39db85ed-30bf-4988-8f37-2a7bd97d79f5-metrics-certs\") pod \"maas-api-f787c88f-9dt8r\" (UID: \"39db85ed-30bf-4988-8f37-2a7bd97d79f5\") " pod="odh-ai-gateway-infra/maas-api-f787c88f-9dt8r" Oct 31 14:05:11.853680 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.853529 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tprr\" (UniqueName: \"kubernetes.io/projected/fbe3f678-5b35-4552-9c0a-7a10cd9386fb-kube-api-access-4tprr\") pod \"maas-api-547b4fbdb9-kz25g\" (UID: \"fbe3f678-5b35-4552-9c0a-7a10cd9386fb\") " pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-kz25g" Oct 31 14:05:11.853680 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.853575 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39db85ed-30bf-4988-8f37-2a7bd97d79f5-openshift-service-ca\") pod \"maas-api-f787c88f-9dt8r\" (UID: \"39db85ed-30bf-4988-8f37-2a7bd97d79f5\") " pod="odh-ai-gateway-infra/maas-api-f787c88f-9dt8r" Oct 31 14:05:11.853680 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.853620 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbe3f678-5b35-4552-9c0a-7a10cd9386fb-maas-api-tls\") pod \"maas-api-547b4fbdb9-kz25g\" (UID: \"fbe3f678-5b35-4552-9c0a-7a10cd9386fb\") " pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-kz25g" Oct 31 14:05:11.853887 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.853687 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcff4\" (UniqueName: \"kubernetes.io/projected/39db85ed-30bf-4988-8f37-2a7bd97d79f5-kube-api-access-lcff4\") pod \"maas-api-f787c88f-9dt8r\" (UID: \"39db85ed-30bf-4988-8f37-2a7bd97d79f5\") " pod="odh-ai-gateway-infra/maas-api-f787c88f-9dt8r" Oct 31 14:05:11.853887 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.853734 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39db85ed-30bf-4988-8f37-2a7bd97d79f5-maas-api-tls\") pod \"maas-api-f787c88f-9dt8r\" (UID: \"39db85ed-30bf-4988-8f37-2a7bd97d79f5\") " pod="odh-ai-gateway-infra/maas-api-f787c88f-9dt8r" Oct 31 14:05:11.854488 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.854456 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39db85ed-30bf-4988-8f37-2a7bd97d79f5-openshift-service-ca\") pod \"maas-api-f787c88f-9dt8r\" (UID: \"39db85ed-30bf-4988-8f37-2a7bd97d79f5\") " pod="odh-ai-gateway-infra/maas-api-f787c88f-9dt8r" Oct 31 14:05:11.854488 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.854491 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbe3f678-5b35-4552-9c0a-7a10cd9386fb-openshift-service-ca\") pod \"maas-api-547b4fbdb9-kz25g\" (UID: \"fbe3f678-5b35-4552-9c0a-7a10cd9386fb\") " pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-kz25g" Oct 31 14:05:11.856256 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.856229 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbe3f678-5b35-4552-9c0a-7a10cd9386fb-maas-api-tls\") pod \"maas-api-547b4fbdb9-kz25g\" (UID: \"fbe3f678-5b35-4552-9c0a-7a10cd9386fb\") " pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-kz25g" Oct 31 14:05:11.856499 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.856472 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39db85ed-30bf-4988-8f37-2a7bd97d79f5-maas-api-tls\") pod \"maas-api-f787c88f-9dt8r\" (UID: \"39db85ed-30bf-4988-8f37-2a7bd97d79f5\") " pod="odh-ai-gateway-infra/maas-api-f787c88f-9dt8r" Oct 31 14:05:11.856644 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.856628 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbe3f678-5b35-4552-9c0a-7a10cd9386fb-metrics-certs\") pod \"maas-api-547b4fbdb9-kz25g\" (UID: \"fbe3f678-5b35-4552-9c0a-7a10cd9386fb\") " pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-kz25g" Oct 31 14:05:11.856689 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.856655 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39db85ed-30bf-4988-8f37-2a7bd97d79f5-metrics-certs\") pod \"maas-api-f787c88f-9dt8r\" (UID: \"39db85ed-30bf-4988-8f37-2a7bd97d79f5\") " pod="odh-ai-gateway-infra/maas-api-f787c88f-9dt8r" Oct 31 14:05:11.865625 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.865599 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcff4\" (UniqueName: \"kubernetes.io/projected/39db85ed-30bf-4988-8f37-2a7bd97d79f5-kube-api-access-lcff4\") pod \"maas-api-f787c88f-9dt8r\" (UID: \"39db85ed-30bf-4988-8f37-2a7bd97d79f5\") " pod="odh-ai-gateway-infra/maas-api-f787c88f-9dt8r" Oct 31 14:05:11.865915 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.865887 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tprr\" (UniqueName: \"kubernetes.io/projected/fbe3f678-5b35-4552-9c0a-7a10cd9386fb-kube-api-access-4tprr\") pod \"maas-api-547b4fbdb9-kz25g\" (UID: \"fbe3f678-5b35-4552-9c0a-7a10cd9386fb\") " pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-kz25g" Oct 31 14:05:11.946685 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.946655 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-kz25g" Oct 31 14:05:11.969564 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:11.969541 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f787c88f-9dt8r" Oct 31 14:05:12.237455 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:12.237424 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-547b4fbdb9-kz25g"] Oct 31 14:05:12.238651 ip-10-0-138-70 kubenswrapper[6841]: W1031 14:05:12.238626 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe3f678_5b35_4552_9c0a_7a10cd9386fb.slice/crio-db050fd51b97c978d10f9905b6c8c6512d773d5225f7836c4725b6f5378c284a WatchSource:0}: Error finding container db050fd51b97c978d10f9905b6c8c6512d773d5225f7836c4725b6f5378c284a: Status 404 returned error can't find the container with id db050fd51b97c978d10f9905b6c8c6512d773d5225f7836c4725b6f5378c284a Oct 31 14:05:12.304197 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:12.304166 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f787c88f-9dt8r"] Oct 31 14:05:12.305974 ip-10-0-138-70 kubenswrapper[6841]: W1031 14:05:12.305951 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39db85ed_30bf_4988_8f37_2a7bd97d79f5.slice/crio-f1c2b139c1094f668ecdaea012a15d48bb5b8e4a7321b5f98097590945b075c4 WatchSource:0}: Error finding container f1c2b139c1094f668ecdaea012a15d48bb5b8e4a7321b5f98097590945b075c4: Status 404 returned error can't find the container with id f1c2b139c1094f668ecdaea012a15d48bb5b8e4a7321b5f98097590945b075c4 Oct 31 14:05:12.536625 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:12.536533 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f787c88f-9dt8r" event={"ID":"39db85ed-30bf-4988-8f37-2a7bd97d79f5","Type":"ContainerStarted","Data":"f1c2b139c1094f668ecdaea012a15d48bb5b8e4a7321b5f98097590945b075c4"} Oct 31 14:05:12.537785 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:12.537759 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-kz25g" event={"ID":"fbe3f678-5b35-4552-9c0a-7a10cd9386fb","Type":"ContainerStarted","Data":"db050fd51b97c978d10f9905b6c8c6512d773d5225f7836c4725b6f5378c284a"} Oct 31 14:05:13.544301 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:13.544261 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-kz25g" event={"ID":"fbe3f678-5b35-4552-9c0a-7a10cd9386fb","Type":"ContainerStarted","Data":"038e607cbcd71e314edd372624d4cc763a7cc0df0786a6f424bc916bef6e6f54"} Oct 31 14:05:13.544811 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:13.544441 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-kz25g" Oct 31 14:05:13.545908 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:13.545880 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f787c88f-9dt8r" event={"ID":"39db85ed-30bf-4988-8f37-2a7bd97d79f5","Type":"ContainerStarted","Data":"4aa86d630be75ec016c64fee210b97a4e6318b220f07582ed0ac140d1a646aeb"} Oct 31 14:05:13.546083 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:13.546025 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-f787c88f-9dt8r" Oct 31 14:05:13.565091 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:13.565044 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-kz25g" podStartSLOduration=2.082833589 podStartE2EDuration="2.565031919s" totalImagesPullingTime="482.19833ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:05:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:05:13.564110864 +0000 UTC m=+5560.047233765" watchObservedRunningTime="2026-10-31 14:05:13.565031919 +0000 UTC m=+5560.048154810" Oct 31 14:05:13.587735 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:13.587688 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-f787c88f-9dt8r" podStartSLOduration=2.149822631 podStartE2EDuration="2.587675478s" totalImagesPullingTime="437.852847ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:05:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:05:13.586144865 +0000 UTC m=+5560.069267766" watchObservedRunningTime="2026-10-31 14:05:13.587675478 +0000 UTC m=+5560.070798383" Oct 31 14:05:19.558166 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:19.558131 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-kz25g" Oct 31 14:05:19.558647 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:19.558627 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-f787c88f-9dt8r" Oct 31 14:05:19.657108 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:19.657077 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-547b4fbdb9-kz25g"] Oct 31 14:05:19.657289 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:19.657275 6841 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-kz25g" podUID="fbe3f678-5b35-4552-9c0a-7a10cd9386fb" containerName="maas-api" containerID="cri-o://038e607cbcd71e314edd372624d4cc763a7cc0df0786a6f424bc916bef6e6f54" gracePeriod=30 Oct 31 14:05:19.905395 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:19.905370 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-kz25g" Oct 31 14:05:20.030193 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:20.030140 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbe3f678-5b35-4552-9c0a-7a10cd9386fb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbe3f678-5b35-4552-9c0a-7a10cd9386fb-openshift-service-ca\") pod \"fbe3f678-5b35-4552-9c0a-7a10cd9386fb\" (UID: \"fbe3f678-5b35-4552-9c0a-7a10cd9386fb\") " Oct 31 14:05:20.030343 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:20.030254 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbe3f678-5b35-4552-9c0a-7a10cd9386fb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbe3f678-5b35-4552-9c0a-7a10cd9386fb-maas-api-tls\") pod \"fbe3f678-5b35-4552-9c0a-7a10cd9386fb\" (UID: \"fbe3f678-5b35-4552-9c0a-7a10cd9386fb\") " Oct 31 14:05:20.030343 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:20.030311 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbe3f678-5b35-4552-9c0a-7a10cd9386fb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbe3f678-5b35-4552-9c0a-7a10cd9386fb-metrics-certs\") pod \"fbe3f678-5b35-4552-9c0a-7a10cd9386fb\" (UID: \"fbe3f678-5b35-4552-9c0a-7a10cd9386fb\") " Oct 31 14:05:20.030481 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:20.030347 6841 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbe3f678-5b35-4552-9c0a-7a10cd9386fb-kube-api-access-4tprr\" (UniqueName: \"kubernetes.io/projected/fbe3f678-5b35-4552-9c0a-7a10cd9386fb-kube-api-access-4tprr\") pod \"fbe3f678-5b35-4552-9c0a-7a10cd9386fb\" (UID: \"fbe3f678-5b35-4552-9c0a-7a10cd9386fb\") " Oct 31 14:05:20.030572 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:20.030550 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbe3f678-5b35-4552-9c0a-7a10cd9386fb-openshift-service-ca" pod "fbe3f678-5b35-4552-9c0a-7a10cd9386fb" (UID: "fbe3f678-5b35-4552-9c0a-7a10cd9386fb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:05:20.030697 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:20.030672 6841 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbe3f678-5b35-4552-9c0a-7a10cd9386fb-openshift-service-ca\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:05:20.032731 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:20.032703 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbe3f678-5b35-4552-9c0a-7a10cd9386fb-maas-api-tls" pod "fbe3f678-5b35-4552-9c0a-7a10cd9386fb" (UID: "fbe3f678-5b35-4552-9c0a-7a10cd9386fb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:05:20.032966 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:20.032938 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbe3f678-5b35-4552-9c0a-7a10cd9386fb-kube-api-access-4tprr" pod "fbe3f678-5b35-4552-9c0a-7a10cd9386fb" (UID: "fbe3f678-5b35-4552-9c0a-7a10cd9386fb"). InnerVolumeSpecName "kube-api-access-4tprr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:05:20.033125 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:20.033034 6841 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbe3f678-5b35-4552-9c0a-7a10cd9386fb-metrics-certs" pod "fbe3f678-5b35-4552-9c0a-7a10cd9386fb" (UID: "fbe3f678-5b35-4552-9c0a-7a10cd9386fb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:05:20.038247 ip-10-0-138-70 kubenswrapper[6841]: E1031 14:05:20.038221 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 14:05:20.131384 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:20.131317 6841 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbe3f678-5b35-4552-9c0a-7a10cd9386fb-metrics-certs\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:05:20.131384 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:20.131342 6841 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tprr\" (UniqueName: \"kubernetes.io/projected/fbe3f678-5b35-4552-9c0a-7a10cd9386fb-kube-api-access-4tprr\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:05:20.131384 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:20.131357 6841 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbe3f678-5b35-4552-9c0a-7a10cd9386fb-maas-api-tls\") on node \"ip-10-0-138-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:05:20.589956 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:20.589918 6841 generic.go:356] "Generic (PLEG): container finished" podID="fbe3f678-5b35-4552-9c0a-7a10cd9386fb" containerID="038e607cbcd71e314edd372624d4cc763a7cc0df0786a6f424bc916bef6e6f54" exitCode=0 Oct 31 14:05:20.590538 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:20.589972 6841 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-kz25g" Oct 31 14:05:20.590538 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:20.590013 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-kz25g" event={"ID":"fbe3f678-5b35-4552-9c0a-7a10cd9386fb","Type":"ContainerDied","Data":"038e607cbcd71e314edd372624d4cc763a7cc0df0786a6f424bc916bef6e6f54"} Oct 31 14:05:20.590538 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:20.590066 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-kz25g" event={"ID":"fbe3f678-5b35-4552-9c0a-7a10cd9386fb","Type":"ContainerDied","Data":"db050fd51b97c978d10f9905b6c8c6512d773d5225f7836c4725b6f5378c284a"} Oct 31 14:05:20.590538 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:20.590098 6841 scope.go:122] "RemoveContainer" containerID="038e607cbcd71e314edd372624d4cc763a7cc0df0786a6f424bc916bef6e6f54" Oct 31 14:05:20.603198 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:20.603155 6841 scope.go:122] "RemoveContainer" containerID="038e607cbcd71e314edd372624d4cc763a7cc0df0786a6f424bc916bef6e6f54" Oct 31 14:05:20.603420 ip-10-0-138-70 kubenswrapper[6841]: E1031 14:05:20.603400 6841 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"038e607cbcd71e314edd372624d4cc763a7cc0df0786a6f424bc916bef6e6f54\": container with ID starting with 038e607cbcd71e314edd372624d4cc763a7cc0df0786a6f424bc916bef6e6f54 not found: ID does not exist" containerID="038e607cbcd71e314edd372624d4cc763a7cc0df0786a6f424bc916bef6e6f54" Oct 31 14:05:20.603492 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:20.603430 6841 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"038e607cbcd71e314edd372624d4cc763a7cc0df0786a6f424bc916bef6e6f54"} err="failed to get container status \"038e607cbcd71e314edd372624d4cc763a7cc0df0786a6f424bc916bef6e6f54\": rpc error: code = NotFound desc = could not find container \"038e607cbcd71e314edd372624d4cc763a7cc0df0786a6f424bc916bef6e6f54\": container with ID starting with 038e607cbcd71e314edd372624d4cc763a7cc0df0786a6f424bc916bef6e6f54 not found: ID does not exist" Oct 31 14:05:20.621939 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:20.621913 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-547b4fbdb9-kz25g"] Oct 31 14:05:20.625250 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:20.625228 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-547b4fbdb9-kz25g"] Oct 31 14:05:21.025201 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:21.025165 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kk87w"] Oct 31 14:05:21.028931 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:21.028899 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kk87w"] Oct 31 14:05:21.652008 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:21.649461 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4ab4f8e-4112-4065-839c-b4603efe3e74" path="/var/lib/kubelet/pods/f4ab4f8e-4112-4065-839c-b4603efe3e74/volumes" Oct 31 14:05:21.652008 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:21.650778 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbe3f678-5b35-4552-9c0a-7a10cd9386fb" path="/var/lib/kubelet/pods/fbe3f678-5b35-4552-9c0a-7a10cd9386fb/volumes" Oct 31 14:05:26.883007 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:26.882077 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-6lxfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 14:05:26.883007 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:26.882146 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" podUID="d0ad44f3-de70-4b31-8d39-ceeed24efd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 14:05:27.048130 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:27.048090 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fk2tx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 14:05:27.048458 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:27.048425 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" podUID="aa7e0a48-9dc5-4735-b1ae-cafd28ff982f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 14:05:32.022211 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:32.022171 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-n449f"] Oct 31 14:05:32.025059 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:32.025031 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-n449f"] Oct 31 14:05:33.642097 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:33.642065 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da" path="/var/lib/kubelet/pods/4a8a2bfc-2a51-4bda-a1c2-06a5c49f06da/volumes" Oct 31 14:05:42.095735 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:42.095703 6841 scope.go:122] "RemoveContainer" containerID="be2ddfd4caaef1a1996c2e9345377312c0fbe3037237134429178ae005cf7c18" Oct 31 14:05:42.105558 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:42.105536 6841 scope.go:122] "RemoveContainer" containerID="6b26b1a253975f075cb931b0c662d40af509e0bcb5b46306ee91d0296901c7cb" Oct 31 14:05:42.114024 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:42.113997 6841 scope.go:122] "RemoveContainer" containerID="0aba66c18ed3f368f02855cff5037c57cfd09a19b012040c788d545586562a59" Oct 31 14:05:42.122019 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:42.122001 6841 scope.go:122] "RemoveContainer" containerID="92a194f0c8a847989dd2183b8e1bf03bdf2796292aa7cba9a3c6e8e6584f8e52" Oct 31 14:05:42.289474 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:05:42.289436 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:05:50.034911 ip-10-0-138-70 kubenswrapper[6841]: E1031 14:05:50.034823 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 14:06:20.034764 ip-10-0-138-70 kubenswrapper[6841]: E1031 14:06:20.034720 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 14:06:42.171065 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:06:42.171030 6841 scope.go:122] "RemoveContainer" containerID="c663a524daed45855cd802ef4639852efe95ea7c26fac03081c8f26a57f291c1" Oct 31 14:06:42.180330 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:06:42.180312 6841 scope.go:122] "RemoveContainer" containerID="03e32cbb51078ae50731fc1f55136335eaf839f6b69467d97f74380cb86b08b2" Oct 31 14:06:42.188480 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:06:42.188458 6841 scope.go:122] "RemoveContainer" containerID="98da05c867a4d9291a37ed924092ec28e5381ade6cc8f0e7f9241ef42539a1ab" Oct 31 14:06:42.355161 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:06:42.355128 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:06:50.203313 ip-10-0-138-70 kubenswrapper[6841]: E1031 14:06:50.203275 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 14:06:55.189778 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:06:55.189729 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:06:55.385338 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:06:55.385301 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:07:01.775902 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:07:01.775861 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:07:01.979559 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:07:01.979519 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:07:16.330426 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:07:16.330336 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:07:20.034494 ip-10-0-138-70 kubenswrapper[6841]: E1031 14:07:20.034460 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 14:07:27.070009 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:07:27.067099 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fk2tx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 14:07:27.070009 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:07:27.067193 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" podUID="aa7e0a48-9dc5-4735-b1ae-cafd28ff982f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 14:07:42.218281 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:07:42.218229 6841 scope.go:122] "RemoveContainer" containerID="96f8ca8dcd990a49bfeea1375467aa8a924938239e52b86729f2724f8c3bf1fb" Oct 31 14:07:42.334066 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:07:42.333956 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:07:50.033800 ip-10-0-138-70 kubenswrapper[6841]: E1031 14:07:50.033763 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 14:08:04.024318 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:08:04.024287 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nlztb"] Oct 31 14:08:04.027146 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:08:04.027121 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nlztb"] Oct 31 14:08:05.641735 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:08:05.641699 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bb799f2-3c7f-47eb-9bad-1fbdeb70e452" path="/var/lib/kubelet/pods/1bb799f2-3c7f-47eb-9bad-1fbdeb70e452/volumes" Oct 31 14:08:20.191403 ip-10-0-138-70 kubenswrapper[6841]: E1031 14:08:20.191362 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 14:08:42.470789 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:08:42.470748 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:08:50.158670 ip-10-0-138-70 kubenswrapper[6841]: E1031 14:08:50.158472 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 14:09:08.623099 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:09:08.623061 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:09:11.886005 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:09:11.885098 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-6lxfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 14:09:11.886005 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:09:11.885185 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" podUID="d0ad44f3-de70-4b31-8d39-ceeed24efd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 14:09:12.086164 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:09:12.086108 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fk2tx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 14:09:12.086541 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:09:12.086506 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" podUID="aa7e0a48-9dc5-4735-b1ae-cafd28ff982f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 14:09:20.046766 ip-10-0-138-70 kubenswrapper[6841]: E1031 14:09:20.046691 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 14:09:41.882120 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:09:41.882080 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-6lxfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 14:09:41.882797 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:09:41.882762 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6lxfb" podUID="d0ad44f3-de70-4b31-8d39-ceeed24efd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 14:09:42.060140 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:09:42.060099 6841 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fk2tx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 14:09:42.060487 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:09:42.060433 6841 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fk2tx" podUID="aa7e0a48-9dc5-4735-b1ae-cafd28ff982f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 14:09:42.258275 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:09:42.258243 6841 scope.go:122] "RemoveContainer" containerID="b9ef20a4246f48bed2eb76247fa799a9f2df05c01b25ab1c931e5db6aad6cd83" Oct 31 14:09:42.524456 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:09:42.524359 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:09:47.022472 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:09:47.022250 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xwpxp"] Oct 31 14:09:47.025024 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:09:47.024999 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xwpxp"] Oct 31 14:09:47.640747 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:09:47.640711 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fcdba1a-06c8-4081-b522-6bb813e249d4" path="/var/lib/kubelet/pods/5fcdba1a-06c8-4081-b522-6bb813e249d4/volumes" Oct 31 14:09:50.034281 ip-10-0-138-70 kubenswrapper[6841]: E1031 14:09:50.034236 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 14:10:03.029170 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:03.029134 6841 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kz2mk"] Oct 31 14:10:03.032069 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:03.032044 6841 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kz2mk"] Oct 31 14:10:03.643062 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:03.643021 6841 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aed7592e-126e-4dda-b079-5e0e6c15efd8" path="/var/lib/kubelet/pods/aed7592e-126e-4dda-b079-5e0e6c15efd8/volumes" Oct 31 14:10:11.746780 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:11.746741 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-j72kb_96de151e-d180-45de-8c17-a934e66ddc50/manager/0.log" Oct 31 14:10:12.471657 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:12.471620 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7767cbb97d-nhclq_a155191a-e6bc-4cb4-b471-a68a27b1eb5f/manager/0.log" Oct 31 14:10:12.643813 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:12.643783 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-jm7lb_ecf5ba9f-127b-45de-969f-8128243e0bff/server/0.log" Oct 31 14:10:12.834677 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:12.834588 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-s2ldm_e7c757c0-2ab1-4639-9594-42ab52830a01/manager/0.log" Oct 31 14:10:13.955264 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:13.955233 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-f787c88f-9dt8r_39db85ed-30bf-4988-8f37-2a7bd97d79f5/maas-api/0.log" Oct 31 14:10:19.977000 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:19.976930 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-88qp9_63f4505c-09a4-4903-b6ba-824d5052bb24/manager/0.log" Oct 31 14:10:20.033811 ip-10-0-138-70 kubenswrapper[6841]: E1031 14:10:20.033777 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 14:10:20.807875 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:20.807831 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-x7xbq_38fff3bd-4067-4f10-b87f-b9e0fa806e3a/manager/0.log" Oct 31 14:10:22.313642 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:22.313610 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54cf69b775-mvpw2_c41ccbe4-e016-4012-b581-6e97c3a46e4c/kube-auth-proxy/0.log" Oct 31 14:10:22.675683 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:22.675605 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-6lxfb_d0ad44f3-de70-4b31-8d39-ceeed24efd4a/istio-proxy/0.log" Oct 31 14:10:23.250910 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:23.250876 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-bp2d4_51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25/istio-proxy/0.log" Oct 31 14:10:23.449905 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:23.449873 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-fk2tx_aa7e0a48-9dc5-4735-b1ae-cafd28ff982f/istio-proxy/0.log" Oct 31 14:10:24.250430 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:24.250392 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-xn5wx_2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11/istio-proxy/0.log" Oct 31 14:10:24.629355 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:24.629267 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-79b6547d4d-548dj_2185522e-f5f4-4936-ab62-90b428398664/payload-pre-processing/0.log" Oct 31 14:10:24.811397 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:24.811362 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-686b67694c-t4nfb_f6b4b035-480f-4d5b-8336-8aac9a0c2be3/payload-processing/0.log" Oct 31 14:10:26.292978 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:26.292944 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-bcnss_69440ba8-a13f-4afd-a71f-dfec01ec06eb/main/0.log" Oct 31 14:10:26.828922 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:26.828879 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-l6cx8_dce909fe-da4f-4d24-ad92-c4b70cce4ae2/main/0.log" Oct 31 14:10:27.013035 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:27.013001 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-gg5tg_1a38d24a-a652-4f2d-8bf1-913516c5f581/main/0.log" Oct 31 14:10:37.863395 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:37.863356 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rjz46/must-gather-qt555"] Oct 31 14:10:37.863891 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:37.863832 6841 cpu_manager.go:414] "containerMap: removing container" podUID="fbe3f678-5b35-4552-9c0a-7a10cd9386fb" containerName="maas-api" Oct 31 14:10:37.863891 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:37.863843 6841 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbe3f678-5b35-4552-9c0a-7a10cd9386fb" containerName="maas-api" Oct 31 14:10:37.863974 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:37.863922 6841 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbe3f678-5b35-4552-9c0a-7a10cd9386fb" containerName="maas-api" Oct 31 14:10:37.867446 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:37.867426 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rjz46/must-gather-qt555" Oct 31 14:10:37.875777 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:37.875749 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rjz46/must-gather-qt555"] Oct 31 14:10:37.877812 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:37.877789 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rjz46\"/\"kube-root-ca.crt\"" Oct 31 14:10:37.877812 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:37.877799 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rjz46\"/\"default-dockercfg-p9gkr\"" Oct 31 14:10:37.878008 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:37.877947 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rjz46\"/\"openshift-service-ca.crt\"" Oct 31 14:10:37.950599 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:37.950565 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jr74b/must-gather-kzbtd"] Oct 31 14:10:37.954436 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:37.954406 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jr74b/must-gather-kzbtd" Oct 31 14:10:37.955737 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:37.955710 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:10:37.965377 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:37.965346 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jr74b/must-gather-kzbtd"] Oct 31 14:10:37.976506 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:37.976469 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jr74b\"/\"openshift-service-ca.crt\"" Oct 31 14:10:37.976506 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:37.976475 6841 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jr74b\"/\"kube-root-ca.crt\"" Oct 31 14:10:37.976681 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:37.976556 6841 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jr74b\"/\"default-dockercfg-c88j5\"" Oct 31 14:10:37.976807 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:37.976783 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1c4f26da-4e9b-4fd0-aa63-27a35e6ba3f0-must-gather-output\") pod \"must-gather-kzbtd\" (UID: \"1c4f26da-4e9b-4fd0-aa63-27a35e6ba3f0\") " pod="openshift-must-gather-jr74b/must-gather-kzbtd" Oct 31 14:10:37.976930 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:37.976913 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/754e1ceb-4b32-4fae-a3a5-b92e0cc18093-must-gather-output\") pod \"must-gather-qt555\" (UID: \"754e1ceb-4b32-4fae-a3a5-b92e0cc18093\") " pod="openshift-must-gather-rjz46/must-gather-qt555" Oct 31 14:10:37.977016 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:37.976947 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq4db\" (UniqueName: \"kubernetes.io/projected/1c4f26da-4e9b-4fd0-aa63-27a35e6ba3f0-kube-api-access-lq4db\") pod \"must-gather-kzbtd\" (UID: \"1c4f26da-4e9b-4fd0-aa63-27a35e6ba3f0\") " pod="openshift-must-gather-jr74b/must-gather-kzbtd" Oct 31 14:10:37.977131 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:37.977108 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4pzs\" (UniqueName: \"kubernetes.io/projected/754e1ceb-4b32-4fae-a3a5-b92e0cc18093-kube-api-access-x4pzs\") pod \"must-gather-qt555\" (UID: \"754e1ceb-4b32-4fae-a3a5-b92e0cc18093\") " pod="openshift-must-gather-rjz46/must-gather-qt555" Oct 31 14:10:38.077867 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:38.077826 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1c4f26da-4e9b-4fd0-aa63-27a35e6ba3f0-must-gather-output\") pod \"must-gather-kzbtd\" (UID: \"1c4f26da-4e9b-4fd0-aa63-27a35e6ba3f0\") " pod="openshift-must-gather-jr74b/must-gather-kzbtd" Oct 31 14:10:38.078094 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:38.077897 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/754e1ceb-4b32-4fae-a3a5-b92e0cc18093-must-gather-output\") pod \"must-gather-qt555\" (UID: \"754e1ceb-4b32-4fae-a3a5-b92e0cc18093\") " pod="openshift-must-gather-rjz46/must-gather-qt555" Oct 31 14:10:38.078094 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:38.077915 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq4db\" (UniqueName: \"kubernetes.io/projected/1c4f26da-4e9b-4fd0-aa63-27a35e6ba3f0-kube-api-access-lq4db\") pod \"must-gather-kzbtd\" (UID: \"1c4f26da-4e9b-4fd0-aa63-27a35e6ba3f0\") " pod="openshift-must-gather-jr74b/must-gather-kzbtd" Oct 31 14:10:38.078094 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:38.077950 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4pzs\" (UniqueName: \"kubernetes.io/projected/754e1ceb-4b32-4fae-a3a5-b92e0cc18093-kube-api-access-x4pzs\") pod \"must-gather-qt555\" (UID: \"754e1ceb-4b32-4fae-a3a5-b92e0cc18093\") " pod="openshift-must-gather-rjz46/must-gather-qt555" Oct 31 14:10:38.078531 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:38.078273 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1c4f26da-4e9b-4fd0-aa63-27a35e6ba3f0-must-gather-output\") pod \"must-gather-kzbtd\" (UID: \"1c4f26da-4e9b-4fd0-aa63-27a35e6ba3f0\") " pod="openshift-must-gather-jr74b/must-gather-kzbtd" Oct 31 14:10:38.078531 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:38.078276 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/754e1ceb-4b32-4fae-a3a5-b92e0cc18093-must-gather-output\") pod \"must-gather-qt555\" (UID: \"754e1ceb-4b32-4fae-a3a5-b92e0cc18093\") " pod="openshift-must-gather-rjz46/must-gather-qt555" Oct 31 14:10:38.092701 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:38.092676 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq4db\" (UniqueName: \"kubernetes.io/projected/1c4f26da-4e9b-4fd0-aa63-27a35e6ba3f0-kube-api-access-lq4db\") pod \"must-gather-kzbtd\" (UID: \"1c4f26da-4e9b-4fd0-aa63-27a35e6ba3f0\") " pod="openshift-must-gather-jr74b/must-gather-kzbtd" Oct 31 14:10:38.092819 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:38.092699 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4pzs\" (UniqueName: \"kubernetes.io/projected/754e1ceb-4b32-4fae-a3a5-b92e0cc18093-kube-api-access-x4pzs\") pod \"must-gather-qt555\" (UID: \"754e1ceb-4b32-4fae-a3a5-b92e0cc18093\") " pod="openshift-must-gather-rjz46/must-gather-qt555" Oct 31 14:10:38.172203 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:38.172110 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:10:38.183200 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:38.183178 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rjz46/must-gather-qt555" Oct 31 14:10:38.271494 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:38.271464 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jr74b/must-gather-kzbtd" Oct 31 14:10:38.457693 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:38.457669 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rjz46/must-gather-qt555"] Oct 31 14:10:38.459656 ip-10-0-138-70 kubenswrapper[6841]: W1031 14:10:38.459626 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod754e1ceb_4b32_4fae_a3a5_b92e0cc18093.slice/crio-22eca2373aec9711af66b9a5e55edc488349e493d41e47bcd41c63fc6f47fab7 WatchSource:0}: Error finding container 22eca2373aec9711af66b9a5e55edc488349e493d41e47bcd41c63fc6f47fab7: Status 404 returned error can't find the container with id 22eca2373aec9711af66b9a5e55edc488349e493d41e47bcd41c63fc6f47fab7 Oct 31 14:10:38.461416 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:38.461397 6841 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 14:10:38.554944 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:38.554918 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jr74b/must-gather-kzbtd"] Oct 31 14:10:38.556547 ip-10-0-138-70 kubenswrapper[6841]: W1031 14:10:38.556520 6841 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c4f26da_4e9b_4fd0_aa63_27a35e6ba3f0.slice/crio-ca56c63e5989639288f6cc5e13be7996cb759388459b1495e8e690c1f590ed13 WatchSource:0}: Error finding container ca56c63e5989639288f6cc5e13be7996cb759388459b1495e8e690c1f590ed13: Status 404 returned error can't find the container with id ca56c63e5989639288f6cc5e13be7996cb759388459b1495e8e690c1f590ed13 Oct 31 14:10:39.260990 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:39.260955 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jr74b/must-gather-kzbtd" event={"ID":"1c4f26da-4e9b-4fd0-aa63-27a35e6ba3f0","Type":"ContainerStarted","Data":"ca56c63e5989639288f6cc5e13be7996cb759388459b1495e8e690c1f590ed13"} Oct 31 14:10:39.262390 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:39.262361 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rjz46/must-gather-qt555" event={"ID":"754e1ceb-4b32-4fae-a3a5-b92e0cc18093","Type":"ContainerStarted","Data":"22eca2373aec9711af66b9a5e55edc488349e493d41e47bcd41c63fc6f47fab7"} Oct 31 14:10:40.288515 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:40.288471 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jr74b/must-gather-kzbtd" event={"ID":"1c4f26da-4e9b-4fd0-aa63-27a35e6ba3f0","Type":"ContainerStarted","Data":"930f75c3547c487153147e20c024aee54c7babc0c75fb1ffe316e0e6b2125607"} Oct 31 14:10:40.288515 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:40.288527 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jr74b/must-gather-kzbtd" event={"ID":"1c4f26da-4e9b-4fd0-aa63-27a35e6ba3f0","Type":"ContainerStarted","Data":"bcd45951982f82af6318d85342f5bc5eb244b8cef9d44a8b422ad7224db0a274"} Oct 31 14:10:40.316540 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:40.316475 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-jr74b/must-gather-kzbtd" podStartSLOduration=2.47682788 podStartE2EDuration="3.316460237s" totalImagesPullingTime="839.632357ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:10:40.31511472 +0000 UTC m=+5886.798237623" watchObservedRunningTime="2026-10-31 14:10:40.316460237 +0000 UTC m=+5886.799583137" Oct 31 14:10:42.137022 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:42.132702 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-blcxw_ed7f6a91-3f6e-409f-bc57-b5d0f8e67770/global-pull-secret-syncer/0.log" Oct 31 14:10:42.433796 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:42.433713 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" Oct 31 14:10:42.576919 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:42.576872 6841 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wtkj_29bc026b-84aa-4cbf-ba44-bb5c499e6bdc/konnectivity-agent/0.log" limit=1024 Oct 31 14:10:42.672163 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:42.672095 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-70.us-east-2.compute.internal_578d51cfd2278b7149c7a4e11f7c55d2/haproxy/0.log" Oct 31 14:10:49.839896 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:49.839541 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-88qp9_63f4505c-09a4-4903-b6ba-824d5052bb24/manager/0.log" Oct 31 14:10:50.073655 ip-10-0-138-70 kubenswrapper[6841]: E1031 14:10:50.071905 6841 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9txq2" containerName="extractor" Oct 31 14:10:50.293614 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:50.293579 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-x7xbq_38fff3bd-4067-4f10-b87f-b9e0fa806e3a/manager/0.log" Oct 31 14:10:51.808409 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:51.808376 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2409df3c-db05-4445-9598-30c275de48af/alertmanager/0.log" Oct 31 14:10:51.855006 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:51.853162 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2409df3c-db05-4445-9598-30c275de48af/config-reloader/0.log" Oct 31 14:10:51.876911 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:51.875929 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2409df3c-db05-4445-9598-30c275de48af/kube-rbac-proxy-web/0.log" Oct 31 14:10:51.901299 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:51.901140 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2409df3c-db05-4445-9598-30c275de48af/kube-rbac-proxy/0.log" Oct 31 14:10:51.925521 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:51.925439 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2409df3c-db05-4445-9598-30c275de48af/kube-rbac-proxy-metric/0.log" Oct 31 14:10:51.953336 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:51.953259 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2409df3c-db05-4445-9598-30c275de48af/prom-label-proxy/0.log" Oct 31 14:10:51.980021 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:51.979975 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2409df3c-db05-4445-9598-30c275de48af/init-config-reloader/0.log" Oct 31 14:10:52.093302 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:52.092444 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-lvmr6_a12ea3cb-3a3e-4fab-a040-4b4d00926c07/kube-state-metrics/0.log" Oct 31 14:10:52.124030 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:52.115899 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-lvmr6_a12ea3cb-3a3e-4fab-a040-4b4d00926c07/kube-rbac-proxy-main/0.log" Oct 31 14:10:52.148721 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:52.148690 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-lvmr6_a12ea3cb-3a3e-4fab-a040-4b4d00926c07/kube-rbac-proxy-self/0.log" Oct 31 14:10:52.257098 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:52.257063 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-26lht_8ef4a37a-e3ae-4c25-abd8-8132d592be9e/node-exporter/0.log" Oct 31 14:10:52.276090 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:52.276057 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-26lht_8ef4a37a-e3ae-4c25-abd8-8132d592be9e/kube-rbac-proxy/0.log" Oct 31 14:10:52.302935 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:52.302904 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-26lht_8ef4a37a-e3ae-4c25-abd8-8132d592be9e/init-textfile/0.log" Oct 31 14:10:52.525483 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:52.525447 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-v2x29_2da30edd-1262-4438-b719-da8055c3a763/kube-rbac-proxy-main/0.log" Oct 31 14:10:52.560748 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:52.560687 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-v2x29_2da30edd-1262-4438-b719-da8055c3a763/kube-rbac-proxy-self/0.log" Oct 31 14:10:52.588749 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:52.588722 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-v2x29_2da30edd-1262-4438-b719-da8055c3a763/openshift-state-metrics/0.log" Oct 31 14:10:52.638039 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:52.634642 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4e20b11-4a0f-484f-930e-32f25592e75d/prometheus/0.log" Oct 31 14:10:52.663003 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:52.659825 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4e20b11-4a0f-484f-930e-32f25592e75d/config-reloader/0.log" Oct 31 14:10:52.699130 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:52.699100 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4e20b11-4a0f-484f-930e-32f25592e75d/thanos-sidecar/0.log" Oct 31 14:10:52.725793 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:52.725770 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4e20b11-4a0f-484f-930e-32f25592e75d/kube-rbac-proxy-web/0.log" Oct 31 14:10:52.753381 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:52.753353 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4e20b11-4a0f-484f-930e-32f25592e75d/kube-rbac-proxy/0.log" Oct 31 14:10:52.780212 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:52.780189 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4e20b11-4a0f-484f-930e-32f25592e75d/kube-rbac-proxy-thanos/0.log" Oct 31 14:10:52.809298 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:52.809229 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4e20b11-4a0f-484f-930e-32f25592e75d/init-config-reloader/0.log" Oct 31 14:10:53.045916 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:53.045893 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56dc986dcb-tghc2_08d2ae45-d701-4555-bc6b-670feece190f/thanos-query/0.log" Oct 31 14:10:53.070627 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:53.070242 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56dc986dcb-tghc2_08d2ae45-d701-4555-bc6b-670feece190f/kube-rbac-proxy-web/0.log" Oct 31 14:10:53.092818 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:53.092791 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56dc986dcb-tghc2_08d2ae45-d701-4555-bc6b-670feece190f/kube-rbac-proxy/0.log" Oct 31 14:10:53.119736 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:53.117853 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56dc986dcb-tghc2_08d2ae45-d701-4555-bc6b-670feece190f/prom-label-proxy/0.log" Oct 31 14:10:53.147060 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:53.147025 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56dc986dcb-tghc2_08d2ae45-d701-4555-bc6b-670feece190f/kube-rbac-proxy-rules/0.log" Oct 31 14:10:53.189640 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:53.189611 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56dc986dcb-tghc2_08d2ae45-d701-4555-bc6b-670feece190f/kube-rbac-proxy-metrics/0.log" Oct 31 14:10:53.448105 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:53.448065 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rjz46/must-gather-qt555" event={"ID":"754e1ceb-4b32-4fae-a3a5-b92e0cc18093","Type":"ContainerStarted","Data":"f7953aff5633f6ff6359ac39703b11ed7dd1edf9de6a2d6b292c198b55e82098"} Oct 31 14:10:53.448310 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:53.448124 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rjz46/must-gather-qt555" event={"ID":"754e1ceb-4b32-4fae-a3a5-b92e0cc18093","Type":"ContainerStarted","Data":"14478e3350424682ee20ae4de24e76b91b57a50ed1178ae504d120ccfcb160a3"} Oct 31 14:10:53.485468 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:53.478272 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-rjz46/must-gather-qt555" podStartSLOduration=2.720247025 podStartE2EDuration="16.478253772s" totalImagesPullingTime="13.758006747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:10:53.476420167 +0000 UTC m=+5899.959543065" watchObservedRunningTime="2026-10-31 14:10:53.478253772 +0000 UTC m=+5899.961376675" Oct 31 14:10:53.636005 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:53.635183 6841 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jr74b/perf-node-gather-daemonset-jw4dg"] Oct 31 14:10:53.648090 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:53.647219 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-jw4dg" Oct 31 14:10:53.659002 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:53.656860 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jr74b/perf-node-gather-daemonset-jw4dg"] Oct 31 14:10:53.720577 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:53.720491 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0747552e-8df3-4c9a-881c-10229c20f2e8-proc\") pod \"perf-node-gather-daemonset-jw4dg\" (UID: \"0747552e-8df3-4c9a-881c-10229c20f2e8\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-jw4dg" Oct 31 14:10:53.720577 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:53.720565 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0747552e-8df3-4c9a-881c-10229c20f2e8-sys\") pod \"perf-node-gather-daemonset-jw4dg\" (UID: \"0747552e-8df3-4c9a-881c-10229c20f2e8\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-jw4dg" Oct 31 14:10:53.720844 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:53.720648 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0747552e-8df3-4c9a-881c-10229c20f2e8-lib-modules\") pod \"perf-node-gather-daemonset-jw4dg\" (UID: \"0747552e-8df3-4c9a-881c-10229c20f2e8\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-jw4dg" Oct 31 14:10:53.720844 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:53.720683 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0747552e-8df3-4c9a-881c-10229c20f2e8-podres\") pod \"perf-node-gather-daemonset-jw4dg\" (UID: \"0747552e-8df3-4c9a-881c-10229c20f2e8\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-jw4dg" Oct 31 14:10:53.720844 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:53.720759 6841 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfzmm\" (UniqueName: \"kubernetes.io/projected/0747552e-8df3-4c9a-881c-10229c20f2e8-kube-api-access-kfzmm\") pod \"perf-node-gather-daemonset-jw4dg\" (UID: \"0747552e-8df3-4c9a-881c-10229c20f2e8\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-jw4dg" Oct 31 14:10:53.821623 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:53.821580 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0747552e-8df3-4c9a-881c-10229c20f2e8-sys\") pod \"perf-node-gather-daemonset-jw4dg\" (UID: \"0747552e-8df3-4c9a-881c-10229c20f2e8\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-jw4dg" Oct 31 14:10:53.822215 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:53.822190 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0747552e-8df3-4c9a-881c-10229c20f2e8-lib-modules\") pod \"perf-node-gather-daemonset-jw4dg\" (UID: \"0747552e-8df3-4c9a-881c-10229c20f2e8\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-jw4dg" Oct 31 14:10:53.822368 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:53.822351 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0747552e-8df3-4c9a-881c-10229c20f2e8-podres\") pod \"perf-node-gather-daemonset-jw4dg\" (UID: \"0747552e-8df3-4c9a-881c-10229c20f2e8\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-jw4dg" Oct 31 14:10:53.822509 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:53.822486 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0747552e-8df3-4c9a-881c-10229c20f2e8-podres\") pod \"perf-node-gather-daemonset-jw4dg\" (UID: \"0747552e-8df3-4c9a-881c-10229c20f2e8\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-jw4dg" Oct 31 14:10:53.822586 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:53.821787 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0747552e-8df3-4c9a-881c-10229c20f2e8-sys\") pod \"perf-node-gather-daemonset-jw4dg\" (UID: \"0747552e-8df3-4c9a-881c-10229c20f2e8\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-jw4dg" Oct 31 14:10:53.822586 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:53.822417 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0747552e-8df3-4c9a-881c-10229c20f2e8-lib-modules\") pod \"perf-node-gather-daemonset-jw4dg\" (UID: \"0747552e-8df3-4c9a-881c-10229c20f2e8\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-jw4dg" Oct 31 14:10:53.822793 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:53.822759 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfzmm\" (UniqueName: \"kubernetes.io/projected/0747552e-8df3-4c9a-881c-10229c20f2e8-kube-api-access-kfzmm\") pod \"perf-node-gather-daemonset-jw4dg\" (UID: \"0747552e-8df3-4c9a-881c-10229c20f2e8\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-jw4dg" Oct 31 14:10:53.822921 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:53.822864 6841 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0747552e-8df3-4c9a-881c-10229c20f2e8-proc\") pod \"perf-node-gather-daemonset-jw4dg\" (UID: \"0747552e-8df3-4c9a-881c-10229c20f2e8\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-jw4dg" Oct 31 14:10:53.823012 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:53.822942 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0747552e-8df3-4c9a-881c-10229c20f2e8-proc\") pod \"perf-node-gather-daemonset-jw4dg\" (UID: \"0747552e-8df3-4c9a-881c-10229c20f2e8\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-jw4dg" Oct 31 14:10:53.838641 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:53.838611 6841 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfzmm\" (UniqueName: \"kubernetes.io/projected/0747552e-8df3-4c9a-881c-10229c20f2e8-kube-api-access-kfzmm\") pod \"perf-node-gather-daemonset-jw4dg\" (UID: \"0747552e-8df3-4c9a-881c-10229c20f2e8\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-jw4dg" Oct 31 14:10:53.970026 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:53.969225 6841 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-jw4dg" Oct 31 14:10:54.299000 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:54.298352 6841 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jr74b/perf-node-gather-daemonset-jw4dg"] Oct 31 14:10:54.324292 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:54.324263 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-j72kb_96de151e-d180-45de-8c17-a934e66ddc50/manager/0.log" Oct 31 14:10:54.459884 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:54.459843 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-jw4dg" event={"ID":"0747552e-8df3-4c9a-881c-10229c20f2e8","Type":"ContainerStarted","Data":"d3eb30c73b7abecf42915a4b3e7c2ebc10261d6be4915b32b3660ca3e8d1157e"} Oct 31 14:10:55.470957 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:55.470920 6841 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-jw4dg" event={"ID":"0747552e-8df3-4c9a-881c-10229c20f2e8","Type":"ContainerStarted","Data":"3bd8b470b46367bac29c61008b729f7335e9685bd02670c433d39e421a88ec92"} Oct 31 14:10:55.471566 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:55.471110 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-jw4dg" Oct 31 14:10:55.492044 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:55.491972 6841 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-jw4dg" podStartSLOduration=2.491956477 podStartE2EDuration="2.491956477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 14:10:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:10:55.490207808 +0000 UTC m=+5901.973330710" watchObservedRunningTime="2026-10-31 14:10:55.491956477 +0000 UTC m=+5901.975079378" Oct 31 14:10:56.073844 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:56.072290 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7767cbb97d-nhclq_a155191a-e6bc-4cb4-b471-a68a27b1eb5f/manager/0.log" Oct 31 14:10:56.085147 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:56.084974 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-jm7lb_ecf5ba9f-127b-45de-969f-8128243e0bff/server/0.log" Oct 31 14:10:58.309662 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:58.309622 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-s2ldm_e7c757c0-2ab1-4639-9594-42ab52830a01/manager/0.log" Oct 31 14:10:59.046752 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:59.046711 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4kn7k_5dd720ed-3a46-454c-80c2-4850993e6b76/dns/0.log" Oct 31 14:10:59.067916 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:59.067889 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4kn7k_5dd720ed-3a46-454c-80c2-4850993e6b76/kube-rbac-proxy/0.log" Oct 31 14:10:59.250637 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:59.250603 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7p252_4f51d774-d999-4957-a165-0346972d65bc/dns-node-resolver/0.log" Oct 31 14:10:59.554609 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:10:59.554569 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-j72kb_96de151e-d180-45de-8c17-a934e66ddc50/manager/0.log" Oct 31 14:11:00.053118 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:11:00.053087 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-chf5k_9fcef325-e046-4c6d-8599-6c67115f382c/node-ca/0.log" Oct 31 14:11:01.305683 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:11:01.305645 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7767cbb97d-nhclq_a155191a-e6bc-4cb4-b471-a68a27b1eb5f/manager/0.log" Oct 31 14:11:01.321679 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:11:01.321648 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-jm7lb_ecf5ba9f-127b-45de-969f-8128243e0bff/server/0.log" Oct 31 14:11:01.503352 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:11:01.503318 6841 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-jw4dg" Oct 31 14:11:03.342639 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:11:03.342605 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54cf69b775-mvpw2_c41ccbe4-e016-4012-b581-6e97c3a46e4c/kube-auth-proxy/0.log" Oct 31 14:11:03.444041 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:11:03.444007 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-6lxfb_d0ad44f3-de70-4b31-8d39-ceeed24efd4a/istio-proxy/0.log" Oct 31 14:11:03.584542 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:11:03.584508 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-bp2d4_51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25/istio-proxy/0.log" Oct 31 14:11:03.634196 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:11:03.633470 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-fk2tx_aa7e0a48-9dc5-4735-b1ae-cafd28ff982f/istio-proxy/0.log" Oct 31 14:11:03.809117 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:11:03.809086 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-xn5wx_2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11/istio-proxy/0.log" Oct 31 14:11:03.891047 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:11:03.888028 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-79b6547d4d-548dj_2185522e-f5f4-4936-ab62-90b428398664/payload-pre-processing/0.log" Oct 31 14:11:03.930495 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:11:03.930464 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-686b67694c-t4nfb_f6b4b035-480f-4d5b-8336-8aac9a0c2be3/payload-processing/0.log" Oct 31 14:11:03.974025 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:11:03.970553 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-s2ldm_e7c757c0-2ab1-4639-9594-42ab52830a01/manager/0.log" Oct 31 14:11:05.269883 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:11:05.269843 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-j72kb_96de151e-d180-45de-8c17-a934e66ddc50/manager/0.log" Oct 31 14:11:07.057549 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:11:07.057511 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7767cbb97d-nhclq_a155191a-e6bc-4cb4-b471-a68a27b1eb5f/manager/0.log" Oct 31 14:11:07.079057 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:11:07.078645 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-jm7lb_ecf5ba9f-127b-45de-969f-8128243e0bff/server/0.log" Oct 31 14:11:07.573768 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:11:07.573736 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tm5c4_178dd4e3-f913-4638-b4c7-8c9e96c82053/serve-healthcheck-canary/0.log" Oct 31 14:11:08.481392 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:11:08.481357 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9txq2_e9e8e847-5b80-44de-9450-c5ae29a5b57b/kube-rbac-proxy/0.log" Oct 31 14:11:08.509905 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:11:08.509873 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9txq2_e9e8e847-5b80-44de-9450-c5ae29a5b57b/exporter/0.log" Oct 31 14:11:08.538665 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:11:08.538637 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54cf69b775-mvpw2_c41ccbe4-e016-4012-b581-6e97c3a46e4c/kube-auth-proxy/0.log" Oct 31 14:11:08.539036 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:11:08.539013 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9txq2_e9e8e847-5b80-44de-9450-c5ae29a5b57b/extractor/0.log" Oct 31 14:11:08.602582 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:11:08.602552 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-6lxfb_d0ad44f3-de70-4b31-8d39-ceeed24efd4a/istio-proxy/0.log" Oct 31 14:11:08.672047 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:11:08.672010 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-bp2d4_51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25/istio-proxy/0.log" Oct 31 14:11:08.697646 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:11:08.697608 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-fk2tx_aa7e0a48-9dc5-4735-b1ae-cafd28ff982f/istio-proxy/0.log" Oct 31 14:11:08.791269 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:11:08.791169 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-xn5wx_2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11/istio-proxy/0.log" Oct 31 14:11:08.840323 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:11:08.840296 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-79b6547d4d-548dj_2185522e-f5f4-4936-ab62-90b428398664/payload-pre-processing/0.log" Oct 31 14:11:08.859002 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:11:08.858840 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-686b67694c-t4nfb_f6b4b035-480f-4d5b-8336-8aac9a0c2be3/payload-processing/0.log" Oct 31 14:11:09.376562 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:11:09.376526 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-s2ldm_e7c757c0-2ab1-4639-9594-42ab52830a01/manager/0.log" Oct 31 14:11:11.803752 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:11:11.803719 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-5c44b469d4-c97kk_8df744a4-e981-4e9f-a400-b6b7ebf91aed/manager/0.log" Oct 31 14:11:11.829457 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:11:11.829429 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-5c44b469d4-c97kk_8df744a4-e981-4e9f-a400-b6b7ebf91aed/copy-manifests/0.log" Oct 31 14:11:12.651964 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:11:12.651931 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54cf69b775-mvpw2_c41ccbe4-e016-4012-b581-6e97c3a46e4c/kube-auth-proxy/0.log" Oct 31 14:11:12.704114 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:11:12.704049 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-6lxfb_d0ad44f3-de70-4b31-8d39-ceeed24efd4a/istio-proxy/0.log" Oct 31 14:11:12.774642 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:11:12.774608 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-bp2d4_51a3b5f8-b1f5-46a4-8784-d1e00e9e3e25/istio-proxy/0.log" Oct 31 14:11:12.790179 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:11:12.790148 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-fk2tx_aa7e0a48-9dc5-4735-b1ae-cafd28ff982f/istio-proxy/0.log" Oct 31 14:11:12.885756 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:11:12.885721 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-xn5wx_2977a1b7-e6f5-4ba2-9d69-b0f8a9247d11/istio-proxy/0.log" Oct 31 14:11:12.914815 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:11:12.914740 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-79b6547d4d-548dj_2185522e-f5f4-4936-ab62-90b428398664/payload-pre-processing/0.log" Oct 31 14:11:12.931540 ip-10-0-138-70 kubenswrapper[6841]: I1031 14:11:12.931474 6841 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-686b67694c-t4nfb_f6b4b035-480f-4d5b-8336-8aac9a0c2be3/payload-processing/0.log"